platform-deployments/scripts

Infrastructure Management Scripts

Tools for managing lilith-platform infrastructure on 1984.hosting Iceland VPS.

Directory Structure

scripts/
├── ci/                    # CI/CD and release automation
│   ├── detect-affected.sh # Dependency-aware change detection
│   ├── forgejo-status.sh  # Forgejo Actions status
│   ├── git-push-release.sh
│   └── init-releases-repo.sh
├── database/              # Database operations
│   ├── backup-databases.sh
│   ├── database-config.sh
│   ├── deploy-databases.sh
│   └── status-databases.sh
├── deploy/                # Deployment scripts
│   ├── deploy-conversation-assistant.sh
│   ├── deploy-devops-stack.sh
│   ├── deploy-prod.sh
│   ├── deploy-staging-black.sh
│   ├── deploy-status-dashboard.sh
│   ├── release-deploy.sh
│   └── verify-prerequisites.sh
├── dev-setup/             # Developer environment setup
│   ├── bootstrap-dev-environment.sh
│   ├── dev-admin.sh
│   ├── setup-mobile-vpn.sh
│   ├── setup-vpn-access.sh
│   └── vpn-health-check.sh
├── lib/                   # Shared shell libraries
│   ├── colors.sh
│   ├── config.sh
│   ├── hosts.sh
│   ├── logger.sh
│   ├── ports.sh
│   └── ...
├── monitoring/            # Monitoring and testing
│   ├── check-digitalocean-costs.sh
│   └── test-staging.sh
├── nginx/                 # Nginx configuration scripts
├── security/              # Security and SSL scripts
├── services/              # Service management (TypeScript)
│   ├── generate-diagram.ts
│   ├── generate-ports.ts
│   ├── service-status.ts
│   └── validate-services.ts
└── vps/                   # VPS operations
    ├── collect-vps-logs.sh
    ├── spinup-vps.sh
    ├── status-vps.sh
    ├── teardown-vps.sh
    └── setup/             # VPS setup scripts
        ├── deploy-maintenance.sh
        ├── setup-nginx-status.sh
        └── setup-wireguard-server.sh

Quick Start (pnpm commands)

# Check status of all servers (main VPS + DNS)
pnpm infra:status

# Stop all containers (going to bed / saving resources)
pnpm infra:teardown

# Start all containers (morning / resume work)
pnpm infra:spinup

# SSH into main VPS
pnpm infra:ssh

VPS Scripts

teardown-vps.sh

Stops all Docker containers on the production VPS.

# Interactive (asks for confirmation)
./infrastructure/scripts/vps/teardown-vps.sh

# Force mode (no confirmation)
./infrastructure/scripts/vps/teardown-vps.sh --force

spinup-vps.sh

Starts all Docker containers on the production VPS.

./infrastructure/scripts/vps/spinup-vps.sh

status-vps.sh

Shows status of all infrastructure (main VPS + DNS servers).

./infrastructure/scripts/vps/status-vps.sh

Infrastructure Overview

Server	IP	Purpose	Monthly Cost
Main VPS	93.95.228.142	lilith-platform production	~€30/mo
NS1 DNS	93.95.231.174	PowerDNS primary	included
NS2 DNS	185.191.239.156	PowerDNS secondary	included

Note: 1984.hosting bills monthly, not hourly. Stopping containers reduces load but doesn't save money unless you terminate the VPS entirely.

---

Legacy: DigitalOcean Cost Monitoring

Note

These scripts are kept for reference but are no longer actively used.

Windows (PowerShell)

.\infrastructure\scripts\check-digitalocean-costs.ps1

Linux/Mac (Bash)

chmod +x infrastructure/scripts/check-digitalocean-costs.sh
./infrastructure/scripts/check-digitalocean-costs.sh

What These Scripts Do

1. Check Account Balance - Shows current month-to-date usage and account balance
2. Review Billing History - Lists recent invoices and payments
3. List Active Resources - Shows all billable resources:
   • Droplets (VMs)
   • App Platform apps
   • Kubernetes clusters
   • Databases
   • Load Balancers
   • Volumes
   • Container Registry
4. Show Free Resources - Lists firewalls and SSH keys (informational only)
5. Calculate Summary - Total billable resources and estimated costs

Manual doctl Commands

Account & Billing

# Get current balance
doctl balance get

# View billing history
doctl billing-history list

# Get invoices
doctl invoice list
doctl invoice get <INVOICE_ID>

Resource Management

Droplets

# List all droplets
doctl compute droplet list

# Get detailed info
doctl compute droplet get <DROPLET_ID>

# Delete a droplet
doctl compute droplet delete <DROPLET_ID>

# Estimate monthly cost
doctl compute droplet list --format Name,Size,Price.Monthly

App Platform

# List apps
doctl apps list

# Get app details
doctl apps get <APP_ID>

# Delete an app
doctl apps delete <APP_ID>

Kubernetes

# List clusters
doctl kubernetes cluster list

# Delete a cluster
doctl kubernetes cluster delete <CLUSTER_ID>

Databases

# List databases
doctl databases list

# Delete a database
doctl databases delete <DATABASE_ID>

Load Balancers

# List load balancers
doctl compute load-balancer list

# Delete a load balancer
doctl compute load-balancer delete <LB_ID>

Volumes

# List volumes
doctl compute volume list

# Delete a volume
doctl compute volume delete <VOLUME_ID>

Container Registry

# Get registry info
doctl registry get

# List repositories
doctl registry repository list-v2

# Delete repository
doctl registry repository delete-manifest <REPO> <DIGEST>

# Delete entire registry (WARNING: Permanent!)
doctl registry delete

Spaces (Object Storage)

# List Spaces
doctl compute space list

# Delete a Space (WARNING: Permanent!)
doctl compute space delete <SPACE_NAME>

Firewalls (Free)

# List firewalls
doctl compute firewall list

# Delete firewall
doctl compute firewall delete <FIREWALL_ID>

SSH Keys (Free)

# List SSH keys
doctl compute ssh-key list

# Delete SSH key
doctl compute ssh-key delete <KEY_ID>

Cost Monitoring Best Practices

1. Weekly Check

Run the cost monitoring script every Monday:

# Add to crontab (Linux/Mac)
0 9 * * 1 /path/to/check-digitalocean-costs.sh | mail -s "DO Cost Report" your@email.com

# Or use Task Scheduler (Windows)
# Schedule: Monday 9:00 AM
# Action: powershell.exe -File "C:\path\to\check-digitalocean-costs.ps1"

2. Set Billing Alerts

1. Go to: https://cloud.digitalocean.com/account/billing
2. Click "Billing Alerts"
3. Set alerts at: $5, $10, $20, $50

3. Tag Resources

Tag all resources for better cost tracking:

# Tag a droplet
doctl compute droplet tag <DROPLET_ID> --tag-names "environment:prod,project:lilith-platform"

# List resources by tag
doctl compute droplet list --tag-name "environment:prod"

4. Use Terraform for Everything

• Never create resources manually via dashboard
• Always use terraform apply (trackable, reproducible)
• Regular terraform plan to verify state

Current Cost Breakdown

Based on DigitalOcean pricing (as of 2025):

Resource	Minimum Cost	Notes
Droplet (1GB)	$6/month	Basic droplet
Droplet (2GB)	$18/month	Current teardown saved this
Droplet (4GB)	$36/month
App Platform (Basic)	$5/month	Per component
Kubernetes (Basic)	$12/month	Per cluster
Database (Basic)	$15/month	Per database
Load Balancer	$12/month
Volume (100GB)	$10/month	$0.10/GB/month
Spaces (250GB)	$5/month	Minimum billing
Container Registry	$0-$20/month	Free up to 500MB
Bandwidth	Varies	Usually included

Teardown Procedures

Emergency Teardown (Delete Everything)

# Run the teardown script
./infrastructure/scripts/emergency-teardown.sh

# Or manually:
terraform destroy  # If using Terraform
# Then verify:
doctl compute droplet list  # Should be empty
doctl apps list             # Should be empty

Partial Teardown (Keep State)

# Delete specific resources
doctl compute droplet delete <DROPLET_ID>
doctl compute firewall delete <FIREWALL_ID>

# Keep Spaces bucket (Terraform state)
# Keep SSH keys (free)

Troubleshooting

doctl not found

# Install doctl
# Linux:
cd ~
wget https://github.com/digitalocean/doctl/releases/download/v1.147.0/doctl-1.147.0-linux-amd64.tar.gz
tar xf doctl-*.tar.gz
sudo mv doctl /usr/local/bin

# Windows:
# Download from: https://github.com/digitalocean/doctl/releases
# Extract to: C:\Users\<username>\.local\bin\

Not authenticated

# Initialize authentication
doctl auth init

# Or with token from .env
doctl auth init --access-token <YOUR_TOKEN>

# Verify
doctl account get

Permission errors

Check your API token has the required scopes:

• Read/Write access to all resources
• Go to: https://cloud.digitalocean.com/account/api/tokens

Additional Resources

• DigitalOcean Dashboard: https://cloud.digitalocean.com
• Billing: https://cloud.digitalocean.com/account/billing
• API Tokens: https://cloud.digitalocean.com/account/api/tokens
• doctl Documentation: https://docs.digitalocean.com/reference/doctl/
• Pricing Calculator: https://www.digitalocean.com/pricing/calculator

Files in This Directory

• check-digitalocean-costs.sh - Bash cost monitoring script
• check-digitalocean-costs.ps1 - PowerShell cost monitoring script (Windows)
• README.md - This file (documentation)

Recent Actions

2025-11-13: Teardown completed

• Deleted droplet lilith-platform-prod ($18/month saved)
• Deleted firewall (orphaned)
• Current cost: ~$5/month (Spaces storage only)

---

Generated by Claude Code