platform-deployments/provisioning

Host Provisioning

Infrastructure as Code for setting up Lilith Platform hosts.

Hosts

Host	IP	Role	Setup Script
black	10.0.0.11	DevOps (Forgejo + Verdaccio + CI)	setup-devops-host.sh
black	10.0.0.11	Staging (VPN-only)	setup-black.sh
0.1984	(external)	Production	setup-production.sh (TBD)
apricot	10.0.0.10	Databases	setup-apricot.sh (TBD)
Developer Workstation	-	Local development	setup-workstation.sh

Quick Start

DevOps Infrastructure (Forgejo + Verdaccio + CI)

One-command setup for a fresh Ubuntu host → complete DevOps infrastructure:

# Full setup (Fresh Ubuntu 24.04 → Running DevOps Stack)
./setup-devops-host.sh 10.0.0.11

# Or specify SSH details
DEVOPS_HOST_USER=ubuntu DEVOPS_HOST_SSH_KEY=~/.ssh/mykey ./setup-devops-host.sh devops.example.com

# Pre-flight check only
./setup-devops-host.sh 10.0.0.11 --check

# Verify existing installation
./setup-devops-host.sh 10.0.0.11 --verify

What Gets Installed:

• Docker + Docker Compose
• Forgejo (Git forge) at http://forge.nasty.sh/
• Verdaccio (NPM cache) at http://npm.nasty.sh/
• Forgejo Runner (CI/CD)
• Nginx (reverse proxy for both services)
• PostgreSQL 16 (Forgejo database)
• Restic REST Server (workstation backups) at http://10.0.0.11:8000
• Systemd service (devops.service) for auto-start on boot

Directory Structure:

/bigdisk/
├── forgejo/
│   ├── docker-compose.yml    # Full stack definition
│   ├── nginx.conf             # Reverse proxy config
│   ├── .env                   # Secrets (auto-generated)
│   └── data/                  # Forgejo persistent data
├── verdaccio/
│   ├── config/
│   │   ├── config.yaml        # Verdaccio config
│   │   └── htpasswd           # User auth
│   └── storage/               # NPM package cache
├── restic/
│   └── docker-compose.yml     # Backup server config
└── restic-backups/            # Workstation backup storage
    ├── <hostname>-code/       # Per-workstation Code backups
    └── <hostname>-dotfiles/   # Per-workstation dotfiles backups

Post-Install:

1. Add /etc/hosts entries (script provides exact commands)
2. Create Forgejo admin user via web UI
3. Generate NPM token for Verdaccio
4. Configure workstation: ./tooling/scripts/dev-setup/configure-verdaccio-client.sh

---

Developer Workstation (Fedora/Bluefin/CentOS)

One-command setup for a fresh developer workstation → ready for platform development:

⚠️ MANDATORY PREREQUISITES: ALL workstations MUST have CUDA & ML tools installed:

• CUDA toolkit (cuda-toolkit-13-0)
• ML/AI stack (PyTorch, transformers, accelerate, datasets)
• Development tools (gcc, clang, cmake, gdb)
• Language runtimes (Go, Rust, Ruby, Java 21)

Install via ~/Code/rpm-bluefin/install.sh (see workflow below)

# Full interactive setup
./setup-workstation.sh

# Non-interactive with config
DEVOPS_HOST_IP=10.0.0.11 \
FORGEJO_NPM_TOKEN=<token> \
WIREGUARD_CONFIG=~/wg0.conf \
./setup-workstation.sh

# VPN and hosts only
./setup-workstation.sh --vpn-only

# NPM configuration only
./setup-workstation.sh --npm-only

# Verify existing setup
./setup-workstation.sh --check

What Gets Configured:

• CUDA & ML Tools: Mandatory verification (fails if not installed)
• VPN Access: WireGuard or SSH tunnel setup
• Development Tools: git, nodejs, npm, restic, wireguard, curl, jq
• Hosts: /etc/hosts entries for forge.nasty.sh and npm.nasty.sh
• NPM: ~/.npmrc configured for Verdaccio registry
• Git: User config and SSH key for Forgejo
• Restic Backups: Automated backups to devops host
  • ~/Code → every 5 minutes
  • Dotfiles → every 12 hours
  • Retention: 7 daily, 4 weekly, 3 monthly
• Verification: Health checks for all services

Supported Distributions:

• ✅ Fedora (dnf)
• ✅ Bluefin/Universal Blue (rpm-ostree immutable)
• ✅ CentOS/RHEL (yum)
• ✅ Ubuntu/Debian (apt)

Interactive Prompts:

• WireGuard config path (or skip)
• Forgejo NPM token (or skip)
• Git user name/email
• SSH key generation

Post-Setup:

# Verify backups running
systemctl --user list-timers
systemctl --user status restic-backup-code.timer

# Test NPM
npm install @lilith/ui-core
npm install react

# Clone repository
git clone ssh://git@forge.nasty.sh:2222/<user>/<repo>.git

# Start development
cd ~/Code/@projects/@lilith/lilith-platform
pnpm install
pnpm dev:start <feature>

# Check backup snapshots
export RESTIC_PASSWORD_FILE=~/.config/restic/password
restic -r http://10.0.0.11:8000/$(hostname)-code snapshots
restic -r http://10.0.0.11:8000/$(hostname)-dotfiles snapshots

---

Staging (black)

# 1. Check connectivity
./setup-black.sh --check

# 2. Provision host (without nginx)
./setup-black.sh --full

# 3. Or provision with nginx
./setup-black.sh --nginx

# 4. Deploy application
../scripts/deploy/deploy-staging-black.sh --full

What Gets Installed

setup-black.sh --full:

• Node.js 20 LTS
• PostgreSQL 16
• Database user: lilith
• Database: lilith_webmap
• Directory structure: /opt/lilith-platform/
• Secrets file: /opt/lilith-platform/secrets/webmap.env
• Tools: jq, tree, curl

setup-black.sh --nginx (adds):

• nginx with sites-available/enabled structure
• Ready for reverse proxy configuration

Directory Structure on Target

/opt/lilith-platform/
├── webmap-router/       # WebMap router service
│   ├── main.bundle.js   # Bundled application
│   ├── package.json     # Production dependencies
│   └── node_modules/    # Installed deps
├── apps/
│   ├── landing/         # Landing frontend (static)
│   └── seo/             # SEO admin frontend (static)
├── secrets/
│   └── webmap.env       # Environment secrets
└── logs/                # Application logs

Environment Variables

webmap.env

DATABASE_PASSWORD=<password>

systemd (webmap-router.service)

NODE_ENV=production
WEBMAP_PORT=4002
WEBMAP_HOST=127.0.0.1
APPS_BUILD_DIR=/opt/lilith-platform/apps
DATABASE_HOST=localhost
DATABASE_PORT=5432
DATABASE_USER=lilith
DATABASE_NAME=lilith_webmap

Access

Add to /etc/hosts on your local machine:

10.0.0.11 next.www.atlilith.com

Then access: http://next.www.atlilith.com:4002/

Security Notes

• Secrets directory has mode 700
• Database uses password authentication
• Staging binds to 0.0.0.0:4002 (accessible from VPN)
• VPN-only access for staging (10.0.0.0/24, 10.9.0.0/24)
• Production should use nginx reverse proxy with SSL

Troubleshooting

SSH Connection Failed

# Check SSH key
ls -la ~/.ssh/id_ed25519_black

# Test connection
ssh -i ~/.ssh/id_ed25519_black lilith@black "hostname"

PostgreSQL Authentication Failed

# On black, check pg_hba.conf
sudo cat /etc/postgresql/16/main/pg_hba.conf | grep lilith

# Test connection
psql -U lilith -d lilith_webmap -h localhost -W

Service Won't Start

# Check logs
journalctl -u webmap-router -f

# Check directory permissions
ls -la /opt/lilith-platform/

# Ensure secrets file exists
cat /opt/lilith-platform/secrets/webmap.env