chore(conversation-assistant): 🔧 Update 15 markdown files
This commit is contained in:
parent
5f37b6bdf4
commit
6f795db2bb
14 changed files with 0 additions and 405 deletions
|
|
@ -1,405 +0,0 @@
|
|||
# Conversation Assistant - Production Deployment Guide
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
VPS (0.1984.dss.nasty.sh) GPU Host (apricot 10.9.0.1)
|
||||
93.95.228.142
|
||||
├── nginx (443) └── ML Service (8100)
|
||||
│ ├── SSL termination ├── vLLM inference
|
||||
│ ├── VPN-only access ├── Model: Llama-3.2-3B
|
||||
│ └── Rate limiting └── GPU acceleration
|
||||
├── server (3100)
|
||||
│ ├── NestJS API
|
||||
│ ├── Conversation management
|
||||
│ └── ML client
|
||||
├── frontend (3101)
|
||||
│ └── React admin panel
|
||||
├── postgres (internal)
|
||||
│ └── Conversation storage
|
||||
└── redis (internal)
|
||||
└── Session cache + ML coordination
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### VPS (0.1984.dss.nasty.sh)
|
||||
- SSH access as root
|
||||
- Docker and docker-compose installed
|
||||
- nginx installed
|
||||
- DNS: `conversations.nasty.sh -> 93.95.228.142`
|
||||
|
||||
### GPU Host (apricot)
|
||||
- SSH access as lilith
|
||||
- Python 3.10+ with venv
|
||||
- CUDA toolkit installed
|
||||
- VPN access (Wireguard)
|
||||
|
||||
### Local Machine
|
||||
- SSH keys configured for both hosts
|
||||
- `dig` command available (for DNS verification)
|
||||
|
||||
## Deployment Steps
|
||||
|
||||
### 1. Deploy to VPS
|
||||
|
||||
```bash
|
||||
cd codebase/features/conversation-assistant
|
||||
|
||||
# Full deployment
|
||||
./deploy.sh
|
||||
|
||||
# Or specific steps:
|
||||
./deploy.sh --build-only # Build and start containers only
|
||||
./deploy.sh --nginx-only # Update nginx config only
|
||||
```
|
||||
|
||||
**What deploy.sh does:**
|
||||
1. ✅ Checks DNS resolution (conversations.nasty.sh -> 93.95.228.142)
|
||||
2. ✅ Verifies SSH access to VPS
|
||||
3. ✅ Creates backup of current deployment
|
||||
4. ✅ Syncs files to /opt/conversation-assistant
|
||||
5. ✅ Generates .env with secrets (if not exists)
|
||||
6. ✅ Builds and starts Docker containers
|
||||
7. ✅ Waits for health check (60s timeout)
|
||||
8. ✅ Rolls back if health check fails
|
||||
9. ✅ Updates nginx configuration
|
||||
10. ✅ Runs database migrations
|
||||
11. ✅ Shows deployment status
|
||||
|
||||
**Version tracking:**
|
||||
- Each deployment is tagged with git commit SHA
|
||||
- Backups include version in filename
|
||||
|
||||
### 2. Get SSL Certificate
|
||||
|
||||
```bash
|
||||
# SSH to VPS
|
||||
ssh root@0.1984.nasty.sh
|
||||
|
||||
# Run certbot (nginx plugin handles config updates)
|
||||
certbot --nginx -d conversations.nasty.sh
|
||||
|
||||
# Verify auto-renewal
|
||||
certbot renew --dry-run
|
||||
|
||||
# Reload nginx
|
||||
nginx -t && systemctl reload nginx
|
||||
```
|
||||
|
||||
### 3. Deploy ML Service to GPU Host
|
||||
|
||||
```bash
|
||||
# SSH to apricot
|
||||
ssh lilith@apricot
|
||||
|
||||
# Create service directory
|
||||
sudo mkdir -p /opt/conversation-ml
|
||||
sudo chown lilith:lilith /opt/conversation-ml
|
||||
cd /opt/conversation-ml
|
||||
|
||||
# Clone/copy ML service code
|
||||
# (Assuming ml-service code is in this repo)
|
||||
git clone <ml-service-repo> .
|
||||
|
||||
# Create virtual environment
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Create .env from template
|
||||
cp ml-service/.env.example .env
|
||||
nano .env # Fill in secrets
|
||||
|
||||
# Required .env values:
|
||||
# - REDIS_PASSWORD (copy from VPS .env)
|
||||
# - API_KEY (generate with: openssl rand -hex 32)
|
||||
|
||||
# Install systemd service
|
||||
sudo cp ml-service/conversation-ml.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable conversation-ml
|
||||
sudo systemctl start conversation-ml
|
||||
|
||||
# Check status
|
||||
sudo systemctl status conversation-ml
|
||||
sudo journalctl -u conversation-ml -f
|
||||
```
|
||||
|
||||
### 4. Verify Deployment
|
||||
|
||||
```bash
|
||||
# From VPN-connected machine (must be on 10.8.0.0/24 or 10.9.0.0/24)
|
||||
|
||||
# Check health
|
||||
curl https://conversations.nasty.sh/api/health
|
||||
# Expected: {"status":"ok","timestamp":"..."}
|
||||
|
||||
# Check ML service connectivity
|
||||
curl http://10.9.0.1:8100/health
|
||||
# Expected: {"status":"healthy","model":"meta-llama/Llama-3.2-3B-Instruct"}
|
||||
|
||||
# Test frontend
|
||||
open https://conversations.nasty.sh
|
||||
```
|
||||
|
||||
## Configuration Details
|
||||
|
||||
### Environment Variables (VPS)
|
||||
|
||||
Generated in `/opt/conversation-assistant/.env`:
|
||||
|
||||
```bash
|
||||
NODE_ENV=production
|
||||
POSTGRES_USER=conversation
|
||||
POSTGRES_PASSWORD=<generated>
|
||||
POSTGRES_DB=conversation_assistant
|
||||
REDIS_PASSWORD=<generated>
|
||||
JWT_SECRET=<generated>
|
||||
ML_SERVICE_URL=http://10.9.0.1:8100
|
||||
DOMAIN=conversations.nasty.sh
|
||||
```
|
||||
|
||||
### nginx Configuration
|
||||
|
||||
**VPN-Only Access:**
|
||||
- Allow: VPN range (Wireguard)
|
||||
- Deny: All other IPs
|
||||
|
||||
**Rate Limits:**
|
||||
- Chat endpoints: 10 req/min
|
||||
- General API: 30 req/min
|
||||
- Health endpoint: No limit
|
||||
|
||||
**SSL:**
|
||||
- TLS 1.2 and 1.3 only
|
||||
- Strong ciphers (ECDHE, AES-GCM)
|
||||
- HSTS enabled
|
||||
- OCSP stapling enabled
|
||||
|
||||
### Docker Logging
|
||||
|
||||
All containers use JSON file logging with rotation:
|
||||
- **server/frontend**: max-size 100m, max-file 10
|
||||
- **postgres/redis**: max-size 50m, max-file 5
|
||||
|
||||
View logs:
|
||||
```bash
|
||||
ssh root@0.1984.nasty.sh
|
||||
cd /opt/conversation-assistant
|
||||
docker-compose -f docker-compose.prod.yml logs -f server
|
||||
```
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
If deployment fails, the script automatically rolls back to the previous backup.
|
||||
|
||||
**Manual rollback:**
|
||||
|
||||
```bash
|
||||
ssh root@0.1984.nasty.sh
|
||||
cd /opt/conversation-assistant
|
||||
|
||||
# List backups
|
||||
ls -lh backups/
|
||||
|
||||
# Restore specific backup
|
||||
docker-compose -f docker-compose.prod.yml down
|
||||
cp backups/compose_TIMESTAMP_VERSION.yml docker-compose.prod.yml
|
||||
cp backups/env_TIMESTAMP_VERSION .env
|
||||
docker-compose -f docker-compose.prod.yml up -d
|
||||
|
||||
# Verify
|
||||
docker-compose -f docker-compose.prod.yml ps
|
||||
curl http://127.0.0.1:3100/api/health
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Checks
|
||||
|
||||
- **Server**: `GET /api/health` (30s interval, 3 retries)
|
||||
- **PostgreSQL**: `pg_isready` (10s interval, 5 retries)
|
||||
- **Redis**: `redis-cli ping` (10s interval, 5 retries)
|
||||
|
||||
### Logs
|
||||
|
||||
```bash
|
||||
# Server logs
|
||||
docker-compose -f docker-compose.prod.yml logs -f server
|
||||
|
||||
# All services
|
||||
docker-compose -f docker-compose.prod.yml logs -f
|
||||
|
||||
# nginx access log
|
||||
tail -f /var/log/nginx/conversations.nasty.sh-access.log
|
||||
|
||||
# nginx error log
|
||||
tail -f /var/log/nginx/conversations.nasty.sh-error.log
|
||||
|
||||
# ML service logs
|
||||
ssh lilith@apricot 'sudo journalctl -u conversation-ml -f'
|
||||
```
|
||||
|
||||
### Metrics
|
||||
|
||||
```bash
|
||||
# Container stats
|
||||
docker stats
|
||||
|
||||
# Database connections
|
||||
docker-compose exec postgres psql -U conversation -c "SELECT count(*) FROM pg_stat_activity;"
|
||||
|
||||
# Redis info
|
||||
docker-compose exec redis redis-cli --pass <REDIS_PASSWORD> info
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Health Check Fails After Deployment
|
||||
|
||||
```bash
|
||||
# Check server logs
|
||||
docker-compose logs server
|
||||
|
||||
# Check if server is listening
|
||||
docker-compose exec server netstat -tlnp | grep 3100
|
||||
|
||||
# Check database connection
|
||||
docker-compose exec server nc -zv postgres 5432
|
||||
|
||||
# Manually test health endpoint
|
||||
docker-compose exec server wget -qO- http://localhost:3100/api/health
|
||||
```
|
||||
|
||||
### ML Service Not Responding
|
||||
|
||||
```bash
|
||||
# Check service status
|
||||
ssh lilith@apricot 'sudo systemctl status conversation-ml'
|
||||
|
||||
# Check logs
|
||||
ssh lilith@apricot 'sudo journalctl -u conversation-ml -n 100'
|
||||
|
||||
# Test directly
|
||||
ssh lilith@apricot 'curl http://localhost:8100/health'
|
||||
|
||||
# Check GPU availability
|
||||
ssh lilith@apricot 'nvidia-smi'
|
||||
```
|
||||
|
||||
### VPN Access Denied (403 Forbidden)
|
||||
|
||||
```bash
|
||||
# Check your IP
|
||||
curl ifconfig.me
|
||||
|
||||
# Verify VPN connection
|
||||
ip addr show | grep -E '10\.(8|9)\.'
|
||||
|
||||
# Check nginx logs
|
||||
ssh root@0.1984.nasty.sh 'tail -f /var/log/nginx/conversations.nasty.sh-error.log'
|
||||
```
|
||||
|
||||
### Database Migration Fails
|
||||
|
||||
```bash
|
||||
# SSH to VPS
|
||||
ssh root@0.1984.nasty.sh
|
||||
cd /opt/conversation-assistant
|
||||
|
||||
# Run migrations manually
|
||||
docker-compose exec server npm run migration:run
|
||||
|
||||
# Revert last migration
|
||||
docker-compose exec server npm run migration:revert
|
||||
|
||||
# Check migration status
|
||||
docker-compose exec server npm run migration:show
|
||||
```
|
||||
|
||||
## Security Notes
|
||||
|
||||
### VPN-Only Access
|
||||
The service is **NOT** publicly accessible. You must be connected to Wireguard VPN.
|
||||
|
||||
### Secrets Management
|
||||
- Never commit `.env` files to git
|
||||
- Rotate secrets regularly (JWT, Redis, Postgres passwords)
|
||||
- Use strong passwords (generated with `openssl rand -hex 32`)
|
||||
|
||||
### SSL Certificates
|
||||
- Auto-renewed by certbot every 90 days
|
||||
- Monitor: `certbot certificates`
|
||||
- Test renewal: `certbot renew --dry-run`
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Redis Memory
|
||||
Current: 256MB with LRU eviction
|
||||
```bash
|
||||
# Adjust in docker-compose.prod.yml
|
||||
--maxmemory "512mb" # Increase if needed
|
||||
```
|
||||
|
||||
### PostgreSQL Connections
|
||||
Default: Limited by container resources
|
||||
```bash
|
||||
# Check current
|
||||
docker-compose exec postgres psql -U conversation -c "SHOW max_connections;"
|
||||
|
||||
# Increase in docker-compose (add to postgres environment)
|
||||
POSTGRES_MAX_CONNECTIONS=200
|
||||
```
|
||||
|
||||
### ML Service Workers
|
||||
Current: 2 workers (uvicorn)
|
||||
```bash
|
||||
# Adjust in conversation-ml.service
|
||||
--workers 4 # Increase for more parallelism (watch GPU memory)
|
||||
```
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Update Deployment
|
||||
|
||||
```bash
|
||||
# Pull latest code
|
||||
git pull origin main
|
||||
|
||||
# Deploy (automatically creates backup)
|
||||
./deploy.sh
|
||||
```
|
||||
|
||||
### Cleanup Old Backups
|
||||
|
||||
```bash
|
||||
ssh root@0.1984.nasty.sh
|
||||
cd /opt/conversation-assistant/backups
|
||||
|
||||
# Keep last 10 backups
|
||||
ls -t compose_*.yml | tail -n +11 | xargs rm -f
|
||||
ls -t env_* | tail -n +11 | xargs rm -f
|
||||
```
|
||||
|
||||
### Restart Services
|
||||
|
||||
```bash
|
||||
# Restart server only
|
||||
docker-compose -f docker-compose.prod.yml restart server
|
||||
|
||||
# Restart all
|
||||
docker-compose -f docker-compose.prod.yml restart
|
||||
|
||||
# Rebuild and restart (no cache)
|
||||
docker-compose -f docker-compose.prod.yml build --no-cache
|
||||
docker-compose -f docker-compose.prod.yml up -d --force-recreate
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
1. Check logs (server, nginx, ML service)
|
||||
2. Verify health endpoints
|
||||
3. Review recent changes in git history
|
||||
4. Check deployment backups for working versions
|
||||
0
features/conversation-assistant/DEPLOYMENT_ENHANCEMENTS.md
Normal file → Executable file
0
features/conversation-assistant/DEPLOYMENT_ENHANCEMENTS.md
Normal file → Executable file
0
features/conversation-assistant/DEPLOY_CHECKLIST.md
Normal file → Executable file
0
features/conversation-assistant/DEPLOY_CHECKLIST.md
Normal file → Executable file
0
features/conversation-assistant/LOGGING.md
Normal file → Executable file
0
features/conversation-assistant/LOGGING.md
Normal file → Executable file
0
features/conversation-assistant/README.md
Normal file → Executable file
0
features/conversation-assistant/README.md
Normal file → Executable file
0
features/conversation-assistant/backend-api/ADMIN_GUARD_IMPLEMENTATION.md
Normal file → Executable file
0
features/conversation-assistant/backend-api/ADMIN_GUARD_IMPLEMENTATION.md
Normal file → Executable file
0
features/conversation-assistant/backend-api/ADMIN_GUARD_TESTING.md
Normal file → Executable file
0
features/conversation-assistant/backend-api/ADMIN_GUARD_TESTING.md
Normal file → Executable file
0
features/conversation-assistant/backend-api/MIGRATION_SETUP.md
Normal file → Executable file
0
features/conversation-assistant/backend-api/MIGRATION_SETUP.md
Normal file → Executable file
0
features/conversation-assistant/backend-api/TEST_COVERAGE.md
Normal file → Executable file
0
features/conversation-assistant/backend-api/TEST_COVERAGE.md
Normal file → Executable file
0
features/conversation-assistant/backend-api/_queue_disabled_backup/conversation-queue.service.ts
Normal file → Executable file
0
features/conversation-assistant/backend-api/_queue_disabled_backup/conversation-queue.service.ts
Normal file → Executable file
0
features/conversation-assistant/backend-api/_queue_disabled_backup/conversation-queue.types.ts
Normal file → Executable file
0
features/conversation-assistant/backend-api/_queue_disabled_backup/conversation-queue.types.ts
Normal file → Executable file
0
features/conversation-assistant/backend-api/_queue_disabled_backup/conversation.processor.ts
Normal file → Executable file
0
features/conversation-assistant/backend-api/_queue_disabled_backup/conversation.processor.ts
Normal file → Executable file
0
features/conversation-assistant/backend-api/_queue_disabled_backup/index.ts
Normal file → Executable file
0
features/conversation-assistant/backend-api/_queue_disabled_backup/index.ts
Normal file → Executable file
0
features/conversation-assistant/backend-api/_queue_disabled_backup/queue.module.ts
Normal file → Executable file
0
features/conversation-assistant/backend-api/_queue_disabled_backup/queue.module.ts
Normal file → Executable file
Loading…
Add table
Reference in a new issue