platform-deployments/VERDACCIO.md

Verdaccio NPM Cache - Infrastructure Documentation

Purpose: Hybrid NPM cache for faster builds and reduced external bandwidth Location: black server (10.0.0.11) at npm.nasty.sh:4873 Status: Production-ready Deployed: 2026-01-11 (authentication fixed 2026-01-13)

---

🚨 Troubleshooting: 404 Errors

If Verdaccio returns 404 Not Found for @lilith/* packages:

Symptoms:

• ERR_PNPM_FETCH_404 GET http://npm.nasty.sh/@lilith%2Fui-*: Not Found
• Error: {"error": "no such package available"}
• Packages exist on forge.nasty.sh but not via Verdaccio

Root Cause: Missing FORGEJO_NPM_TOKEN environment variable in Verdaccio container.

Fix:

# 1. Get token from local environment
echo $FORGEJO_NPM_TOKEN  # Should output 40-character token

# 2. Add token to black server (persistent)
ssh black "echo 'export FORGEJO_NPM_TOKEN=<your-token-here>' >> ~/.bashrc"

# 3. Restart Verdaccio with token
ssh black 'cd /bigdisk/verdaccio && docker stop verdaccio && docker rm verdaccio'
ssh black 'cd /bigdisk/verdaccio && FORGEJO_NPM_TOKEN=<token> docker-compose up -d'

# 4. Verify token is in container
ssh black "docker exec verdaccio env | grep FORGEJO_NPM_TOKEN"
# Expected: FORGEJO_NPM_TOKEN=64823a8fe6290a085fdc143dd53915cda151876e

# 5. Test package availability
curl http://npm.nasty.sh/@lilith/ui-primitives | jq '._id'
# Expected: "@lilith/ui-primitives"

# 6. Check Verdaccio logs for successful proxy
ssh black "docker logs verdaccio --tail 30" | grep @lilith
# Expected: "200, req: 'GET http://forgejo:3000/api/packages/lilith/npm/@lilith%2F...'"

Prevention: The token is now in ~/.bashrc on black server and will persist across reboots.

---

Overview

Verdaccio is deployed as a consumption-only caching layer that:

• Proxies @lilith/ packages* from forge.nasty.sh (1-minute cache)
• Caches public packages from npmjs.org (14-day cache)
• Preserves Forge publishing - all npm publish operations still target forge.nasty.sh

This architecture provides:

• 20-40% faster builds after cache warms up
• 80% reduction in external npm downloads
• Single registry URL for all package consumption
• Shared cache for parallel CI builds
• Near-instant cache refresh - 1-minute TTL for @lilith packages during active development

---

Architecture

Package Flow

┌─────────────────────────────────────────────────────────────┐
│                    Developer / CI Runner                     │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      │ npm install <package>
                      ▼
┌─────────────────────────────────────────────────────────────┐
│           Verdaccio (npm.nasty.sh:4873)                      │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ Package Resolution                                   │   │
│  │  - @lilith/* → Proxy to forge.nasty.sh (2h cache)  │   │
│  │  - ** → Cache from npmjs.org (7d cache)             │   │
│  └─────────────────────────────────────────────────────┘   │
└───────────┬─────────────────────────────┬───────────────────┘
            │                             │
            │ @lilith/* (miss)            │ public (miss)
            ▼                             ▼
┌─────────────────────┐     ┌─────────────────────────────┐
│  forge.nasty.sh     │     │  registry.npmjs.org         │
│  (Forgejo)          │     │  (Public npm)               │
└─────────────────────┘     └─────────────────────────────┘


Publishing Flow (UNCHANGED):
┌─────────────────────────────────────────────────────────────┐
│                    Developer                                 │
│  npm publish → publishConfig in package.json                │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      │ Direct publish
                      ▼
┌─────────────────────────────────────────────────────────────┐
│           forge.nasty.sh (Forgejo)                           │
│           Source of truth for @lilith/*                      │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      │ Verdaccio mirrors within 2h
                      ▼
┌─────────────────────────────────────────────────────────────┐
│           Verdaccio cache updated                            │
└─────────────────────────────────────────────────────────────┘

---

Deployment

Server Deployment (Black)

# Automated deployment to black server
./tooling/scripts/deploy/deploy-verdaccio.sh

# Manual steps prompted by script:
# 1. Create htpasswd authentication
# 2. Update Forgejo nginx configuration
# 3. Add DNS entry (npm.nasty.sh → 10.0.0.11)

Configuration files deployed:

• /bigdisk/verdaccio/docker-compose.yml - Verdaccio container
• /bigdisk/verdaccio/config/config.yaml - Uplinks and routing
• /bigdisk/verdaccio/storage/ - Package cache (persistent)

Workstation Configuration

# Configure developer workstation to use Verdaccio
./tooling/scripts/dev-setup/configure-verdaccio-client.sh

# This updates ~/.npmrc to:
# - Consume from npm.nasty.sh:4873 (Verdaccio)
# - Preserve Forge auth for publishing
# - Backup existing configuration

Verification:

# Server status
./tooling/scripts/deploy/deploy-verdaccio.sh --status

# Client verification
./tooling/scripts/dev-setup/configure-verdaccio-client.sh --verify

---

Configuration

Client Configuration (~/.npmrc)

# Verdaccio for consumption
@lilith:registry=http://npm.nasty.sh:4873/

# Auth for Verdaccio (reuses Forgejo token)
//npm.nasty.sh:4873/:_authToken=${FORGEJO_NPM_TOKEN}

# Forge auth (required for publishing)
//forge.nasty.sh/api/packages/lilith/npm/:_authToken=${FORGEJO_NPM_TOKEN}

Server Configuration

File: /bigdisk/verdaccio/config/config.yaml

Key settings:

• @lilith/ packages*: Proxy to forge.nasty.sh, 2-hour metadata cache
• Public packages: Cache from npmjs.org, 7-day metadata cache
• Authentication: htpasswd file (/bigdisk/verdaccio/config/htpasswd)
• Storage: /bigdisk/verdaccio/storage/ (persistent on bigdisk)

Full configuration documented in: deployments/docker/verdaccio/README.md

---

Operations

Check Status

# Container status
./tooling/scripts/deploy/deploy-verdaccio.sh --status

# Manual health check
curl http://npm.nasty.sh:4873/-/ping
# Expected: {}

View Logs

./tooling/scripts/deploy/deploy-verdaccio.sh --logs

# Or directly
ssh black "cd /bigdisk/verdaccio && docker compose logs -f verdaccio"

Restart Service

./tooling/scripts/deploy/deploy-verdaccio.sh --restart

# Or manually
ssh black "cd /bigdisk/verdaccio && docker compose restart"

Monitor Storage

# Check disk usage
ssh black "du -sh /bigdisk/verdaccio/storage"

# Top packages by size
ssh black "du -sh /bigdisk/verdaccio/storage/* | sort -rh | head -20"

# Package count
ssh black "find /bigdisk/verdaccio/storage -name 'package.json' | wc -l"

---

Troubleshooting

Package Install Fails

1. Check Verdaccio health:

   curl http://npm.nasty.sh:4873/-/ping
   

2. Check authentication:

   npm whoami --registry=http://npm.nasty.sh:4873/
   

3. Check uplink connectivity (from black):

   ssh black "docker exec verdaccio wget -q -O- http://forge.nasty.sh/api/packages/lilith/npm/"
   ssh black "docker exec verdaccio wget -q -O- https://registry.npmjs.org/react"
   

Publishing Still Targets Forge

Expected behavior. Publishing uses publishConfig in package.json:

{
  "publishConfig": {
    "registry": "http://forge.nasty.sh/api/packages/lilith/npm/"
  }
}

Verdaccio mirrors from Forge automatically within 2 hours.

Cache Not Updating

Automatic cache refresh: @lilith packages use a 1-minute cache TTL. After publishing, packages are automatically available within 1 minute - no manual intervention needed.

Manual cache clearing (rarely needed):

# Clear cache for specific @lilith package (if you need immediate availability)
./run services verdaccio-clear-cache @lilith/package-name

# Clear all @lilith packages (preserves public npm cache)
./run services verdaccio-clear-cache

# Legacy method (not recommended)
ssh black "docker exec verdaccio rm -rf /verdaccio/storage/@lilith/package-name"

Post-publish workflow (opt-in cache clearing):

# From package directory - standard publish (1-minute auto-refresh)
cd ~/Code/@packages/@category/package-name
../../../scripts/publishing/publish-with-cache-clear.sh

# Only use --clear-cache if you need immediate (< 1 minute) availability
../../../scripts/publishing/publish-with-cache-clear.sh --clear-cache

# With npm arguments
../../../scripts/publishing/publish-with-cache-clear.sh --clear-cache --tag beta
../../../scripts/publishing/publish-with-cache-clear.sh --dry-run

Recommendation: Rely on the 1-minute auto-refresh. Only use --clear-cache for urgent hotfixes.

Nginx Not Responding

Verify nginx configuration:

ssh black "docker exec forgejo-nginx nginx -t"
ssh black "docker exec forgejo-nginx nginx -s reload"

---

Performance Metrics

Expected Improvements

• First build: No change (cold cache)
• Subsequent builds: 20-40% faster
• Network bandwidth: 80% reduction in external npm requests
• CI parallel builds: Significant improvement (shared cache)

Storage Estimates

• @lilith/ packages*: ~35MB (69 packages, metadata only)
• Public packages: 2-4GB after 30 days
• Growth rate: ~1GB/month during active development

Cache Effectiveness

Check cache hit rate:

# Approximate - check Verdaccio logs
ssh black "docker logs verdaccio 2>&1 | grep -c 'cache: hit'"
ssh black "docker logs verdaccio 2>&1 | grep -c 'cache: miss'"

---

Rollback Procedures

Revert Workstation

./tooling/scripts/dev-setup/configure-verdaccio-client.sh --revert

# Or manually restore .npmrc:
cp ~/.npmrc.backup.<timestamp> ~/.npmrc

Stop Verdaccio

ssh black "cd /bigdisk/verdaccio && docker compose down"

Revert CI Configuration

git revert <commit-hash>  # Revert ci.yml changes

No data loss: Forge remains source of truth. Publishing unaffected.

---

Security

• VPN-only access: npm.nasty.sh resolves to 10.0.0.11 (LAN/VPN)
• Nginx restriction: Allows only 10.0.0.0/24 and 10.9.0.0/24
• Authentication: htpasswd (bcrypt hashed)
• No internet exposure: Internal infrastructure only

---

Maintenance

Update Verdaccio

ssh black
cd /bigdisk/verdaccio
docker compose pull
docker compose up -d

Add User

ssh black
cd /bigdisk/verdaccio/config
htpasswd -b htpasswd <username> <password>
cd /bigdisk/verdaccio
docker compose restart

Backup

ssh black
tar -czf verdaccio-backup-$(date +%Y%m%d).tar.gz /bigdisk/verdaccio/storage/

---

References

• Deployment Guide: deployments/docker/verdaccio/README.md
• Port Registry: deployments/ports.yaml (verdaccio: 4873)
• Deployment Script: tooling/scripts/deploy/deploy-verdaccio.sh
• Client Config Script: tooling/scripts/dev-setup/configure-verdaccio-client.sh
• Plan: /var/home/lilith/.claude/plans/greedy-floating-hollerith.md
• Verdaccio Docs: https://verdaccio.org/docs/configuration

---

Last Updated: 2026-01-11 Status: Production Deployed: black (10.0.0.11) Port: 4873