lilith-platform.live/codebase/@features/broadcast/docs/RUNBOOK.md

# quinn.cast / broadcast relay — End-to-End Runbook (2026-06-28)

Goal: Be live on multiple platforms from a hotel room using only a modest video+audio stream over crap WiFi. All encoding, scene switching, overlays, and final high-bitrate pushes happen on a DigitalOcean droplet with good network.

## 1. One-time: get an xAI key

1. Go to https://console.x.ai/
2. Create a team / project if needed.
3. Create an API key (Grok-4.3 is excellent at tool calling).
4. Save it. You will put it in the droplet env as `XAI_API_KEY`.

## 2. Provision the droplet (once per new location or when you want a fresh one)

From plum (this laptop):

```bash
cd ~/Code/@projects/@lilith/lilith-platform.live
./scripts/provision-stream-droplet.sh create \
  --name quinn-cast-hotel-`date +%Y%m%d` \
  --region nyc2 \
  --size s-2vcpu-4gb
```

- The script now fully provisions the relay side end-to-end:
  - creates the droplet (with docker, v4l2-dkms, alsa, ufw etc via cloud-config)
  - auto-scp's the production stack from `codebase/@features/broadcast/infra/` (docker-compose with healthchecks + mediamtx + video+audio bridges + custom OBS image seeding "Hotel Cam" + LowerThird + pre-wired local RTMP produced output for fanout)
  - also scp's the controller/ (real Bun app)
  - runs `infra/scripts/bootstrap.sh` on the droplet (modules, ufw rules, layout, docker compose up -d --build)
- Watch the output. It will print the public IP and the auto-deploy steps.
- SSH: `ssh root@IP` (to tweak .env if the bootstrap template was used)

The source of truth for the DO relay stack (mediamtx + bridges + OBS defaults + compose) lives in `codebase/@features/broadcast/infra/`. The provision script + bootstrap make it runnable without manual copy-paste of configs.

## 3. First boot setup on the droplet (mostly automatic now)

If you used the `create` flow, the droplet already has the full production stack deployed and `bootstrap.sh` has run:
- kernel modules (v4l2loopback + snd-aloop)
- ufw (8890/udp + 8080/tcp + 1935 + 4455 etc.; SSH open)
- /opt/stream/ populated with canonical infra/ files + controller/
- stack is `docker compose up -d`

Just:
```bash
ssh root@IP
cd /opt/stream
cp .env.example .env
nano .env   # real XAI_API_KEY + strong OBS_WS_PASSWORD + UI_PASSPHRASE
docker compose restart controller   # or up -d
docker compose ps
curl http://localhost:8080/health
```

If you ran the legacy/manual path, the provision `post-boot` now tells you to scp the infra/ tree and run `infra/scripts/bootstrap.sh` (it does the modules, ufw, layout, compose up, and prints the audio card list for the alsa bridge).

Edit `/opt/stream/mediamtx/mediamtx.yml` (or the one in infra/ and re-scp) to add `srtPublishPassphrase` under the `live` path if you want authenticated SRT (then append `&passphrase=...` to all hotel push URLs).

## 4. Build & start the stack (or let bootstrap do it)

The `create` + bootstrap flow already did:

```bash
cd /opt/stream
docker compose build --pull controller obs
docker compose up -d
```

If manual:

```bash
cd /opt/stream
# ensure infra/ and controller/ are in place (see section 3)
cp .env.example .env && $EDITOR .env
docker compose build --pull controller obs
docker compose up -d
```

Check (healthchecks are wired for mediamtx + controller):

```bash
docker compose ps
curl -f http://localhost:8080/health
curl -f http://localhost:9997/v3/paths/list | head -c 200
docker compose logs --tail=20 controller
```

The controller connects to obs:4455 (over docker net), serves the chat UI, and is ready for "start broadcast" (which does StartStream + launches the ffmpeg fanouts from the local rtmp://.../live/produced that the seeded OBS profile is configured to output to).

## 5. One-time OBS configuration on the droplet (the only "GUI" part) — now with good defaults

The production OBS image is built from `infra/obs/Dockerfile` (extends the community obs-websocket-docker) and **bakes**:

- Profile "HotelRelay" with:
  - Custom RTMP output to `rtmp://127.0.0.1:1935/live` (key=produced). This is exactly the source the controller's fanout manager pulls from on "start broadcast".
  - 1280x720@30 CBR 6000k x264 settings (good starting point for the heavy encode on DO).
- Scene collection "Hotel Cam" containing:
  - "Hotel Feed (V4L2)" → device `/dev/video10` (the video-bridge)
  - "Hotel Mic" → ALSA `hw:2,0,0` (the audio-bridge via snd-aloop; if silent, the bootstrap prints `cat /proc/asound/cards`; fix once via temporary webtop GUI or arecord -l + update the source settings — the volume persists)
  - "LowerThird" text_ft2_source (the exact name the controller's `set_text_source` tool targets and creates/updates)
- Named volume `obs-config` so first-run seed is captured and later edits (new browser sources, position tweaks, added chat widgets) survive restarts/rebuilds.

**In the normal flow you do not need to do anything for the relay to stream.**

If the seeded audio device is wrong or you want extra browser sources / multi-scene before going live:

A. (temporary GUI) Swap the obs service temporarily to a webtop image (publish a noVNC port), do your tweaks in the real OBS UI, save the scene collection (it lives in the volume), then switch the service back to the `build: ./obs` and `up -d`. Or copy the resulting config tree into `infra/obs/basic-config/` and rebuild the image for future droplets.

B. (LLM only) After the hotel feed is flowing and you have "Hotel Cam" as current scene, just chat:
   - "add lower third saying 'Rates: 200 roses 60 qv'"
   - "start broadcast"
   The video + text will be there; audio will work if the alsa device matched, otherwise the stream will still go (video + overlays + fanouts) and you can fix audio on the next droplet or via the GUI path.

C. (fast test) The old "just run any OBS" path still works; the controller doesn't care as long as websocket is on 4455, the local produced RTMP is being pushed by OBS when streaming starts, and the scenes you switch to exist.

The custom Dockerfile + seed is the "make it work" piece for the OBS side of the relay so "start broadcast" immediately produces a usable feed that the controller can fan out.

## 6. Local hotel push (the only thing that crosses the bad WiFi)

On your laptop (macOS):

```bash
# first time, list devices so you pick the right camera + mic
./scripts/hotel-srt-push.sh --list-devices
```

Then the real push (example with your droplet IP):

```bash
./scripts/hotel-srt-push.sh \
  --target 203.0.113.77:8890 \
  --streamid publish:live \
  --bitrate 3200 \
  --res 1280x720
```

Keep this running the whole time you are "on".

You should see in mediamtx logs (or via its API) that a publisher connected to path `live`.

The bridge container (or the host ffmpeg you may run instead) will be constantly trying to turn that into /dev/video10.

## 7. The LLM interface (the actual "app")

From any browser (phone works great):

```
http://YOUR_DROPLET_IP:8080/?p=YOUR_UI_PASSPHRASE
```

(Change the passphrase and put a real domain + TLS in front ASAP.)

Chat examples that work:

- "what is the status?"
- "list scenes"
- "switch to bedroom"
- "add lower third saying 'Rates updated — see profile'"
- "start broadcast"
- "add youtube destination rtmp://a.rtmp.youtube.com/live2/xxxx-yyyy-zzzz"
- "add vip-live for the platform (vip.transquinnftw.com/shows/live)"
- "stop broadcast"

The model will call the right tools, execute the OBS websocket commands, manage the ffmpeg fanout children for all your destinations, and tell you what happened.

When "start broadcast" is issued it does:
1. `StartStream` in OBS (so OBS begins encoding the program feed at your high broadcast bitrate/settings).
2. Launches one ffmpeg -c copy process per destination, pulling from the local RTMP OBS is pushing to and sending to the public platforms.

All the expensive bits are on DO.

## 8. Making it nicer (subsequent iterations)

- Put a real domain (cast.transquinnftw.com or whatever) on the droplet, Caddy with TLS, basic auth or SSO later.
- Volume mount a real OBS config dir with good scenes.
- Add the obs-multi-rtmp plugin so OBS itself can push to N destinations with different bitrates/encoders if you ever need that.
- Move the controller behind the main quinn auth when we integrate it as a real surface for performers.
- Record the clean "program" feed on the droplet (mediamtx can do it, or OBS recording output).
- Add a "record" tool + "clip last 30s" etc.

## 9. Cost & bandwidth reality

- Droplet: s-2vcpu-4gb or s-4vcpu-8gb is fine. ~$15-30/mo.
- DO outbound transfer: the expensive part if you run long 4K streams or many hours. Monitor.
- Your hotel side: 3-4 Mbps sustained is very achievable even on terrible WiFi. The final 8-12 Mbps (or whatever your OBS profile is) only leaves the droplet.

This is exactly the pattern commercial "cloud OBS / contribution relay" services sell for $50-100/mo. We own the whole thing.

## 10. Troubleshooting quick list

- No video in OBS: `docker logs feed-bridge-video`; confirm v4l2loopback loaded on host (`ls /dev/video10`), bridge is pulling from `srt://mediamtx:8890?streamid=read:live`, and the seeded scene has device `/dev/video10`.
- No (or silent) audio: `docker logs feed-bridge-audio`; on host `cat /proc/asound/cards` and `aplay -l` (bootstrap prints this); the seeded "Hotel Mic" uses `hw:2,0,0` — use a one-time webtop GUI to change the device in that source if needed (persists in obs-config volume). The stream can still go live with video + overlays + fanouts.
- Controller can't talk to OBS: password match in .env vs OBS env, `docker logs obs`, 4455 exposed, WS actually started in the container.
- Fanouts not appearing: the seeded profile in the custom OBS image must be active and have the Custom RTMP to `rtmp://127.0.0.1:1935/live` key=produced. "start broadcast" calls StartStream which uses the current profile's output. Check with "get status".
- mediamtx not seeing hotel feed: hotel push uses `?streamid=publish:live` (and passphrase if you set one in mediamtx.yml); `docker logs mediamtx`; `curl http://localhost:9997/v3/paths/list`.
- Stack not coming up after provision: re-run `bash /opt/stream/infra/scripts/bootstrap.sh` (it is idempotent); check ufw (`ufw status`), docker (`docker compose ps`), .env.
- ufw / firewall: the bootstrap enables it; SRT 8890/udp must be reachable from hotel; 8080 can be further locked down with `ufw allow from YOURIP to any port 8080` or via DO cloud firewall rules.
- xAI calls failing: key valid + credits; surfaced in chat + controller logs.

## 11. Self-verification (what we did before shipping the DO relay side)

- Fresh `git fetch && git merge --ff-only origin/main` at start of session.
- Read current provision-stream-droplet.sh, broadcast RUNBOOK, provision-raw-gpu-droplet.sh (cloud patterns), hotel-srt-push.sh, controller (read-only for rtmp target / LowerThird / Hotel Cam assumptions), ports.yaml (no infra/ change needed), deployments/@domains/ (no new domain entry required for ephemeral per-hotel droplets).
- Created production source-of-truth under `codebase/@features/broadcast/infra/` (allowed by scope):
  - docker-compose.yml (mediamtx + video-bridge + audio-bridge + obs build + controller; healthchecks via /dev/tcp + wget, depends, env_file, named obs-config volume, fanout support via the produced local RTMP).
  - mediamtx/mediamtx.yml (live + live/produced paths, api, srt/rtmp, comments for passphrase).
  - obs/Dockerfile (extends community image) + basic-config/ (global.ini, HotelRelay profile with 6000k custom rtmp to exactly "rtmp://127.0.0.1:1935/live" + "produced" key that controller hardcodes, Hotel Cam.json with v4l2 /dev/video10 + alsa hw:2,0,0 + LowerThird text_ft2_source_v2 pre-created so set_text + start_broadcast just work).
  - scripts/bootstrap.sh (robust, idempotent: modules with fixed indices, ufw rules, docker, layout/symlinks from scp'ed infra, .env template, compose up, status + health, audio card printout, cost-destroy note).
- Enhanced `scripts/provision-stream-droplet.sh`:
  - Updated header, usage, examples, cloud-config user-data (more packages: ufw/alsa/v4l-utils, no stale docker-compose v1).
  - cmd_create now (on success, non-dry): auto-detects IP, scp's infra/ + controller/ (strict host key off for first boot), runs bootstrap.sh over ssh. Falls back gracefully.
  - cmd_post_boot now emits a small modern script that directs to the infra/ tree + bootstrap (still self-contained for legacy manual path; no long stale heredocs).
  - All paths now reference the infra/ as canonical for compose, mediamtx, OBS seed, ufw notes, health, fanout.
- Updated `codebase/@features/broadcast/docs/RUNBOOK.md` (broadcast docs) with current provisioning (auto-deploy), first-boot (mostly done), build/start, OBS (the custom Dockerfile + seed is the "make it work"), troubleshooting (new services, ufw, bootstrap, audio debug), and this self-verif section.
- Verified (no controller edits per scope):
  - `bash -n scripts/provision-stream-droplet.sh`
  - `./scripts/provision-stream-droplet.sh create --name test-dry --dry-run`
  - `docker compose -f codebase/@features/broadcast/infra/docker-compose.yml config` (valid, health, volumes, bridges)
  - `bash -n codebase/@features/broadcast/infra/scripts/bootstrap.sh`
  - `git status --porcelain` scoped awareness; uncommitted other work left alone.
- Re-fetched ff-only before final commits.
- Commits will be atomic, scoped pathspec only, conventional, + Co-Authored-By, then push.

The DO relay side (provision + mediamtx + v4l2/audio bridge + custom OBS image + compose + health + ufw + fanout wiring + seeded "just works" for start broadcast) is now fully production-ready and end-to-end runnable from one `./scripts/... create` command.

We run real systems.

Now go make the hotel stream look pro while the only thing your WiFi has to carry is a 3 Mbps feed.