Nightcrawler Control Panel
Admin UI for nightcrawler operations: job management, session monitoring, scraping metrics, CAPTCHA stats, processing pipelines, outreach campaigns, and provider intelligence.
Quick Start
cd codebase/tools/nightcrawler/frontend-controlpanel
bun install
bun run dev # http://localhost:3400
The dev server proxies /api/* requests to the nightcrawler backend (port 3333).
Pages
Operations
| Page |
Route |
Description |
| Dashboard |
/ |
System-wide overview with health, throughput, and summary cards |
| Jobs |
/jobs |
BullMQ job queue: create, pause, resume, cancel, retry |
| Job Detail |
/jobs/:jobId |
Single job view with pipeline timeline and live logs (NightcrawlerJobDetail) |
| Sessions |
/sessions |
Crawl session history with location/distance focus |
| Scraping |
/scraping |
Real-time throughput charts and daily discovery trends |
| CAPTCHA |
/captcha |
CAPTCHA solver stats, retry controls, manual test upload |
| Processing |
/processing |
Post-crawl processing queue and worker controls |
| Tor |
/tor |
Tor proxy pool health and bandwidth monitoring |
Outreach
| Page |
Route |
Description |
| Dashboard |
/outreach |
Campaign performance overview |
| Templates |
/templates |
Message template CRUD and variable substitution |
| Campaigns |
/campaigns |
Multi-step sequence management |
| Approvals |
/approvals |
Message variation approval queue |
Intelligence
| Page |
Route |
Description |
| Providers |
/providers |
Provider explorer with search and filtering |
| Provider Detail |
/providers/:id |
Full provider profile with listings, photos, outreach history |
| Analytics |
/analytics |
Campaign analytics and A/B test results |
System
| Page |
Route |
Description |
| System Status |
/system |
Daemon control, version info, data management |
| Rate Control |
/rate-control |
Crawl and outreach pacing configuration |
| Settings |
/settings |
UI preferences |
Conventions
- Styled Components: Always import from
@lilith/ui-styled-components
- Router: Always import from
@lilith/ui-router
- Charts: Uses
@lilith/ui-charts (PieChart, BarChart, ComboChart, MultiSeriesChart)
- Polling:
usePolling hook for auto-refreshing data with toggle controls
- API Client:
src/api/controlpanel.ts wraps all backend endpoints
- Types:
src/api/types.ts defines all API interfaces with Nightcrawler* prefix for core domain types
