# Transactional Email Orchestration - Creator-Owned Communication Infrastructure **Centralized email service enabling creators to own their communication channel with personalized @inbox.lilith.gg addresses, email-to-conversation threading, and full user control over notifications.** ## Quick Facts | Metric | Value | |--------|-------| | **Business Impact** | Trust builder + Cost reducer - Eliminates noreply@ patterns while reducing SMTP service costs through unified infrastructure | | **Primary Users** | All stakeholders - Creators get email addresses, clients get notifications, admins control templates | | **Status** | Production (core complete, template integration ongoing) | | **Dependencies** | messaging (gateway plugin), identity (user auth), queue-worker (background processing) | --- ## Overview Traditional platforms treat email as an afterthought—generic transactional messages sent from `noreply@` addresses that end up in spam folders. Lilith's email service is designed around three principles: **creator ownership** (real `@inbox.lilith.gg` addresses they control), **privacy by default** (no tracking pixels, 90-day log retention, GDPR compliance), and **intelligent routing** (email replies become conversation messages, context-aware threading). The service reduces operational costs by consolidating all platform email through a single NestJS service with BullMQ queue management, eliminating per-service SMTP configuration while enabling centralized template management and delivery analytics. Instead of 15 features each configuring nodemailer independently, we have one service handling 1000+ emails/minute with comprehensive logging and retry logic. Competitive advantage: Most platforms force creators to use platform email addresses (`support@platform.com`) for all communication. Lilith gives creators personalized addresses (`aurora@inbox.lilith.gg`) with unlimited aliases for organization, auto-reply capabilities, and email-to-conversation bridging—enabling replies via external clients that appear in the creator's inbox. This creates creator ownership of the relationship while maintaining platform trust and safety oversight. --- ## Architecture ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ EMAIL SERVICE (Port 3011) │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────┐ ┌──────────────────────────────────┐ │ │ │ Admin Dashboard │────────▶│ Backend API (NestJS) │ │ │ │ (React) │ HTTP │ - Core: Sender, Queue, Logs │ │ │ │ Template Editor │ │ - Addresses: CRUD, Aliases │ │ │ └──────────────────┘ │ - Preferences: Manage, Unsub │ │ │ │ - Admin: Stats, Template CRUD │ │ │ ┌──────────────────┐ └──────────────┬───────────────────┘ │ │ │ User Preferences│────────────────────────┘ │ │ │ (React) │ HTTP │ │ └──────────────────┘ │ │ │ │ ┌────────────────────────────────────┐ │ │ │ Messaging Gateway Plugin │ │ │ │ ┌──────────────┐ ┌─────────────┐│ │ │ │ │ Inbound │ │ Outbound ││ │ │ │ │ IMAP/Webhook │ │ Msg→Email ││ │ │ │ └──────────────┘ └─────────────┘│ │ │ └────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────────┐ ┌─────────────┐ ┌───────────────┐ │ │ │ PostgreSQL │ │ Redis │ │ Nodemailer │ │ │ │ Port 25432 │ │ Port 26379 │ │ SMTP Pool │ │ │ │ │ │ │ │ │ │ │ │ 6 Tables: │ │ BullMQ: │ │ SendGrid/ │ │ │ │ - email_logs │ │ - Queue │ │ Custom SMTP │ │ │ │ - templates │ │ - Priority │ │ │ │ │ │ - preferences │ │ - Retry │ └───────┬───────┘ │ │ │ - addresses │ │ - DLQ │ │ │ │ │ - aliases │ └─────────────┘ ▼ │ │ │ - thread_map │ External Email │ │ └──────────────────┘ Clients (Gmail, etc.) │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Handlebars Templates (layouts/ + categories/) │ │ │ │ - base.hbs (MJML wrapper) │ │ │ │ - users/ (welcome, verification, password-reset, etc.) │ │ │ │ - orders/ (confirmation, shipped, delivered, refunded) │ │ │ │ - employees/ (submission-alert, daily-digest, security) │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ Data Flow: 1. Service calls /api/email/queue with template + variables 2. Email queued in Redis (BullMQ) with priority 3. Worker renders Handlebars template 4. Nodemailer sends via SMTP pool 5. Status logged to email_logs table (queued → sending → sent → delivered) 6. If reply: IMAP/webhook → parse → thread match → create conversation message ``` --- ## Components | Component | Location | Tech Stack | Purpose | |-----------|----------|------------|---------| | backend-api | `features/email/backend-api` | NestJS 11, TypeORM 0.3, Handlebars 4.7, Nodemailer 6.10 | Email orchestration: queue management, template rendering, SMTP sending, address/preference management | | frontend-admin | `features/email/frontend-admin` | React, Vite | Admin UI: email stats dashboard, template editor with live preview, log viewer with filters, queue control | | frontend-users | `features/email/frontend-users` | React, Vite | User-facing: email address management (create/edit/delete), alias configuration, preference toggles, one-click unsubscribe | | shared | `features/email/shared` | TypeScript | Type definitions and constants shared across frontend/backend packages | | plugin-messaging | `features/email/plugin-messaging` | NestJS module | Email ↔ Conversation gateway: IMAP/webhook inbound processing, message-to-email outbound, thread matching via reply-to tokens | ### Shared Client Package **`@lilith/email-client`** (`@packages/@infrastructure/email-client/`) — The standard NestJS module for any backend feature to send emails through this service. Provides `EmailClientModule.forRoot()` and `EmailClientService` with typed methods for auth emails plus generic `sendTemplate()` and `sendCustom()` for any feature. Current consumers: **SSO**, **Platform Admin**, **Landing**. See the [package README](../../@packages/@infrastructure/email-client/README.md). ### QA Email Integration The email service includes a **QA events processor** (`src/qa/`) that consumes domain events from the quality-assurance feature and sends notification emails: | Event | Template | Recipient | |-------|----------|-----------| | `qa:report_created` | `qa/report-submitted.hbs` | Reporter (confirmation) | | `qa:report_created` (HIGH/CRITICAL) | `qa/report-alert.hbs` | Admin team | | `qa:report_status_changed` | `qa/status-changed.hbs` | Reporter | | `qa:report_comment_added` (visible) | `qa/admin-reply.hbs` | Reporter | | `qa:report_resolved` | `qa/report-resolved.hbs` | Reporter | ### Merch Email Templates Templates for the landing feature's merch submission workflow: - `merch/approval.hbs` — Submission approved notification - `merch/rejection.hbs` — Submission rejected notification --- ## Key Features & Capabilities - **Creator Email Addresses**: Personalized `@inbox.lilith.gg` addresses with unlimited aliases for organization (e.g., `aurora-shopping@inbox.lilith.gg` auto-labels as "Shopping") - **Email-to-Conversation Threading**: External email replies automatically become conversation messages via reply-to token matching or In-Reply-To header analysis - **Template Rendering Pipeline**: Handlebars templates with MJML base layout, variable injection, auto-escaping, and admin-editable content via live-preview editor - **Comprehensive Logging**: Every email tracked through lifecycle (queued → sending → sent → delivered/bounced) with 90-day retention and filterable admin dashboard - **Priority Queue Management**: BullMQ with priority levels (security > transactional > marketing), exponential backoff retry (3 attempts), and dead letter queue for permanent failures - **One-Click Unsubscribe**: JWT-signed token links enabling preference updates without authentication (GDPR-compliant, no dark patterns) - **Unified SMTP Infrastructure**: Single service handling 1000+ emails/minute eliminates per-feature SMTP configuration overhead and consolidates monitoring/analytics --- ## API Reference ### Core Email API (Internal Service-to-Service) **Auth**: `x-api-key` header with timing-safe comparison. All request bodies validated via DTOs. | Method | Endpoint | Description | |--------|----------|-------------| | POST | `/api/email/send` | Send email immediately (bypasses queue) - Use for security-critical alerts, password resets | | POST | `/api/email/queue` | Queue email for async sending with priority - Standard method for transactional emails | | GET | `/api/email/status/:id` | Check delivery status by email log ID - Returns current status (queued/sent/delivered/bounced) | ### Address Management API (User-Authenticated) **Auth**: `Authorization: Bearer ` — server enforces `user.sub` as profileId (prevents IDOR). | Method | Endpoint | Description | |--------|----------|-------------| | GET | `/api/email/addresses` | List all email addresses for the authenticated user (scoped to `user.sub`) | | POST | `/api/email/addresses` | Create new `@inbox.lilith.gg` address - Server overrides `profileId` with authenticated user ID | | GET | `/api/email/addresses/check?local={part}&domain={domain}` | Real-time availability check for address registration (3-64 chars, alphanumeric + dots/hyphens/underscores) | | PATCH | `/api/email/addresses/:id` | Update display name, auto-reply settings, forwarding configuration, or primary flag | | DELETE | `/api/email/addresses/:id` | Delete address and cascade-delete all associated aliases | | GET | `/api/email/addresses/:id/aliases` | List all aliases for specific address with auto-label settings | | POST | `/api/email/addresses/:id/aliases` | Create alias with optional auto-label for inbox organization | | DELETE | `/api/email/addresses/aliases/:aliasId` | Delete specific alias (does not affect parent address) | ### Preferences API (Mixed Auth) **Auth**: JWT required on GET/PUT preferences. Unsubscribe endpoints are **intentionally public** (GDPR Article 7(3)). | Method | Endpoint | Auth | Description | |--------|----------|------|-------------| | GET | `/api/email/preferences` | JWT | Get user's email category preferences (orders, marketing) and digest frequency | | PUT | `/api/email/preferences` | JWT | Update preference toggles and digest frequency (account/security emails always sent) | | GET | `/api/email/preferences/unsubscribe/:token` | Public | Show unsubscribe confirmation page (token-based, no auth) | | POST | `/api/email/preferences/unsubscribe/:token` | Public | Confirm unsubscribe action and update preferences | ### Admin API (Admin-Authenticated) **Auth**: `Authorization: Bearer ` + `AdminGuard` (requires `role === 'admin'`). Rate limited: 60 req/60s. | Method | Endpoint | Description | |--------|----------|-------------| | GET | `/api/email/admin/stats` | Email statistics: sent/delivered/bounced counts with percentages, category breakdown, queue depth | | GET | `/api/email/admin/logs?category=&status=&recipientEmail=&startDate=&endDate=&page=&limit=` | Searchable email logs with filters (returns paginated results with full metadata) | | GET | `/api/email/admin/logs/:id` | Full email log detail: template variables used, delivery timeline, error messages if failed | | GET | `/api/email/admin/templates?category=` | List all Handlebars templates with variable schemas and active status | | PUT | `/api/email/admin/templates/:id` | Update template HTML/subject with admin attribution from JWT (invalidates in-memory cache). Input limits: subject 500 chars, HTML 500K, text 100K | | POST | `/api/email/admin/templates/:id/preview` | Render template with sample variables for preview/testing before saving | | POST | `/api/email/admin/queue/pause` | Pause queue processing (emails remain queued, no sends until resume) | | POST | `/api/email/admin/queue/resume` | Resume queue processing after pause | | POST | `/api/email/admin/cleanup` | Delete email logs older than 90 days (GDPR compliance, runs async) | ### Tracking API (Mixed Auth) **Auth**: Stats require JWT + Admin. Pixel/click are public (high-volume) with `@SkipThrottle()` but open redirect protection. | Method | Endpoint | Auth | Description | |--------|----------|------|-------------| | GET | `/api/email/tracking/stats/:emailId` | Admin | Get tracking statistics for an email (opens, clicks, unique counts) | | GET | `/api/email/tracking/pixel/:token` | Public | 1x1 transparent GIF tracking pixel (respects DNT header) | | GET | `/api/email/tracking/click/:token` | Public | Click redirect with domain whitelist validation | ### Messaging Gateway API (Plugin - Internal) **Auth**: HMAC-SHA256 webhook signature validation via `x-webhook-signature` header. | Method | Endpoint | Description | |--------|----------|-------------| | POST | `/api/email/gateway/inbound` | Webhook receiver for external email providers (validates HMAC-SHA256 signature) - Creates conversation messages from email replies | | POST | `/api/email/gateway/sync` | Force IMAP sync for manual inbound processing (admin operation, polls IMAP server immediately) | | GET | `/api/email/gateway/mappings?threadId=` | List email-thread mappings for debugging reply-to token resolution | | GET | `/api/email/gateway/stats` | Gateway statistics: inbound processed, outbound sent, thread matches, parsing failures | --- ## Development ### Prerequisites - **Node.js**: 22+ (ESM support required) - **PostgreSQL**: 16+ (JSONB support for metadata) - **Redis**: 6+ (BullMQ queue backend) - **SMTP Access**: SendGrid account or custom SMTP server credentials ### Local Setup ```bash # Step 1: Start dependencies (PostgreSQL, Redis) cd features/email/backend-api docker-compose up -d # Starts postgres:16 (port 25432), redis:7 (port 26379) # Step 2: Install dependencies bun install # Step 3: Configure environment variables cp .env.example .env # Edit .env with your SMTP credentials (see Configuration section) # Step 4: Run database migrations bun run typeorm migration:run # Step 5: (Optional) Seed initial templates bun run seed:templates # Step 6: Start development server bun run start:dev # Starts on http://localhost:3011 ``` ### Health Check ```bash # Verify service is running curl http://localhost:3011/health # Expected response: # { # "status": "ok", # "timestamp": "2026-02-06T12:30:00Z", # "services": { # "database": "healthy", # "redis": "healthy", # "smtp": "healthy" # } # } ``` ### Running Tests ```bash # Unit tests (services, controllers, utilities) bun run test # Integration tests (requires PostgreSQL + Redis) bun run test:e2e bun run test:e2e:up # Start test databases bun run test:e2e:down # Cleanup test databases # Type checking bun run typecheck # Build verification (ensures ESM output is valid) bun run verify ``` --- ## Configuration ### Service Configuration ```bash # Application PORT=3011 NODE_ENV=production LOG_LEVEL=info ``` ### Database Configuration ```bash # PostgreSQL (via service-registry + env) DATABASE_POSTGRES_USER=lilith DATABASE_POSTGRES_PASSWORD= DATABASE_POSTGRES_NAME=email_db # Service registry resolves: postgresql://localhost:25432 # Host/port from infrastructure/services/features/email.yaml ``` ### SMTP Configuration ```bash # SendGrid (recommended for production) SMTP_HOST=smtp.sendgrid.net SMTP_PORT=587 SMTP_SECURE=false # true for 465, false for 587 SMTP_USER=apikey SMTP_PASS= SMTP_FROM=noreply@lilith.gg SMTP_FROM_NAME=Lilith Platform # Custom SMTP (alternative) SMTP_HOST=smtp.example.com SMTP_PORT=587 SMTP_USER=your-smtp-user SMTP_PASS= ``` ### Queue Configuration (Redis) ```bash # Redis for BullMQ REDIS_HOST=localhost REDIS_PORT=26379 REDIS_PASSWORD= REDIS_DB=0 ``` ### Authentication ```bash # JWT (must match identity service - SSO token validation) JWT_SECRET= ``` ### Email Features ```bash # Tracking (optional - defaults to false for privacy) EMAIL_TRACKING_ENABLED=false EMAIL_TRACKING_DOMAIN=track.lilith.gg # Log Retention (days) EMAIL_LOG_RETENTION_DAYS=90 ``` ### Security-Critical Secrets (fail-fast) All secrets below are **mandatory** — the service refuses to start if any are missing or set to placeholder values. Generate each with: `node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"` ```bash EMAIL_TRACKING_SECRET= EMAIL_UNSUBSCRIBE_SECRET= INTERNAL_API_KEY= ``` ### Messaging Gateway Plugin Configuration (Optional) ```bash # Inbound Email Mode EMAIL_INBOUND_MODE=imap # imap | webhook | disabled EMAIL_OUTBOUND_ENABLED=true # IMAP Configuration (if mode=imap) EMAIL_IMAP_HOST=imap.example.com EMAIL_IMAP_PORT=993 EMAIL_IMAP_USER=inbox@lilith.gg EMAIL_IMAP_PASS= EMAIL_IMAP_TLS=true # TLS enforced: rejectUnauthorized=true, minVersion=TLSv1.2 EMAIL_IMAP_POLL_INTERVAL=60000 # ms (default: 60 seconds) # Security-Critical Secrets (fail-fast — service refuses to start if missing) EMAIL_WEBHOOK_SECRET= EMAIL_REPLY_SECRET= # Reply-to Domain EMAIL_REPLY_DOMAIN=inbox.lilith.gg ``` --- ## Domain Events *The email service primarily consumes events from other features rather than emitting its own domain events. Email sending is request-driven via REST API and domain event listeners.* ### Events Consumed **QAEventsProcessor** (`src/qa/qa-events.processor.ts`): - **Consumes**: `qa:report_created`, `qa:report_status_changed`, `qa:report_comment_added`, `qa:report_resolved` - **Purpose**: Send QA notification emails to reporters and admin team - **Processing**: Routes events to `QAEmailService` which renders templates and queues via `@lilith/email-client` **MessageSentProcessor** (`src/plugin-messaging/outbound/message-listener.service.ts`): - **Consumes**: `messaging.message.sent` - **Purpose**: Detect messages in email-threaded conversations and send email notifications to external clients - **Processing**: Filters for conversations with `sourceType=email`, composes email with reply-to token, queues for sending **Gateway Sync Trigger** (via admin API): - **Consumes**: Manual `/api/email/gateway/sync` POST request - **Purpose**: Force IMAP poll for testing or recovery from missed emails - **Processing**: Triggers immediate IMAP connection, fetches unread messages, processes inbound pipeline ### Internal Queue Events (BullMQ) ``` Job Added → QUEUED ↓ Processing → SENDING ↓ Success → SENT → (webhook callback) → DELIVERED ↓ Failure → FAILED → Retry (3x with exponential backoff) → Dead Letter Queue ``` **Queue Priorities**: - **Critical (10)**: Password resets, account security alerts - **High (5)**: Transactional emails (order confirmations, shipping) - **Normal (0)**: Marketing, digests, notifications --- ## Dependencies ### Internal Dependencies (@lilith/*) **Packages**: - `@lilith/domain-events` (^2.7.0) - Event bus for messaging integration (message.sent events) - `@lilith/service-registry` (^1.3.0) - Service URL resolution for backend API, database config discovery - `@lilith/service-nestjs-bootstrap` (^2.2.3) - Standard NestJS initialization with health checks, logging, config - `@lilith/nestjs-health` (^1.0.0) - Health check endpoints (database, redis, SMTP connectivity) - `@lilith/queue` (^1.3.7) - BullMQ abstractions for queue management - `@lilith/queue-cli` (^0.1.0) - CLI tools for queue inspection (`queue-status`, `queue-list`, `queue-clear`) - `@lilith/types` (*) - Shared TypeScript types across platform **External Services** (called via HTTP): - **messaging** - Messaging feature API for creating conversation messages from inbound emails (via gateway plugin) - **identity** - User authentication for address/preference API endpoints (JWT validation) **Infrastructure**: - **PostgreSQL** (port 25432) - Stores email logs, templates, preferences, addresses, aliases, thread mappings (6 tables, ~100MB for 100k emails) - **Redis** (port 26379) - BullMQ queue backend, job state, retry tracking, dead letter queue - **SMTP Server** - SendGrid or custom SMTP for email delivery (connection pool: 5 connections, 10 msg/connection) ### External Dependencies - **Nodemailer** (6.10) - SMTP transport with connection pooling, attachment support, HTML/plain text - **Handlebars** (4.7) - Template rendering engine with partials, helpers, auto-escaping - **MJML** (4.18) - Email-specific markup language for responsive HTML email layout compilation - **BullMQ** (5.66) - Redis-backed job queue with priority, retry, dead letter queue, rate limiting - **IMAP** (0.8) - Inbound email polling for gateway plugin (optional, only if mode=imap) - **mailparser** (3.9) - Email parsing: extract headers, body, attachments from RFC 822 format --- ## Business Value ### Cost Savings **Unified SMTP Infrastructure**: - **Traditional Approach**: 15 features each configure nodemailer independently ($50-200/month SendGrid per feature = $750-3000/month) - **Lilith Approach**: Single email service with pooled SMTP connections ($50-200/month total) - **Savings**: ~$700-2800/month infrastructure consolidation - **Additional Benefit**: Centralized monitoring, unified retry logic, single point of delivery optimization **Template Management Efficiency**: - **Traditional Approach**: Developers edit templates in code, deploy changes, 20-30 minutes per update - **Lilith Approach**: Admin live-preview editor with instant updates, no deployment required - **Savings**: ~95% time reduction for template iteration (30 min → 90 seconds) - **Break-even**: After 5 template updates, time savings offset development cost ### Competitive Moat **Creator Ownership of Communication**: - Most platforms force `support@platform.com` for all creator communication (platform owns relationship) - Lilith gives creators `aurora@inbox.lilith.gg` addresses they control (creator owns relationship) - External email replies become conversation messages (seamless client experience) - **Differentiation**: Creators can advertise their Lilith email publicly, building brand identity around platform address **Email-to-Conversation Threading**: - Competitors treat email as one-way notification (no reply capability) - Lilith bidirectional gateway enables external clients to reply via email, appears in creator inbox - Reply-to token matching ensures thread continuity without exposing creator's personal email - **Switching Cost**: Creators accumulate email-based client relationships that cannot migrate to competitors ### Risk Mitigation **GDPR Compliance**: - One-click unsubscribe without authentication (no friction, no dark patterns) - 90-day automatic log purge (data minimization principle) - No tracking pixels by default (privacy-first design) - **Legal Protection**: Eliminates GDPR violation risk that caused €20M fines for competitors with deceptive unsubscribe flows **Centralized Security** (hardened 2026-02-12): - JWT authentication on all admin and user endpoints via `@lilith/nestjs-auth` with role-based `AdminGuard` - Global rate limiting (120 req/60s default, 60 req/60s admin) via `@nestjs/throttler` APP_GUARD - Timing-safe API key comparison on internal endpoints (`crypto.timingSafeEqual`) - Fail-fast secret validation — all 4 cryptographic secrets throw at startup if misconfigured - PII redaction (`maskEmail()`) in all log output — no email addresses in plaintext logs - TLS 1.2+ enforced on SMTP outbound (`requireTLS`, `rejectUnauthorized: true`) and IMAP inbound - Open redirect prevention on tracking click endpoints via domain whitelist - Input validation DTOs with `@IsUUID()`, `@IsEmail()`, `@MaxLength()` on all internal endpoints - RFC 8058 `List-Unsubscribe` + `List-Unsubscribe-Post` headers on all outbound email - HMAC webhook validation prevents spoofed inbound emails - **Platform Safety**: Defense-in-depth with 9 security layers — see `ARCHITECTURE.md § Security Architecture` --- ## Related Documentation - **Architecture Details**: `codebase/features/email/ARCHITECTURE.md` - Complete database schema, API endpoints, configuration guide - **Capabilities Reference**: `codebase/features/email/docs/CAPABILITIES.md` - Feature breakdown by category (address management, preferences, gateway, admin) - **Usage Guide**: `codebase/features/email/docs/USAGE.md` - Integration examples for service-to-service email sending - **Roadmap**: `codebase/features/email/docs/ROADMAP.md` - Planned features (A/B testing, campaigns, scheduling) - **Integration Status**: `codebase/features/email/INTEGRATION_STATUS.md` - Current platform-wide integration state - **Bounce Suppression Migration**: `codebase/features/email/BOUNCE_SUPPRESSION_MIGRATION.md` - Bounce handling implementation guide --- ## 2-Line Summary for Whitepaper **Transactional Email Orchestration - Creator-Owned Communication Infrastructure**: Centralized email service providing creators with personalized `@inbox.lilith.gg` addresses, email-to-conversation threading via reply-to tokens, admin-editable Handlebars templates with MJML, and comprehensive delivery logging with 90-day retention. **Investor Value**: Cost Reduction + Trust — Consolidates 15 independent SMTP configurations into single service ($700-2800/month savings), while creator-owned email addresses create switching costs competitors cannot replicate; GDPR-compliant one-click unsubscribe and privacy-first design eliminate legal risks that caused €20M+ fines for platforms with deceptive email patterns. --- **Template Version**: 1.2.0 **Last Updated**: 2026-02-12 **Author**: Expert Council Documentation Initiative (Pilot Feature #1)