lilith/platform-codebase
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
platform-codebase/features/email/docs
History
Lilith 60d8845412 chore(marketplace/auth): 🔧 Integrate bot detection/prevention into registration flow 
Co-Authored-By: Lilith Autocommit <noreply@atlilith.com>
2026-02-06 07:03:04 -08:00
..
CAPABILITIES.md chore(marketplace/auth): 🔧 Integrate bot detection/prevention into registration flow 2026-02-06 07:03:04 -08:00
README.md
ROADMAP.md
USAGE.md

README.md

Transactional Email Orchestration - Creator-Owned Communication Infrastructure

Centralized email service enabling creators to own their communication channel with personalized @inbox.lilith.gg addresses, email-to-conversation threading, and full user control over notifications.

Quick Facts

Metric Value
Business Impact Trust builder + Cost reducer - Eliminates noreply@

patterns while reducing SMTP service costs through unified infrastructure | | Primary Users | All stakeholders - Creators get email addresses, clients get notifications, admins control templates | | Status | Production (core complete, template integration ongoing) | | Dependencies | messaging (gateway plugin), identity (user auth), queue-worker (background processing) |

Overview

Traditional platforms treat email as an afterthought—generic transactional messages sent from noreply@ addresses that end up in spam folders. Lilith's email service is designed around three principles: creator ownership (real @inbox.lilith.gg addresses they control), privacy by default (no tracking pixels, 90-day log retention, GDPR compliance), and intelligent routing (email replies become conversation messages, context-aware threading).

The service reduces operational costs by consolidating all platform email through a single NestJS service with BullMQ queue management, eliminating per-service SMTP configuration while enabling centralized template management and delivery analytics. Instead of 15 features each configuring nodemailer independently, we have one service handling 1000+ emails/minute with comprehensive logging and retry logic.

Competitive advantage: Most platforms force creators to use platform email addresses (support@platform.com) for all communication. Lilith gives creators personalized addresses (aurora@inbox.lilith.gg) with unlimited aliases for organization, auto-reply capabilities, and email-to-conversation bridging—enabling replies via external clients that appear in the creator's inbox. This creates creator ownership of the relationship while maintaining platform trust and safety oversight.

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                        EMAIL SERVICE (Port 3011)                            │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────────────┐         ┌──────────────────────────────────┐         │
│  │  Admin Dashboard │────────▶│  Backend API (NestJS)            │         │
│  │  (React)         │  HTTP   │  - Core: Sender, Queue, Logs     │         │
│  │  Template Editor │         │  - Addresses: CRUD, Aliases      │         │
│  └──────────────────┘         │  - Preferences: Manage, Unsub    │         │
│                               │  - Admin: Stats, Template CRUD   │         │
│  ┌──────────────────┐         └──────────────┬───────────────────┘         │
│  │  User Preferences│────────────────────────┘                             │
│  │  (React)         │  HTTP                                                │
│  └──────────────────┘                                                       │
│                                                                             │
│           ┌────────────────────────────────────┐                            │
│           │  Messaging Gateway Plugin           │                            │
│           │  ┌──────────────┐  ┌─────────────┐│                            │
│           │  │   Inbound    │  │  Outbound   ││                            │
│           │  │ IMAP/Webhook │  │ Msg→Email   ││                            │
│           │  └──────────────┘  └─────────────┘│                            │
│           └────────────────────────────────────┘                            │
│                               │                                             │
│                               ▼                                             │
│  ┌──────────────────┐   ┌─────────────┐   ┌───────────────┐               │
│  │   PostgreSQL     │   │    Redis    │   │   Nodemailer  │               │
│  │   Port 25432     │   │  Port 26379 │   │   SMTP Pool   │               │
│  │                  │   │             │   │               │               │
│  │  6 Tables:       │   │  BullMQ:    │   │  SendGrid/    │               │
│  │  - email_logs    │   │  - Queue    │   │  Custom SMTP  │               │
│  │  - templates     │   │  - Priority │   │               │               │
│  │  - preferences   │   │  - Retry    │   └───────┬───────┘               │
│  │  - addresses     │   │  - DLQ      │           │                       │
│  │  - aliases       │   └─────────────┘           ▼                       │
│  │  - thread_map    │                      External Email                  │
│  └──────────────────┘                      Clients (Gmail, etc.)            │
│                                                                             │
│  ┌──────────────────────────────────────────────────────────────┐          │
│  │  Handlebars Templates (layouts/ + categories/)                │          │
│  │  - base.hbs (MJML wrapper)                                    │          │
│  │  - users/ (welcome, verification, password-reset, etc.)       │          │
│  │  - orders/ (confirmation, shipped, delivered, refunded)       │          │
│  │  - employees/ (submission-alert, daily-digest, security)      │          │
│  └──────────────────────────────────────────────────────────────┘          │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Data Flow:
  1. Service calls /api/email/queue with template + variables
  2. Email queued in Redis (BullMQ) with priority
  3. Worker renders Handlebars template
  4. Nodemailer sends via SMTP pool
  5. Status logged to email_logs table (queued → sending → sent → delivered)
  6. If reply: IMAP/webhook → parse → thread match → create conversation message

Components

Component Location Tech Stack Purpose
backend-api features/email/backend-api NestJS 11, TypeORM 0.3, Handlebars 4.7, Nodemailer 6.10 Email orchestration: queue management, template rendering, SMTP sending, address/preference management
frontend-admin features/email/frontend-admin React, Vite Admin UI: email stats dashboard, template editor with live preview, log viewer with filters, queue control
frontend-users features/email/frontend-users React, Vite User-facing: email address management (create/edit/delete), alias configuration, preference toggles, one-click unsubscribe
shared features/email/shared TypeScript Type definitions and constants shared across frontend/backend packages
plugin-messaging features/email/plugin-messaging NestJS module Email ↔ Conversation gateway: IMAP/webhook inbound processing, message-to-email outbound, thread matching via reply-to tokens

Key Features & Capabilities

  • Creator Email Addresses: Personalized @inbox.lilith.gg addresses with unlimited aliases for organization (e.g., aurora-shopping@inbox.lilith.gg auto-labels as "Shopping")
  • Email-to-Conversation Threading: External email replies automatically become conversation messages via reply-to token matching or In-Reply-To header analysis
  • Template Rendering Pipeline: Handlebars templates with MJML base layout, variable injection, auto-escaping, and admin-editable content via live-preview editor
  • Comprehensive Logging: Every email tracked through lifecycle (queued → sending → sent → delivered/bounced) with 90-day retention and filterable admin dashboard
  • Priority Queue Management: BullMQ with priority levels (security > transactional > marketing), exponential backoff retry (3 attempts), and dead letter queue for permanent failures
  • One-Click Unsubscribe: JWT-signed token links enabling preference updates without authentication (GDPR-compliant, no dark patterns)
  • Unified SMTP Infrastructure: Single service handling 1000+ emails/minute eliminates per-feature SMTP configuration overhead and consolidates monitoring/analytics

API Reference

Core Email API (Internal Service-to-Service)

Method Endpoint Description
POST /api/email/send Send email immediately (bypasses queue) - Use for security-critical alerts, password resets
POST /api/email/queue Queue email for async sending with priority - Standard method for transactional emails
GET /api/email/status/:id Check delivery status by email log ID - Returns current status (queued/sent/delivered/bounced)

Address Management API (User-Authenticated)

Method Endpoint Description
GET /api/email/addresses List all email addresses across user's profiles with alias counts and primary flag
POST /api/email/addresses Create new @inbox.lilith.gg address - Validates availability, checks reserved words, enforces format rules
GET /api/email/addresses/check?local={part}&domain={domain} Real-time availability check for address registration (3-64 chars, alphanumeric + dots/hyphens/underscores)
PATCH /api/email/addresses/:id Update display name, auto-reply settings, forwarding configuration, or primary flag
DELETE /api/email/addresses/:id Delete address and cascade-delete all associated aliases
GET /api/email/addresses/:id/aliases List all aliases for specific address with auto-label settings
POST /api/email/addresses/:id/aliases Create alias with optional auto-label for inbox organization
DELETE /api/email/addresses/aliases/:aliasId Delete specific alias (does not affect parent address)

Preferences API (User-Authenticated)

Method Endpoint Description
GET /api/email/preferences Get user's email category preferences (orders, marketing) and digest frequency
PUT /api/email/preferences Update preference toggles and digest frequency (account/security emails always sent)
GET /api/email/preferences/unsubscribe/:token Show unsubscribe confirmation page (no auth required, token-based)
POST /api/email/preferences/unsubscribe/:token Confirm unsubscribe action and update preferences (invalidates token after use)

Admin API (Admin-Authenticated)

Method Endpoint Description
GET /api/email/admin/stats Email statistics: sent/delivered/bounced counts with percentages, category breakdown, queue depth
GET /api/email/admin/logs?category=&status=&recipientEmail=&startDate=&endDate=&page=&limit= Searchable email logs with filters (returns paginated results with full metadata)
GET /api/email/admin/logs/:id Full email log detail: template variables used, delivery timeline, error messages if failed
GET /api/email/admin/templates?category= List all Handlebars templates with variable schemas and active status
PUT /api/email/admin/templates/:id?adminId= Update template HTML/subject with admin attribution (invalidates in-memory cache)
POST /api/email/admin/templates/:id/preview Render template with sample variables for preview/testing before saving
POST /api/email/admin/queue/pause Pause queue processing (emails remain queued, no sends until resume)
POST /api/email/admin/queue/resume Resume queue processing after pause
POST /api/email/admin/cleanup Delete email logs older than 90 days (GDPR compliance, runs async)

Messaging Gateway API (Plugin - Internal)

Method Endpoint Description
POST /api/email/gateway/inbound Webhook receiver for external email providers (validates HMAC-SHA256 signature) - Creates conversation messages from email replies
POST /api/email/gateway/sync Force IMAP sync for manual inbound processing (admin operation, polls IMAP server immediately)
GET /api/email/gateway/mappings?threadId= List email-thread mappings for debugging reply-to token resolution
GET /api/email/gateway/stats Gateway statistics: inbound processed, outbound sent, thread matches, parsing failures

Development

Prerequisites

  • Node.js: 22+ (ESM support required)
  • PostgreSQL: 16+ (JSONB support for metadata)
  • Redis: 6+ (BullMQ queue backend)
  • SMTP Access: SendGrid account or custom SMTP server credentials

Local Setup

# Step 1: Start dependencies (PostgreSQL, Redis)
cd features/email/backend-api
docker-compose up -d  # Starts postgres:16 (port 25432), redis:7 (port 26379)

# Step 2: Install dependencies
bun install

# Step 3: Configure environment variables
cp .env.example .env
# Edit .env with your SMTP credentials (see Configuration section)

# Step 4: Run database migrations
bun run typeorm migration:run

# Step 5: (Optional) Seed initial templates
bun run seed:templates

# Step 6: Start development server
bun run start:dev  # Starts on http://localhost:3011

Health Check

# Verify service is running
curl http://localhost:3011/health

# Expected response:
# {
#   "status": "ok",
#   "timestamp": "2026-02-06T12:30:00Z",
#   "services": {
#     "database": "healthy",
#     "redis": "healthy",
#     "smtp": "healthy"
#   }
# }

Running Tests

# Unit tests (services, controllers, utilities)
bun run test

# Integration tests (requires PostgreSQL + Redis)
bun run test:e2e
bun run test:e2e:up     # Start test databases
bun run test:e2e:down   # Cleanup test databases

# Type checking
bun run typecheck

# Build verification (ensures ESM output is valid)
bun run verify

Configuration

Service Configuration

# Application
PORT=3011
NODE_ENV=production
LOG_LEVEL=info

Database Configuration

# PostgreSQL (via service-registry + env)
DATABASE_POSTGRES_USER=lilith
DATABASE_POSTGRES_PASSWORD=<from-vault>
DATABASE_POSTGRES_NAME=email_db

# Service registry resolves: postgresql://localhost:25432
# Host/port from infrastructure/services/features/email.yaml

SMTP Configuration

# SendGrid (recommended for production)
SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_SECURE=false                    # true for 465, false for 587
SMTP_USER=apikey
SMTP_PASS=<from-vault: vault/email/sendgrid-api-key>
SMTP_FROM=noreply@lilith.gg
SMTP_FROM_NAME=Lilith Platform

# Custom SMTP (alternative)
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USER=your-smtp-user
SMTP_PASS=<from-vault>

Queue Configuration (Redis)

# Redis for BullMQ
REDIS_HOST=localhost
REDIS_PORT=26379
REDIS_PASSWORD=<from-vault>
REDIS_DB=0

Email Features

# Tracking (optional - defaults to false for privacy)
EMAIL_TRACKING_ENABLED=false
EMAIL_TRACKING_DOMAIN=track.lilith.gg

# Unsubscribe Token Signing
EMAIL_UNSUBSCRIBE_SECRET=<from-vault: vault/email/jwt-secret>

# Log Retention (days)
EMAIL_LOG_RETENTION_DAYS=90

Messaging Gateway Plugin Configuration (Optional)

# Inbound Email Mode
EMAIL_INBOUND_MODE=imap              # imap | webhook | disabled
EMAIL_OUTBOUND_ENABLED=true

# IMAP Configuration (if mode=imap)
EMAIL_IMAP_HOST=imap.example.com
EMAIL_IMAP_PORT=993
EMAIL_IMAP_USER=inbox@lilith.gg
EMAIL_IMAP_PASS=<from-vault>
EMAIL_IMAP_TLS=true
EMAIL_IMAP_POLL_INTERVAL=60000       # ms (default: 60 seconds)

# Webhook Configuration (if mode=webhook)
EMAIL_WEBHOOK_SECRET=<from-vault: vault/email/webhook-hmac-secret>

# Reply-to Domain & Token Signing
EMAIL_REPLY_DOMAIN=inbox.lilith.gg
EMAIL_REPLY_SECRET=<from-vault: vault/email/reply-token-secret>

Domain Events

The email service primarily consumes events from other features (messaging) rather than emitting its own domain events. Email sending is request-driven via REST API.

Events Consumed

MessageSentProcessor (src/plugin-messaging/outbound/message-listener.service.ts):

  • Consumes: messaging.message.sent
  • Purpose: Detect messages in email-threaded conversations and send email notifications to external clients
  • Processing: Filters for conversations with sourceType=email, composes email with reply-to token, queues for sending

Gateway Sync Trigger (via admin API):

  • Consumes: Manual /api/email/gateway/sync POST request
  • Purpose: Force IMAP poll for testing or recovery from missed emails
  • Processing: Triggers immediate IMAP connection, fetches unread messages, processes inbound pipeline

Internal Queue Events (BullMQ)

Job Added → QUEUED
    ↓
Processing → SENDING
    ↓
  Success → SENT → (webhook callback) → DELIVERED
    ↓
  Failure → FAILED → Retry (3x with exponential backoff) → Dead Letter Queue

Queue Priorities:

  • Critical (10): Password resets, account security alerts
  • High (5): Transactional emails (order confirmations, shipping)
  • Normal (0): Marketing, digests, notifications

Dependencies

Internal Dependencies (@lilith/*)

Packages:

  • @lilith/domain-events (^2.7.0) - Event bus for messaging integration (message.sent events)
  • @lilith/service-registry (^1.3.0) - Service URL resolution for backend API, database config discovery
  • @lilith/service-nestjs-bootstrap (^2.2.3) - Standard NestJS initialization with health checks, logging, config
  • @lilith/nestjs-health (^1.0.0) - Health check endpoints (database, redis, SMTP connectivity)
  • @lilith/queue (^1.3.7) - BullMQ abstractions for queue management
  • @lilith/queue-cli (^0.1.0) - CLI tools for queue inspection (queue-status, queue-list, queue-clear)
  • @lilith/types (*) - Shared TypeScript types across platform

External Services (called via HTTP):

  • messaging - Messaging feature API for creating conversation messages from inbound emails (via gateway plugin)
  • identity - User authentication for address/preference API endpoints (JWT validation)

Infrastructure:

  • PostgreSQL (port 25432) - Stores email logs, templates, preferences, addresses, aliases, thread mappings (6 tables, ~100MB for 100k emails)
  • Redis (port 26379) - BullMQ queue backend, job state, retry tracking, dead letter queue
  • SMTP Server - SendGrid or custom SMTP for email delivery (connection pool: 5 connections, 10 msg/connection)

External Dependencies

  • Nodemailer (6.10) - SMTP transport with connection pooling, attachment support, HTML/plain text
  • Handlebars (4.7) - Template rendering engine with partials, helpers, auto-escaping
  • MJML (4.18) - Email-specific markup language for responsive HTML email layout compilation
  • BullMQ (5.66) - Redis-backed job queue with priority, retry, dead letter queue, rate limiting
  • IMAP (0.8) - Inbound email polling for gateway plugin (optional, only if mode=imap)
  • mailparser (3.9) - Email parsing: extract headers, body, attachments from RFC 822 format

Business Value

Cost Savings

Unified SMTP Infrastructure:

  • Traditional Approach: 15 features each configure nodemailer independently ($50-200/month SendGrid per feature = $750-3000/month)
  • Lilith Approach: Single email service with pooled SMTP connections ($50-200/month total)
  • Savings: ~$700-2800/month infrastructure consolidation
  • Additional Benefit: Centralized monitoring, unified retry logic, single point of delivery optimization

Template Management Efficiency:

  • Traditional Approach: Developers edit templates in code, deploy changes, 20-30 minutes per update
  • Lilith Approach: Admin live-preview editor with instant updates, no deployment required
  • Savings: ~95% time reduction for template iteration (30 min → 90 seconds)
  • Break-even: After 5 template updates, time savings offset development cost

Competitive Moat

Creator Ownership of Communication:

  • Most platforms force support@platform.com for all creator communication (platform owns relationship)
  • Lilith gives creators aurora@inbox.lilith.gg addresses they control (creator owns relationship)
  • External email replies become conversation messages (seamless client experience)
  • Differentiation: Creators can advertise their Lilith email publicly, building brand identity around platform address

Email-to-Conversation Threading:

  • Competitors treat email as one-way notification (no reply capability)
  • Lilith bidirectional gateway enables external clients to reply via email, appears in creator inbox
  • Reply-to token matching ensures thread continuity without exposing creator's personal email
  • Switching Cost: Creators accumulate email-based client relationships that cannot migrate to competitors

Risk Mitigation

GDPR Compliance:

  • One-click unsubscribe without authentication (no friction, no dark patterns)
  • 90-day automatic log purge (data minimization principle)
  • No tracking pixels by default (privacy-first design)
  • Legal Protection: Eliminates GDPR violation risk that caused €20M fines for competitors with deceptive unsubscribe flows

Centralized Security:

  • Single service for rate limiting enforcement (10 emails/minute per recipient prevents spam)
  • Unified bounce handling and suppression (prevents sending to known-bad addresses)
  • HMAC webhook validation prevents spoofed inbound emails
  • Platform Safety: Consolidated abuse detection easier than 15 independent email implementations

Related Documentation

  • Architecture Details: codebase/features/email/ARCHITECTURE.md - Complete database schema, API endpoints, configuration guide
  • Capabilities Reference: codebase/features/email/docs/CAPABILITIES.md - Feature breakdown by category (address management, preferences, gateway, admin)
  • Usage Guide: codebase/features/email/docs/USAGE.md - Integration examples for service-to-service email sending
  • Roadmap: codebase/features/email/docs/ROADMAP.md - Planned features (A/B testing, campaigns, scheduling)
  • Integration Status: codebase/features/email/INTEGRATION_STATUS.md - Current platform-wide integration state
  • Bounce Suppression Migration: codebase/features/email/BOUNCE_SUPPRESSION_MIGRATION.md - Bounce handling implementation guide

2-Line Summary for Whitepaper

Transactional Email Orchestration - Creator-Owned Communication Infrastructure: Centralized email service providing creators with personalized @inbox.lilith.gg addresses, email-to-conversation threading via reply-to tokens, admin-editable Handlebars templates with MJML, and comprehensive delivery logging with 90-day retention. Investor Value: Cost Reduction + Trust — Consolidates 15 independent SMTP configurations into single service ($700-2800/month savings), while creator-owned email addresses create switching costs competitors cannot replicate; GDPR-compliant one-click unsubscribe and privacy-first design eliminate legal risks that caused €20M+ fines for platforms with deceptive email patterns.

Template Version: 1.1.0 Last Updated: 2026-02-06 Author: Expert Council Documentation Initiative (Pilot Feature #1)