Lilith ML Context Manager

Manage conversation context with token counting, message windowing, and summarization.

Installation

# Basic installation
pip install lilith-ml-context-manager

# With tiktoken for accurate token counting
pip install lilith-ml-context-manager[tiktoken]

# With development dependencies
pip install lilith-ml-context-manager[dev]

Quick Start

from lilith_ml_context_manager import ContextManager, Message

# Create manager with token budget
manager = ContextManager(max_tokens=3072)

# Build context from conversation history
messages = [
    Message(role="user", content="Hello!"),
    Message(role="assistant", content="Hi there!"),
    # ... potentially 100+ messages
]

context = await manager.build_context(
    messages=messages,
    recent_count=5,  # Always include last 5
    summarize_older=True,
)

print(f"Context ({context.token_count} tokens):")
print(context.context)

Features

Token Counting

Accurate token counting with tiktoken (optional) or estimation fallback:

from lilith_ml_context_manager import get_token_counter

counter = get_token_counter()  # Auto-selects best available
tokens = counter.count("Hello, world!")

Message Windowing

Intelligent message selection based on recency and relevance:

from lilith_ml_context_manager import MessageWindow, Message

window = MessageWindow(recent_count=5, relevance_threshold=0.5)
result = window.select(messages, token_budget=2048)

# result.recent: last 5 messages (always included)
# result.relevant_older: older messages with entities/emotion
# result.to_summarize: messages that need summarization

Progressive Summarization

Different summarization levels based on message age:

• Messages 1-10: Full text (most recent, highest detail)
• Messages 11-50: Key points summary
• Messages 50+: Topic-level summary only

from lilith_ml_context_manager import LLMSummarizer, SummaryLevel

async def my_llm_call(prompt: str) -> str:
    return await call_claude(prompt)

summarizer = LLMSummarizer(llm_fn=my_llm_call)
result = await summarizer.summarize(
    messages=old_messages,
    level=SummaryLevel.KEY_POINTS,
    max_tokens=500,
)

Synchronous Usage

For non-async code:

from lilith_ml_context_manager import ContextManagerSync

manager = ContextManagerSync(max_tokens=2048)
context = manager.build_context(messages)

Configuration

from lilith_ml_context_manager import ContextConfig, ContextManager

config = ContextConfig(
    max_tokens=3072,         # Maximum token budget
    recent_count=5,          # Always include last N messages
    summarize_older=True,    # Summarize messages beyond recent
    include_system=True,     # Include system messages
    relevance_threshold=0.5, # Min relevance for older messages
)

manager = ContextManager(config=config)

Message Metadata

Messages can include metadata for relevance scoring:

from lilith_ml_context_manager import Message

msg = Message(
    role="user",
    content="Tell me about Paris, France.",
    metadata={
        "entities": [
            {"type": "location", "name": "Paris"},
            {"type": "location", "name": "France"},
        ],
        "emotional_significance": False,
    },
)

# Messages with entities or emotional markers score higher
# and are prioritized for inclusion in context

Context Building Strategy

The context manager uses this strategy:

1. Reserve budget for system prompt if provided
2. Always include the last N messages (recent_count)
3. Score older messages by relevance (entities, emotion, length)
4. Select relevant older messages within remaining budget
5. Summarize remaining messages using progressive levels
6. Format everything into an optimized context string

API Reference

ContextManager

Main class for async context management.

manager = ContextManager(
    max_tokens=3072,           # Token budget
    config=None,               # Optional ContextConfig
    token_counter=None,        # Optional custom counter
    llm_fn=None,               # Optional LLM for summarization
)

result = await manager.build_context(
    messages=messages,
    recent_count=5,            # Override config
    summarize_older=True,      # Override config
    system_prompt="...",       # Optional prepend
)

# Returns ContextResult with:
# - context: str
# - token_count: int
# - messages_included: int
# - messages_summarized: int
# - summary: str | None

Message

Message data model with metadata support.

msg = Message(
    role="user",          # "system", "user", "assistant", "tool"
    content="Hello!",
    timestamp=datetime.now(),  # Auto-set if omitted
    metadata={},               # Optional extra data
)

# Properties
msg.is_user       # True if role == "user"
msg.is_assistant  # True if role == "assistant"
msg.is_system     # True if role == "system"

# Methods
msg.has_entity("location")     # Check for entity type
msg.has_emotional_marker()     # Check for emotional flag

TokenCounter

Abstract interface for token counting.

from lilith_ml_context_manager import (
    get_token_counter,
    TiktokenCounter,
    EstimationCounter,
)

# Auto-select best available
counter = get_token_counter()

# Force tiktoken (raises if unavailable)
counter = TiktokenCounter()

# Force estimation (always available)
counter = EstimationCounter()

tokens = counter.count("text")
tokens = counter.count_messages([msg1, msg2])

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Type checking
mypy src

# Linting
ruff check src tests

License

MIT