ml-model-loader/README.md

tqftw-model-loader

Unified ML model loading, caching, and device management.

Features

• GGUF Loader: Load llama-cpp-python models from manifest or direct paths
• HuggingFace Loader: Load transformers models with automatic device placement
• Diffusers Loader: Load Stable Diffusion and SDXL pipelines
• Model Registry: Manifest-based model discovery and caching
• Device Management: Automatic GPU/CPU selection with memory tracking

Installation

# Core package (no ML frameworks)
pip install -e .

# With GGUF support
pip install -e ".[gguf]"

# With all ML frameworks
pip install -e ".[ml]"

Usage

from tqftw_model_loader.gguf_loader import GGUFModelLoader

loader = GGUFModelLoader()
await loader.load("ministral-3b-instruct")

response = loader.generate("Hello, world!")

Manifest

Models are discovered via ~/.cache/models/manifest.json. The loader supports:

• Model ID lookup (e.g., "ministral-3b-instruct")
• Direct file paths (e.g., "/path/to/model.gguf")
• Remote fetching via rsync