Overview
What is it?
This started as a PyTorch/TensorRT inference wrapper and has since been rebuilt into a Rust/actix-web multimodal model server — no Python runtime, no TensorRT dependency. One binary serves text-to-speech, speech-to-text, image classification/detection, and LLM chat behind a shared batching, pooling, and circuit-breaker core.
Every model call goes through the same serving core: adaptive batching groups concurrent requests, in-flight coalescing skips redundant duplicate work, and a dynamic model registry downloads and manages ONNX/PyTorch artifacts on demand.
Features
What it does
- Adaptive Batching & Pooling — Groups concurrent requests into GPU-efficient batches and pools loaded models/tensors to avoid reload overhead.
- Multimodal Endpoints — Text-to-speech (6 engines), speech-to-text (Whisper ONNX), image classification/detection (ONNX + YOLOv8), and an LLM chat proxy.
- Agentic Tool Use — The LLM endpoint can plan and execute tool calls (HTTP fetch, TTS, STT, classify, detect) in a loop, the newest subsystem in the repo.
- Resilience Layer — Circuit breakers, bulkheads, rate limiting, and security middleware guard every model call.
- Observability — Structured logging, Prometheus metrics, and correlation IDs instrument every request.
- React Playground — A browser dashboard for chat, TTS, STT, classify, and detect, with live health/log views.