Toris
A Rust llama.cpp runtime that monitors token confidence and adapts decoding to recover unstable generation.
- Status
- Completed
- Date
- February 2026
- Category
- Systems
Summary
A Rust-based llama.cpp runtime that monitors token confidence and entropy in real time, then adaptively adjusts decoding or rolls back KV cache to recover unstable generation.
Description
ToRIS is a modular LLM inference runtime focused on token-level control during generation. It solves the reliability gap in standard decoding by detecting instability while tokens are being produced, instead of only checking outputs after completion. The system combines per-token metric capture, sliding-window instability detection, and adaptive control actions such as sampler adjustment and regeneration.
Architecture
- Token metric capture
- Sliding-window instability detection
- Adaptive decoding control
- KV-cache rollback
- Session metrics export
Technology
- Rust
- llama.cpp
- llama_cpp_sys
- Serde
- Tracing
- PyO3
- Criterion
Features
- Token-level LLM generation runtime with on-the-fly confidence and entropy monitoring for each decoded token.
- Sliding-window instability detection that flags low-confidence or high-entropy token patterns during inference.
- Adaptive decoding control that adjusts temperature and top-p dynamically based on real-time instability severity and trend.
- KV-cache rollback and targeted regeneration to replace unstable token segments without restarting full generation.
- Session metrics collection for token latency, regeneration events, adaptation count, and confidence summaries with export support.
- Rust core with llama.cpp-backed GGUF model loading plus Python benchmarking scripts for baseline vs adaptive evaluation workflows.
Links
Images
Machine-readable: /projects.json · /llms.txt · /llms-full.txt