inference

10 articles tagged with “inference”

A pure Rust inference engine achieves 96 tok/s on Qwen3.5-35B-A3B—a 3× speedup over vLLM's 31 tok/s, with cold startup in 15 seconds versus vLLM's 10

DeepSeek V4 arrives with two models: a 1.6-trillion-parameter Pro variant and a 284-billion-parameter Flash variant. The headline number is the Pro m

Serving large language models at production scale boils down to one problem: getting the most tokens out of your GPU per second, per dollar. Two open

Running your own LLM inference server gives you complete control over data privacy, latency, and costs. This guide walks through deploying a producti