Skip to main content
← Back to all tags

inference

10 articles tagged with “inference”

vllmdeepseekdgx-sparkgpusglanginferenceself-hostedgb10
ai
· ~8 min read

Serving Frontier MoE Models at Scale: FP8 KV Caches, INT4 Weights, and KV Cache Integrity

Frontier mixture-of-experts models are wonderful to serve and very hard to fit into GPU memory. In August 2026, Cloudflare published the operating de

ai
· ~14 min read

Sovereign AI on Your Desk: Why Two DGX Sparks Beat the Cloud

In 2024, running a 685-billion-parameter model locally meant a rack of A100s, a six-figure power bill, and a PhD in distributed systems. In 2026, it

ai
· ~6 min read

Explicit Prompt Caching for OpenAI GPT-5.6 on Amazon Bedrock: Architecture and Patterns

> **Teaser:** OpenAI GPT-5.6 Sol, Terra, and Luna now support explicit prompt caching on Amazon Bedrock — mark exactly which prefix to cache, set you

ai
· ~5 min read

Inference Meta-Monitoring for SageMaker AI Endpoints: Governance for Production LLM APIs

> **Teaser:** AWS published a reference architecture for inference meta-monitoring on SageMaker AI endpoints — monitoring the monitors themselves. Th

ai
· ~9 min read

Two-Node DGX Spark Cluster: Running DeepSeek V4 Flash at 20 TPS

Two NVIDIA DGX Sparks. One 685B-parameter MoE model. A 3× throughput improvement without a single crash. Here's how. ## The Setup: Two GB10 Workstat

ai
· ~7 min read

Token Cost Efficiency: How Graph Structures Reduce LLM Inference Costs

Token costs are the dominant line item in LLM deployment. GPT-4.1 at \$2.30/million input tokens sounds small until you're processing 50 documents pe