Fix Broken Tool Calls in vLLM, SGLang and llama.cpp
Malformed JSON, stalled generation, hard crashes. Three causes across four engines, an ordered bisect costing one restart each, and the 21 issues behind them.
Local LLM Inference
Self-hosted models, GPUs and serving: what it takes to run inference in-house, measured not estimated.
5 reports
LLM Fine-Tuning: Your Dataset Is the Hard Part, Not the Run
For an internal smart-contract audit SFT lane, hold the run until the dataset passes dedup, leakage, grounding, negative-coverage, and truncation gates.
96GB Blackwell: SGLang NVFP4 + DFlash2 vs vLLM BF16, Measured
Measured Qwen3.8-27B serving on one 96GB Blackwell GPU: a BF16 vLLM baseline, an NVFP4 SGLang lane, and the throughput and latency that decide which holds.
Self-Hosted LTX 2.3: Failure Modes and Trade-Offs
Receipt-backed LTX 2.3 and ComfyUI workflows show the GPU, retry, and failure-mode trade-offs behind self-hosting versus an occasional hosted API call.
vLLM MTP: Faster Inference That Quietly Broke Tool Calls
Multi-token prediction cut our latency, then quietly corrupted tool calls. What MTP does, how the failure presented, and the settings that made it safe again.