Runtime

ExLlamaV2

quantized NVIDIA GPU inference

Platforms

Linux · Windows

Why it matters

Runtime changes fit

Memory behavior, quantization support and throughput vary by runtime.

Evidence

Official/project docs

Inspect source ↗