Runtime
ExLlamaV3
optimized EXL3 quantized inference on modern consumer NVIDIA GPUs
Runtime changes fit
Memory behavior, quantization support and throughput vary by runtime.
optimized EXL3 quantized inference on modern consumer NVIDIA GPUs
Memory behavior, quantization support and throughput vary by runtime.