Qwen2.5-1.5B EXL3 Quantization Benchmark on RTX 3090

This was a small practical test of EXL3’s speed and memory footprint before running a deeper quality evaluation. It is intentionally narrow: one checkpoint, one GPU, one fixed speed prompt, and four deterministic capability prompts.

Headline result

EXL3 4.0 BPW reached a 156.55 token/s median decode rate, 5.80× the BF16/Transformers baseline, while reducing reported model memory from 2,945 MiB to 929 MiB.

Measured results

Model	Disk size	Model memory	Prefill	Decode	vs BF16	Smoke test
BF16 / Transformers	3.10 GB	2,945 MiB	9,458 tok/s	26.97 tok/s	1.00×	3 / 4
EXL3 4.0 BPW	1.31 GB	929 MiB	4,215 tok/s	156.55 tok/s	5.80×	3 / 4
EXL3 3.0 BPW	1.15 GB	764 MiB	8,127 tok/s	141.02 tok/s	5.23×	4 / 4

5.80×4.0 BPW median decode speedup

68.5%4.0 BPW model-memory reduction

74.1%3.0 BPW model-memory reduction

Quantized models

Both public Hugging Face repositories include the EXL3 weights, tokenizer and configuration files, Apache-2.0 license, conversion details, and all six raw BF16/quant benchmark JSON files.

EXL3 4.0 BPW1.31 GB on disk · 929 MiB reported model memoryDownload model Inspect raw runs

EXL3 3.0 BPW1.15 GB on disk · 764 MiB reported model memoryDownload model Inspect raw runs

Method

Checkpoint: Qwen/Qwen2.5-1.5B-Instruct.
Hardware: NVIDIA RTX 3090 with 24 GB VRAM.
Software: PyTorch 2.10.0+cu128 and ExLlamaV3 v1.4.4 at 17bc392.
Conversion: 250 calibration rows × 2,048 tokens, mul1, and a 6-bit lm_head.
Speed: the median of three runs using the same 512-token prompt and 255 timed greedy decode tokens.
Memory: runtime-reported model allocation, excluding KV cache and general process overhead.

Capability smoke test

The four prompts checked exact arithmetic, exact phrase following, compact JSON extraction, and syntactically valid Python. BF16 and EXL3 4.0 BPW both missed the exact-phrase prompt by answering “Thank you.”; EXL3 3.0 BPW passed all four.

This does not show that the 3.0 BPW model is more capable. Four prompts and one deterministic generation per model are only a regression smoke test. Quantization can move a borderline output in either direction.

What the result does and does not establish

The decode and memory result is encouraging, but the three-run timing samples had substantial variance. Treat this experiment as directional, not publication-grade.

BF16 ran through Transformers and EXL3 ran through ExLlamaV3. The speedup is therefore a practical format-plus-runtime result, not the isolated effect of weight quantization. The next useful experiment is held-out perplexity and KL divergence against BF16, followed by fixed-concurrency tests.

Next measurement

Use identical token positions to measure perplexity and token-weighted KL divergence against BF16, then test concurrency 1 and 4 with fixed prompt/output lengths and warm-up policy.