Testing and benchmarking

Test layers

Command What it checks
make ISA=X test C unit tests: test_arena, test_quant, test_kernels (including the packed and batched kernels), test_load, test_sampler, test_server (JSON, UTF-8, stop strings), test_threads, test_tokenizer
make ISA=X e2e tests/test_e2e.sh runs ml_run against the numpy reference, the scalar build, and CLI edge cases, and ml_server through tests/server_check.py
python3 -m pytest tests the Python tools: format round trips, the reference forward, the model and tokenizer converters, RoPE scaling
tests/check_tokenizer.py ml_tok against the Hugging Face tokenizers library (run by make test-all, see below)
make guard-test x86 ISA guard, using qemu-x86_64 -cpu core2duo/Nehalem/SandyBridge/Haswell
make guard-test-arm AArch64 ISA guard, using qemu-aarch64 -cpu cortex-a53/cortex-a76/neoverse-n2/max
make ISA=sve CROSS=aarch64-linux-gnu- test-sve the SVE build at 128-bit and 512-bit vector length
make test-all everything above for every ISA, then a summary table

test_kernels

test_kernels checks each kernel twice:

  • the kernel the build bound (ml_k_<op>) against the scalar kernel;
  • the scalar kernel against a naive double-precision reference in the test.

It runs both checks in every build, so one source tests every ISA.

What Tolerance
Integer matvec with all scales set to 1 bit-exact
Scaled matvec, rope, axpy relative 1e-5 of the sum of term magnitudes
quantize_q8 (zero blocks, tiny blocks, ties) bit-exact against scalar
Other float kernels relative 1e-5
softmax also finite for ±1e4 and -1e30 inputs, and uniform when every input is -inf

The first lines of the output name the kernel each op resolved to, for example matvec_q8 -> ml_k_matvec_q8_avx512vnni.

End-to-end tests

tests/test_e2e.sh ISA BUILD [RUN] checks the following:

  • Logits: for random Q8 and Q4 models, the logits match tools/ref_forward.py within 1e-3 relative error, and the argmax is the same at every position.
  • Generated text: greedy and seeded sampling produce text identical to the host scalar build.
  • Context limit: generation stops at --ctx (default max_seq, at most 4096), and --dump-logits with more tokens than that exits with code 2, as does a --ctx above max_seq. A shorter --ctx gives the same logits.
  • Linear attention (Qwen3.5): random hybrid models (hyb_q8, hyb_q4) match tools/ref_forward.py within 1e-3 over 16 positions, packed and row-major; over 150 positions (hybl_q4, past the size where the delta rule splits across threads) -t 3, --batch 1 and --batch 5 give the same logits bit for bit, the F32 and Q8 caches stay within the 5% criterion, and a generation equals one prefilled token by token.
  • Server: tests/server_check.py starts ml_server on a random port and checks every endpoint over HTTP: a greedy chat reply equals ml_run --chat’s, streamed deltas add up to the same reply, a repeated or multi-turn request gets the same reply whatever the cache held, stop strings cut the reply, errors come in OpenAI’s format, Expect: 100-continue works, a client that leaves mid-stream does not stop the server, a model with linear attention reuses only a continued request and replies the same either way, and an API key is enforced. With the openai package installed, its client is tried too.
  • Threads: -t 3/4/7 give logits identical to -t 1, and ml_run reports the thread count it used.
  • Packing: builds with packed kernels do pack (packed: G matches --help), and --no-pack also matches the reference.
  • Batched prefill: --batch 1, --batch 5 and the default produce byte-identical logits, and generated text is the same with and without batching. Both checks also pass with --kv q8, where -t 3 must also match -t 1.
  • Q8 and F16 KV caches: logits with --kv q8 and --kv f16 match tools/ref_forward.py run with the same option (it emulates both caches), under the --kl criterion described below. Batched prefill matches --batch 1 with either cache.
  • Long context: a 160-position model runs 150 positions, so attention crosses its 64-position blocks. The host scalar build must match the reference within 1e-3 (F32 cache) and 5% (Q8, F16). The build under test is held to 5% without the argmax check (compare_logits.py --any-argmax): builds with FMA round some activations differently and drift by up to 2% over 150 positions, while a wrong block offset or stride garbles the logits (85% error).
  • Real model (make test-all, needs network): stories15M logits from the scalar and avx2 builds, packed and --no-pack, are compared with the numpy reference using compare_logits.py --kl 1e-2. That mode checks that the argmax matches, at least 4 of the top 5 tokens agree, and the KL divergence is small. This criterion is used because a real model can have an activation sitting exactly on a quantization rounding tie. Two valid implementations can then round it differently, which moves logits by up to about 2% while the distribution stays the same (KL about 1e-3). The random-model tests keep the strict 1e-3 relative bound. The avx2 build with --kv q8 is held to the same KL bound against the F32 reference (KL about 1e-3).
  • Chat: --chat replies match the scalar build, -i equals one line of stdin, and a conversation that outgrows the context stops with a message.
  • Bad input: malformed options (for example -n 5x, ids out of range or a NaN temperature) exit with code 2, and --help exits with code 0.

Tokenizers

  • test_tokenizer: encodes texts with two small tokenizers trained by the tokenizers library (tests/data/spm_tiny.json, bpe_tiny.json, one of each kind) and checks that the ids match the library’s, that decoding, byte fallback and added tokens work, and that damaged files are rejected.
  • tests/check_tokenizer.py: encodes and decodes a corpus with ml_tok and with the tokenizers library and requires identical results. The corpus holds contractions, numbers, accents, CJK, Korean, Hindi, Arabic, emoji, every kind of white space and newline, code, added tokens in the text, long repeats, 200 random strings and whole documents. make test-all runs it on the real TinyLlama and Llama 3.2 tokenizer.json files (network needed) and reports SKIP when the tokenizers package is missing; CI installs it.

Loader robustness

test_load patches a copy of a valid model and checks each failure in turn:

  • truncated file;
  • empty file;
  • bad magic or version;
  • dim of 48 (not a multiple of 32);
  • bad head counts;
  • unknown wtype or rope_type;
  • a Q8 quant equal to -128;
  • vocab < 3;
  • dimensions whose sizes would overflow.

It also loads a model whose KV cache is larger than 4 GB. On 32-bit ARMv7 that model must fail with ML_E_NOMEM instead of corrupting memory.

Running ARM tests on x86 

sudo apt-get install qemu-user gcc-aarch64-linux-gnu gcc-arm-linux-gnueabihf
make ISA=neon CROSS=arm-linux-gnueabihf- test e2e
make ISA=neon_dotprod CROSS=aarch64-linux-gnu- test e2e
make ISA=neon_i8mm CROSS=aarch64-linux-gnu- test e2e      # qemu -cpu max has i8mm

Sanitizers 

make ISA=avx2 EXTRA_CFLAGS="-fsanitize=address,undefined -g" EXTRA_LDFLAGS="-fsanitize=address,undefined" test
make ISA=avx2 THREADS=pthread EXTRA_CFLAGS="-fsanitize=thread -g" EXTRA_LDFLAGS=-fsanitize=thread test

CI

.github/workflows/ci.yml has two jobs:

  • matrix (ubuntu-24.04) runs make test-all. That covers:
    • x86 builds natively; ISAs the runner’s CPU lacks are built and reported as SKIP;
    • ARM builds under qemu;
    • CMake and MinGW builds;
    • pytest;
    • a download of stories15M from llama2.c, followed by text generation.
  • arm64 (ubuntu-24.04-arm) runs scalar, neon, neon_dotprod, neon_i8mm and sve tests natively on ARM hardware (each only if the CPU has the extension), then prints a benchmark.

Benchmarking 

python3 tools/gen_random.py --out bench.mlb --dim 512 --layers 6 --heads 8 --kv-heads 4 \
    --hidden 1376 --vocab 4096 --max-seq 128
make ISA=avx2 BENCH_MODEL=bench.mlb bench
build/avx2-pt/ml_bench bench.mlb -n 128 -t 4        # THREADS=pthread build

ml_bench reports:

  • matvec ns/iter, GB/s and GOPS, for 768² and 4096² matrices in Q8 and Q4, and for the FFN shapes of a 0.5B model (4864×896, 896×4864);
  • the same for the packed kernels (matvec_q8_pk, matvec_q4_pk) and the batched ones (matmul_q8_pk, matmul_q4_pk, 128 tokens per call), when the build has them;
  • quantize_q8 time for 4096 values;
  • tokens/s on the model you pass, with greedy decoding.

To compare two versions, build both and run them alternately on one machine, so that the machine’s noise falls on both:

git worktree add ../krafla-main main
python3 scripts/bench.py --new . --base ../krafla-main --model bench.mlb --reps 5

scripts/bench.py builds ml_bench for every ISA the CPU runs and prints the best of the alternating runs and the new/base ratio as Markdown. The Benchmark workflow (.github/workflows/bench.yml) does the same on GitHub’s x86-64 and AArch64 runners; start it by hand from the Actions tab. It compares the branch it runs on against base (default main), with a random Qwen2.5-0.5B-shaped model or the real one, and writes the tables to the run’s summary.

All x86 results below are from a 4-vCPU VM with AVX-512 VNNI. The kernel and prefill tables were measured in one session, as the best of several runs pinned to one core. The VM is shared, so expect ±10% noise.

Kernel throughput, single thread (ml_bench, 4096×4096, GOPS):

ISA Q8 row-major Q8 packed Q4 row-major Q4 packed
sse2 24.7 28.5 18.3 34.5
sse41 35.6 33.5 22.0 47.6
avx2 47.2 48.5 33.8 76.0
avxvnni 47.1 51.3 34.8 77.7
avx512vnni 50.3 52.0 51.3 91.3

At this size Q8 is limited by memory bandwidth, so packing mostly helps Q4. When the matrix fits in cache (768×768), the packed kernels reach 92 GOPS on avx2 and 129–139 GOPS on avx512vnni.

Decoding, dim-2048 model (4 layers, hidden 5632, vocab 32000), packed weights, tok/s, without and with the software prefetch of the weight stream (-DML_USE_PREFETCH=0 builds the former):

ISA Q8 -t 1 Q4 -t 1 Q8 -t 4 Q4 -t 4
sse2 22.6 → 42.3 38.2 → 56.4    
sse41 23.3 → 49.3 40.9 → 81.7    
avx2 29.3 → 52.2 54.9 → 122.4 125 → 242 369 → 451
avxvnni 34.6 → 53.3 56.1 → 141.6    
avx512vnni 46.3 → 58.0 73.6 → 140.1 192 → 254 482 → 503

See Prefetch for why the packed matvecs prefetch 4 KB ahead. On the Arm runner (Neoverse) the same prefetch made NEON dotprod Q8 decoding 26% slower (470 against 638 tok/s, interleaved best of 3) and left Q4 unchanged, so ARM builds leave it off.

Prompt prefill, 64 tokens, dim-2048 model (4 layers, hidden 5632, vocab 32000), packed weights, -t 1 (tok/s):

ISA Q8 --batch 1 Q8 batched Q4 --batch 1 Q4 batched
sse2 34.9 100.8 73.3 92.9
sse41 30.4 114.4 92.9 178.2
avx2 41.4 202.7 156.4 394.9
avxvnni 44.5 311.9 152.7 484.2
avx512vnni 67.4 339.3 211.1 558.3

Packing the SSE2 build raised its Q8 prefill from 51 to 93 tok/s and its Q8 decode from 16 to 24 tok/s; Q4 decode went from 23 to about 40 tok/s.

The batched kernels run each tile through a body instantiated for a constant token count (PK_TILES, ML_UNROLL), so the per-token accumulators stay in registers. Before that, GCC kept them on the stack, and batched prefill was 1.2–1.6× slower (avx2 Q4 292, avxvnni Q8 197, avx512vnni Q4 384 tok/s).

Long context, dim-2048 Q4 model (4 layers, 32 heads, 8 KV heads, hidden 5632), avx2, -t 1, 4096 positions decoded from a 1-token prompt:

Attention version attention time share of forward tok/s over 4096 positions
one call per position (before) 45.2 s 35% 31.7
row blocks, GQA-shared (--kv f32) 19.0 s 19% 40.9
Q8 cache (--kv q8) 13.4 s 14% 43.8

ARM, single thread (GitHub ubuntu-24.04-arm runner, Neoverse-class cores). These numbers come from the CI benchmark step, which publishes them as annotations. The model is dim 1024, 4 layers, hidden 2816.

ISA Q4 4096² GOPS (row-major → packed) Q4 decode tok/s Q4 prefill tok/s Q8 prefill tok/s
neon (no dot product) 16.7 → 30.6 152 → 267 184 → 326 224 → 290
neon_dotprod 29.5 → 67.6 261 → 550 320 → 948 436 → 1144
neon_i8mm 29.5 → 67.6 261 → 550 320 → 1167 433 → 1418

The neon_dotprod packed kernels use vdotq_laneq_s32, so each step reads its 4 activation bytes from a vector lane instead of broadcasting them. neon_i8mm decodes with the same kernels and prefills on smmla, 23–24% faster than sdot.

Perplexity

tools/perplexity.py scores a text through ml_run --dump-logits, for comparing weight formats or KV cache types on a real model:

python3 tools/perplexity.py build/avx2/ml_run llama.mlb llama.mlt wikitext-2-test.txt \
    --ml-tok build/avx2/ml_tok --ctx 512 --chunks 4 -t 4

Docs site and wiki

docs/ is also the GitHub Pages site. In the repository settings, under Pages, it is deployed from branch main, folder /docs. Its configuration is docs/_config.yml and its landing page docs/index.md. To preview it with the github-pages gem: PAGES_REPO_NWO=JonPark0/krafla bundle exec jekyll serve -s docs.

The GitHub wiki is generated from README.md and docs/; edit those, never the wiki. .github/workflows/wiki.yml runs tools/sync_wiki.py on every push to main that touches them and pushes the result with GITHUB_TOKEN; it can also be started by hand from the Actions tab. To sync without Actions:

git clone https://github.com/JonPark0/krafla.wiki.git wiki
python3 tools/sync_wiki.py wiki --repo JonPark0/krafla
cd wiki && git add -A && git commit -m "Sync from docs" && git push