Results: CPU study

An empirical comparison of 12 continual-learning methods on 6 benchmarks, run end to end on a 4-core CPU with contilab. The page has three experiments:

Experiment Config What it answers
Main study configs/experiments/cpu_main.yaml (+ cpu_main_seeds.yaml) All methods with default hyperparameters on 6 benchmarks
Tuned comparison configs/experiments/cpu_tuned.yaml The same methods when each one’s hyperparameters are chosen on validation accuracy (two MNIST benchmarks)
Smaller student configs/experiments/cpu_student.yaml Distilling full-size experts into a student with 5.3× fewer parameters

v0.2 update. The results were refreshed after these changes:

  • Metric fixes: cka_final, buffer_bytes, and the continual-evaluation curves of Joint.
  • New distillation defaults, chosen on validation accuracy in the tuned comparison. The v0.1 defaults under-trained the student.
  • A new method: teacher → student continual distillation.

Accuracy numbers of the other methods are unchanged.

Update after v0.2.0. Split CIFAR-100 joined the main study, and the three MNIST benchmarks have 6 seeds instead of 3, so exact paired tests there can reach p < 0.05. The runs of seeds 0–2 are unchanged. Two statements changed:

  • With 6 seeds, DER++ instead of multi-teacher distillation has the best mean on Split MNIST Class-IL. The two do not differ significantly.
  • Continual distillation no longer beats Naive on every Class-IL benchmark: it falls below Naive on Split CIFAR-100.

The tuned comparison has 6 seeds as well; its selections hold, except that 30 distillation epochs now edge out 10 on Split MNIST Class-IL.

Setup

   
Hardware 4 CPU cores (Intel Xeon @ 2.1 GHz, AMX/AVX512-BF16), 15 GB RAM, no GPU
Main study 12 methods × (3 MNIST benchmarks × 6 seeds + 3 other benchmarks × 3 seeds) = 324 runs: 312 done, 12 diverged (EWC λ = 100), 0 failed
Vision Split MNIST (Class-IL and Task-IL) and Permuted MNIST (10 tasks, 10k/2k samples per task) with an MLP 2×400, 1 epoch, SGD lr 0.03; Split CIFAR-10 and Split CIFAR-100 (Class-IL; 5 × 2 and 10 × 10 classes) with a reduced ResNet-18, 2 epochs, bf16 + torch.compile
Language text5 (AG News → Yelp → Amazon → DBpedia → Yahoo, 33 classes) with Pythia-70M, 1,000 train / 200 test texts per task, 1 epoch, AdamW lr 1e-4, batch 16
Memory budget 500 samples (vision) / 200 (text) for ER, DER++, A-GEM and both distillation methods
Distillation multi-teacher: TIES-merged student, targets routed to each sample’s teacher, 10 passes over the buffer; continual: one pass (train.epochs) over each task plus the buffer
Metrics tier full on Split MNIST Class-IL, extended elsewhere; ρ = 20 (MNIST) / 50 (CIFAR, text)

Deviations from the original design, all made to fit a CPU budget:

  • CIFAR-10 and CIFAR-100 were trained for 2 epochs instead of 3.
  • text5 used 1,000 training and 200 test texts per task, instead of 2,000 and 500.
  • A-GEM on text used a 64-sample reference batch instead of 256.

Accuracy overview

Final average accuracy (ACC), mean ± std over 6 seeds (MNIST benchmarks) or 3 seeds (CIFAR, text5). † = number of seeds that diverged.

Method Split MNIST (Class-IL) Split MNIST (Task-IL) Permuted MNIST (Domain-IL) Split CIFAR-10 (Class-IL) Split CIFAR-100 (Class-IL) text5 + Pythia-70M (Class-IL)
Naive 0.196 ± 0.000 0.980 ± 0.001 0.805 ± 0.011 0.181 ± 0.004 0.049 ± 0.003 0.097 ± 0.016
Joint 0.950 ± 0.002 0.993 ± 0.001 0.909 ± 0.006 0.398 ± 0.099 0.118 ± 0.016 0.529 ± 0.015
EWC 0.197 ± 0.001 0.983 ± 0.002 diverged (6/6) diverged (3/3) diverged (3/3) 0.105 ± 0.015
LwF 0.228 ± 0.005 0.990 ± 0.001 0.817 ± 0.010 0.173 ± 0.003 0.047 ± 0.001 0.105 ± 0.006
ER 0.858 ± 0.012 0.983 ± 0.001 0.869 ± 0.004 0.302 ± 0.017 0.059 ± 0.004 0.329 ± 0.038
DER++ 0.871 ± 0.014 0.991 ± 0.001 0.902 ± 0.002 0.256 ± 0.054 0.059 ± 0.009 0.397 ± 0.068
A-GEM 0.335 ± 0.013 0.984 ± 0.001 0.864 ± 0.006 0.181 ± 0.003 0.048 ± 0.002 0.100 ± 0.008
LoRA-per-task 0.184 ± 0.005 0.986 ± 0.002 0.619 ± 0.021 0.137 ± 0.006 0.015 ± 0.003 0.024 ± 0.002
Task Arithmetic 0.531 ± 0.019 0.979 ± 0.003 0.570 ± 0.038 0.158 ± 0.027 0.011 ± 0.001 0.034 ± 0.003
TIES 0.561 ± 0.020 0.983 ± 0.002 0.677 ± 0.012 0.166 ± 0.015 0.013 ± 0.002 0.033 ± 0.004
Multi-teacher distill. 0.863 ± 0.025 0.986 ± 0.001 0.773 ± 0.010 0.262 ± 0.023 0.040 ± 0.014 0.452 ± 0.013
Continual distill. 0.586 ± 0.070 0.980 ± 0.001 0.818 ± 0.005 0.210 ± 0.041 0.030 ± 0.004 0.331 ± 0.045

Critical-difference diagram

Ranking across the 6 benchmarks (ACC):

Rank Method
1.0 Joint
2.7 DER++
4.2 ER
4.7 Multi-teacher distillation
6.3 A-GEM
6.3 Continual distillation
6.5 LwF
7.8 Naive
8.8 TIES
9.5 EWC
9.8 LoRA-per-task
10.3 Task Arithmetic

The Friedman test gives p = 6.6·10⁻⁶, and the Nemenyi critical difference is 6.80 (α = 0.05). With 6 benchmarks the CD is still wide:

  • Joint is significantly better than Naive, TIES, EWC, LoRA-per-task and Task Arithmetic.
  • DER++ is significantly better than EWC, LoRA-per-task and Task Arithmetic.

Before v0.2, multi-teacher distillation ranked last (9.8) with under-trained defaults.

Paired tests on the MNIST benchmarks

Exact Wilcoxon signed-rank tests over the 6 seeds (seed i of one method paired with seed i of the other). With 6 pairs the smallest possible two-sided p is 0.031, reached when every seed agrees.

Benchmark A vs B Mean ACC (A / B) Seeds where B is higher p
Split MNIST Class-IL Naive vs EWC 0.196 / 0.197 6 / 6 0.031
Split MNIST Class-IL EWC vs LwF 0.197 / 0.228 6 / 6 0.031
Split MNIST Class-IL LwF vs ER 0.228 / 0.858 6 / 6 0.031
Split MNIST Class-IL ER vs DER++ 0.858 / 0.871 6 / 6 0.031
Split MNIST Class-IL DER++ vs Joint 0.871 / 0.950 6 / 6 0.031
Split MNIST Class-IL Multi-teacher distill. vs DER++ 0.863 / 0.871 3 / 6 0.69
Split MNIST Class-IL Multi-teacher distill. vs ER 0.863 / 0.858 3 / 6 0.84
Split MNIST Class-IL Task Arithmetic vs TIES 0.531 / 0.561 6 / 6 0.031
Split MNIST Class-IL Naive vs Continual distill. 0.196 / 0.586 6 / 6 0.031
Split MNIST Task-IL LwF vs DER++ 0.990 / 0.991 3 / 6 0.44
Permuted MNIST Naive vs LwF 0.805 / 0.817 5 / 6 0.094
Permuted MNIST Naive vs Continual distill. 0.805 / 0.818 5 / 6 0.063
Permuted MNIST Multi-teacher distill. vs Naive 0.773 / 0.805 6 / 6 0.031
Permuted MNIST Multi-teacher distill. vs Continual distill. 0.773 / 0.818 6 / 6 0.031
Permuted MNIST A-GEM vs ER 0.864 / 0.869 5 / 6 0.094
Permuted MNIST Task Arithmetic vs TIES 0.570 / 0.677 6 / 6 0.031

Findings

  1. Replay and multi-teacher distillation lead. The best non-Joint method on each benchmark:
    • DER++ on Split MNIST Class-IL (0.871), Split MNIST Task-IL (0.991) and Permuted MNIST (0.902). On Split MNIST Class-IL, multi-teacher distillation (0.863) does not differ significantly from it (p = 0.69); on Task-IL, neither does LwF (0.990, p = 0.44).
    • ER on Split CIFAR-10 (0.302), and ER and DER++ together on Split CIFAR-100 (0.059)
    • multi-teacher distillation on text5 (0.452)

    Average ranks: DER++ 2.7, ER 4.2, multi-teacher distillation 4.7. Replay also has the smallest stability gap, for example 0.150 (DER++) vs 0.995 for Naive on Split MNIST Class-IL.

  2. Multi-teacher distillation buys accuracy with experts.
    • It keeps every task expert: 11 MB of stored models on Split MNIST against 1.8 MB for a single model.
    • It trains every expert as well: 1,915 s per text5 run against 1,044 s for DER++.
    • In return it forgets little and is well calibrated across tasks, with BWT −0.079 and task-recency bias 0.05 on Split MNIST Class-IL.
    • On Permuted MNIST it falls below Naive on every seed (0.773 vs 0.805, p = 0.031): the 500-sample buffer leaves 50 samples for each of the 10 input distributions.
    • On Split CIFAR-100 it also falls below Naive (0.040 vs 0.049). The buffer holds 5 samples per class, and the student reaches a learning accuracy of only 0.099.
  3. Teacher → student continual distillation is the cheap variant. It never stores an expert: the student is the only model kept (1.8 MB). It beats Naive on every Class- and Domain-IL benchmark except Split CIFAR-100:
    • Split MNIST Class-IL 0.586 vs 0.196 (all 6 seeds)
    • Permuted MNIST 0.818 vs 0.805 (5 of 6 seeds, p = 0.063), and ahead of multi-teacher distillation (0.773) on every seed
    • CIFAR-10 0.210 vs 0.181
    • text5 0.331 vs 0.097, on par with ER
    • Split CIFAR-100: 0.030 vs 0.049, below Naive

    Old knowledge reaches the student only through self-distillation on the buffer, so errors compound task by task: BWT −0.499 and recency bias 0.46 on Split MNIST Class-IL. With 10 tasks of 10 classes on CIFAR-100 the recency bias reaches 1.0.

  4. Regularisation alone does not solve Class-IL. EWC (0.197) and LwF (0.228) stay at the Naive level (0.196) on Split MNIST Class-IL, with task-recency bias ≈ 1.0: almost every old-task sample is predicted as a newest-task class. Both are above Naive on all 6 seeds, but EWC by only 0.001. The same holds on text5 (0.105 vs 0.097) and Split CIFAR-100 (LwF 0.047 vs 0.049). Tuning does not change this (see Tuned comparison).
  5. EWC is sensitive to λ. With the literature default λ = 100 and an online (accumulated) Fisher, every seed diverged on Permuted MNIST, Split CIFAR-10 and Split CIFAR-100. With λ = 30, chosen on validation accuracy, Permuted MNIST trains stably and reaches 0.832, above Naive (0.805).
  6. A-GEM helps in Domain-IL but not in Class-IL. It reaches 0.864 on Permuted MNIST but only 0.335 on Split MNIST Class-IL, and stays at the Naive level on CIFAR-10, CIFAR-100 and text5. This matches the known weakness of gradient projection under class-incremental output heads.
  7. Merging trails replay.
    • TIES is above Task Arithmetic on every seed: 0.561 vs 0.531 on Split MNIST Class-IL, 0.677 vs 0.570 on Permuted MNIST (p = 0.031 each).
    • Merged models show little recency bias (0.13 on Split MNIST Class-IL), but learning accuracy drops to 0.78–0.80 because the experts interfere. On Split CIFAR-100 the merged models are near chance (0.011–0.013).
    • As a starting point for distillation, the merged model is what made distillation work (see the tuned comparison).
  8. Task-IL is easy at this scale. Every method reaches ≥ 0.979 on Split MNIST Task-IL.
  9. Language models trade task learning against general ability.
    • Full fine-tuning methods raise the WikiText perplexity of the original LM head by +4,500 to +14,000.
    • LoRA-per-task (+186) and TIES (+1,709) keep it low but barely learn the tasks (learning accuracy 0.21–0.25).
    • Multi-teacher distillation has the best balance: the highest text5 accuracy (0.452) at +3,643.
  10. LoRA-per-task on vision uses a frozen randomly initialised backbone (no pretraining), which limits it to 0.184 on Split MNIST Class-IL and 0.619 on Permuted MNIST. It is a meaningful baseline only with pretrained backbones.
  11. Split CIFAR-100 at 2 epochs is a floor test. Joint reaches only 0.118 on 100 classes, so every continual method sits between chance (0.010) and 0.06. Only replay (ER and DER++, 0.059) is above Naive (0.049). The rankings on this benchmark mostly reflect under-training, not forgetting.

Continual evaluation on Split MNIST Class-IL

Continual evaluation on Split MNIST Class-IL: accuracy of each task over training steps. Dashed lines are task boundaries. Naive, EWC, LwF and LoRA collapse to zero on old tasks immediately, while ER and DER++ keep them.

Accuracy matrices on Split MNIST Class-IL

Tuned comparison

Defaults from the literature are not always good settings. cpu_tuned gives every method with hyperparameters a small grid (method_sweep) on the two MNIST benchmarks:

  • EWC: λ ∈ {1, 3, 10, 30, 100, 300}
  • LwF: α ∈ {0.5, 1, 2, 4}
  • DER++: α ∈ {0.1, 0.3, 1} × β ∈ {0.5, 1}
  • TIES: k ∈ {0.1, 0.2, 0.5}
  • Multi-teacher distillation: init {θ₀, TIES} × routing {entropy, routed} × epochs {1, 10, 30}

For each method, the variant with the best mean validation accuracy over seeds is selected (spec §5.3); diverged seeds score 0. Optimiser, learning rate, epochs, batch size and memory budget stay fixed by the fairness rules. “Default” is the literature default from the main study (v0.1 defaults for distillation). This experiment has 6 seeds, like the MNIST benchmarks of the main study, so its default and untuned values are the same as in the overview above. † = number of diverged seeds.

Method Split MNIST Class-IL: selected default tuned Permuted MNIST: selected default tuned
Naive — 0.196 0.196 — 0.805 0.805
Joint — 0.950 0.950 — 0.909 0.909
EWC λ=300 0.197 0.201 λ=30 diverged 6/6 0.832
LwF α=1 0.229 0.229 α=0.5 0.816 0.828
ER — 0.858 0.858 — 0.869 0.869
DER++ α=1, β=1 0.871 0.899 α=0.3, β=1 0.902 0.906
A-GEM — 0.335 0.335 — 0.864 0.864
LoRA-per-task — 0.184 0.184 — 0.619 0.619
Task Arithmetic — 0.531 0.531 — 0.570 0.570
TIES k=0.1 0.561 0.591 k=0.2 0.677 0.677
Multi-teacher distill. epochs 30, TIES init, routed 0.223 0.871 epochs 30, TIES init, routed 0.097 0.786
Continual distill. — 0.586 0.586 — 0.818 0.818
  • Tuning changes the verdict for distillation. The v0.1 defaults trained the student for 16 SGD steps per task from θ₀ with entropy-mixed targets. A TIES-merged start with routed targets lifts Split MNIST Class-IL from 0.223 to 0.871 and Permuted MNIST from 0.097 to 0.786.
    • These settings are now the defaults, with 10 distillation epochs.
    • With 6 seeds, 30 epochs is selected on both benchmarks. It is 0.008 and 0.013 above 10 epochs (0.863 and 0.773) and takes three times the distillation time, so the default stays at 10.
  • Tuning does not rescue regularisation in Class-IL. No λ makes EWC differ from Naive on Split MNIST Class-IL (0.196–0.201), and no α lifts LwF above 0.230. On Permuted MNIST, λ = 30 avoids the divergence of λ = 100 and λ = 300 and gives 0.832 (Naive 0.805).
  • DER++ gains 0.03 with stronger replay terms (α = β = 1: 0.899 against 0.871). TIES gains 0.03 with sparser task vectors (k = 0.1: 0.591 against 0.561) on Split MNIST Class-IL.
  • Average ranks over the two benchmarks, using the selected variants:
    • Joint 1.0, DER++ 2.0, ER 3.5
    • A-GEM 6.0, multi-teacher distillation 6.0, continual distillation 6.5
    • EWC 7.5, LwF 7.5, TIES 7.5
    • Naive 9.5, TA 9.5, LoRA-per-task 11.5

    Two benchmarks are too few for significance (Friedman p = 0.13).

Teacher → smaller student

cpu_student distils the 2×400 MLP experts (478k parameters) into a 2×100 student (89.6k parameters, 5.3× fewer). A different architecture cannot start from merged expert weights, so both students start from their own initialisation. The fair full-size reference for multi-teacher distillation is therefore the θ₀-initialised variant of the tuned comparison. All rows use seeds 0–2.

Method Student Split MNIST Class-IL Permuted MNIST Stored model bytes (Split / Permuted)
Multi-teacher distill. full size (θ₀ init) 0.856 0.534 10.9 / 20.1 MB (experts + student)
Multi-teacher distill. 2×100 0.857 ± 0.010 0.464 ± 0.010 9.5 / 18.6 MB (experts + student)
Continual distill. full size 0.533 ± 0.038 0.816 ± 0.004 1.8 MB (student only)
Continual distill. 2×100 0.490 ± 0.034 0.798 ± 0.001 0.34 MB (student only)
  • On Split MNIST Class-IL the small student matches the full-size one (0.857 vs 0.856). Permuted MNIST (10 tasks, all 10 classes shared) costs it 0.07.
  • Continual distillation never stores an expert. Its 2×100 student keeps 98 % of the full-size accuracy on Permuted MNIST, in 0.34 MB. That is the smallest deployable result in this study.

Per-benchmark details

Means over 6 seeds (MNIST benchmarks) or 3 seeds (CIFAR, text5).

  • wall_time is training time per run, averaged over seeds 0–2 only: seeds 3–5 ran in a later session with a different CPU load. It includes expert training for consolidation methods and Fisher computation for EWC. Joint and both distillation methods were re-run for v0.2 while other jobs shared the CPU, and wall times of runs from different sessions vary by up to 2× with load.
  • buffer_bytes counts the stored samples (or the distillation transfer set) and DER++ logits.
  • state_bytes counts extra state kept between tasks: the EWC Fisher diagonal and anchor, and the LwF teacher.
  • EWC has no row on Permuted MNIST, Split CIFAR-10 and Split CIFAR-100 because every seed diverged at λ = 100.

Split MNIST (Class-IL)

Method acc bwt learning_acc min_acc stability_gap recency_bias buffer_bytes state_bytes wall_time
Naive 0.196 -0.992 0.989 0.000 0.995 1.000 0 0 11 s
Joint 0.950 — — — — 0.012 0 0 10 s
EWC 0.197 -0.990 0.989 0.000 0.994 0.998 0 3.65 MB 38 s
LwF 0.228 -0.953 0.990 0.025 0.970 0.956 0 1.82 MB 18 s
ER 0.858 -0.148 0.976 0.757 0.155 0.125 1.50 MB 0 14 s
DER++ 0.871 -0.143 0.986 0.764 0.150 0.120 1.52 MB 0 17 s
A-GEM 0.335 -0.818 0.989 0.095 0.865 0.819 1.50 MB 0 36 s
LoRA-per-task 0.184 -0.984 0.972 0.000 0.985 1.000 0 0 17 s
Task Arithmetic 0.531 -0.309 0.778 — — 0.130 0 0 15 s
TIES 0.561 -0.304 0.804 — — 0.126 0 0 16 s
Multi-teacher distill. 0.863 -0.079 0.926 — — 0.053 1.50 MB 0 21 s
Continual distill. 0.586 -0.499 0.985 — — 0.455 1.50 MB 0 23 s

Split MNIST (Task-IL)

Method acc bwt learning_acc min_acc stability_gap buffer_bytes state_bytes wall_time
Naive 0.980 -0.012 0.990 0.970 0.016 0 0 10 s
Joint 0.993 — — — — 0 0 10 s
EWC 0.983 -0.008 0.990 0.975 0.012 0 3.65 MB 39 s
LwF 0.990 -0.001 0.990 0.991 0.003 0 1.82 MB 18 s
ER 0.983 -0.006 0.987 0.980 0.009 1.50 MB 0 14 s
DER++ 0.991 -0.001 0.992 0.985 0.008 1.52 MB 0 18 s
A-GEM 0.984 -0.007 0.990 0.979 0.008 1.50 MB 0 35 s
LoRA-per-task 0.986 0.000 0.986 0.992 0.000 0 0 10 s
Task Arithmetic 0.979 -0.004 0.983 — — 0 0 15 s
TIES 0.983 -0.005 0.987 — — 0 0 15 s
Multi-teacher distill. 0.986 -0.002 0.987 — — 1.50 MB 0 22 s
Continual distill. 0.980 -0.010 0.988 — — 1.50 MB 0 24 s

Permuted MNIST (Domain-IL)

Method acc bwt learning_acc min_acc stability_gap buffer_bytes state_bytes wall_time
Naive 0.805 -0.104 0.898 0.763 0.025 0 0 20 s
Joint 0.909 — — — — 0 0 19 s
LwF 0.817 -0.085 0.893 0.779 0.022 0 1.82 MB 33 s
ER 0.869 -0.046 0.910 0.846 0.021 1.50 MB 0 27 s
DER++ 0.902 -0.004 0.906 0.884 -0.001 1.52 MB 0 34 s
A-GEM 0.864 -0.038 0.898 0.835 0.008 1.50 MB 0 67 s
LoRA-per-task 0.619 -0.100 0.709 0.581 0.042 0 0 52 s
Task Arithmetic 0.570 -0.150 0.706 — — 0 0 36 s
TIES 0.677 -0.108 0.774 — — 0 0 39 s
Multi-teacher distill. 0.773 -0.056 0.824 — — 1.50 MB 0 50 s
Continual distill. 0.818 -0.048 0.861 — — 1.48 MB 0 47 s

Split CIFAR-10 (Class-IL)

Method acc bwt learning_acc min_acc stability_gap recency_bias buffer_bytes state_bytes wall_time
Naive 0.181 -0.799 0.820 0.000 0.795 1.000 0 0 238 s
Joint 0.398 — — — — 0.052 0 0 421 s
LwF 0.173 -0.793 0.807 0.000 0.787 1.000 0 4.18 MB 439 s
ER 0.302 -0.670 0.838 0.063 0.686 0.647 5.86 MB 0 415 s
DER++ 0.256 -0.665 0.788 0.066 0.659 0.763 5.88 MB 0 930 s
A-GEM 0.181 -0.817 0.834 0.000 0.815 1.000 5.86 MB 0 3465 s
LoRA-per-task 0.137 -0.722 0.714 0.000 0.732 1.000 0 0 1635 s
Task Arithmetic 0.158 -0.240 0.350 — — 0.008 0 0 430 s
TIES 0.166 -0.238 0.357 — — 0.017 0 0 431 s
Multi-teacher distill. 0.262 -0.234 0.449 — — 0.202 5.86 MB 0 490 s
Continual distill. 0.210 -0.709 0.777 — — 0.723 5.86 MB 0 565 s

Split CIFAR-100 (Class-IL)

Method acc bwt learning_acc min_acc stability_gap recency_bias buffer_bytes state_bytes wall_time
Naive 0.049 -0.418 0.425 0.000 0.405 1.000 0 0 304 s
Joint 0.118 — — — — 0.158 0 0 303 s
LwF 0.047 -0.401 0.408 0.000 0.386 1.000 0 4.23 MB 448 s
ER 0.059 -0.406 0.424 0.003 0.343 0.972 5.86 MB 0 425 s
DER++ 0.059 -0.423 0.440 0.003 0.394 0.967 6.05 MB 0 619 s
A-GEM 0.048 -0.416 0.422 0.000 0.404 1.000 5.86 MB 0 1452 s
LoRA-per-task 0.015 -0.190 0.185 0.000 0.183 1.000 0 0 1208 s
Task Arithmetic 0.011 -0.077 0.080 — — 0.040 0 0 587 s
TIES 0.013 -0.090 0.094 — — 0.051 0 0 605 s
Multi-teacher distill. 0.040 -0.066 0.099 — — 0.025 5.86 MB 0 818 s
Continual distill. 0.030 -0.288 0.290 — — 1.000 5.80 MB 0 595 s

text5 + Pythia-70M (Class-IL)

Method acc bwt learning_acc min_acc stability_gap general_ppl_delta buffer_bytes state_bytes wall_time
Naive 0.097 -0.589 0.568 0.002 0.498 +7481 0 0 461 s
Joint 0.529 — — — — +4488 0 0 674 s
EWC 0.105 -0.588 0.575 0.003 0.494 +7541 0 340.94 MB 1199 s
LwF 0.105 -0.606 0.590 0.000 0.480 +8488 0 268.72 MB 816 s
ER 0.329 -0.294 0.564 0.292 0.068 +14037 0.39 MB 0 733 s
DER++ 0.397 -0.231 0.582 0.284 0.126 +6036 0.42 MB 0 1044 s
A-GEM 0.100 -0.602 0.582 0.001 0.505 +7056 0.39 MB 0 1563 s
LoRA-per-task 0.024 -0.233 0.210 0.000 0.187 +186 0 0 755 s
Task Arithmetic 0.034 -0.268 0.249 — — +5743 0 0 628 s
TIES 0.033 -0.277 0.254 — — +1709 0 0 674 s
Multi-teacher distill. 0.452 -0.050 0.492 — — +3643 0.39 MB 0 1915 s
Continual distill. 0.331 -0.260 0.539 — — +12459 0.39 MB 0 916 s

Throughput

Training throughput

See Performance. For the MLP, training 8 seeds in one vmap ensemble gives 3.8× the throughput of one seed (40.7k vs 10.8k samples/s). For the reduced ResNet-18, bf16 + torch.compile gives 2.3×.

Success criteria

# Criterion Verdict Evidence
1 Smoke profile: every method and consolidation path finishes within 60 s; all tests pass Met 66 smoke runs within the 60 s test on an idle 4-core CPU; 188 tests pass on Python 3.11 and 3.13 in CI (5 network tests skipped by default)
2 CPU profile matrix completes in the session and produces a report Met, with the budget deviations listed above 324/324 runs finished (312 done, 12 diverged, 0 failed); report.md / report.html generated
3 Split MNIST Class-IL ordering Naive < {EWC, LwF} < {ER, DER++} < Joint (mean ACC) Met, on every seed 0.196 < 0.197 / 0.228 < 0.858 / 0.871 < 0.950. Each step holds on all 6 seeds (exact Wilcoxon p = 0.031, the minimum for 6 pairs). EWC is ahead of Naive by only 0.001
4 contilab bench reports vmap / compile speed-ups Met MLP vmap E=8: 3.8×; ResNet bf16 + compile: 2.3×; ResNet vmap: no gain (reported)
5 Equivalence and consistency tests pass Met vmap members = single runs, compile = eager, EWC(λ=0) / ER(M=0) / DER++(0,0) = Naive, merge identities, resume = uninterrupted run

Limitations

  • Statistical power: the MNIST benchmarks have 6 seeds, enough for an exact paired test to reach p < 0.05 when every seed agrees. CIFAR-10, CIFAR-100 and text5 have 3 seeds: enough for means and spreads, not for per-benchmark significance tests.
  • Scale: this is a CPU-scale study. CIFAR-10 and CIFAR-100 at 2 epochs (Joint 0.398 and 0.118) and text5 at 1,000 texts per task are far below their converged regimes; read absolute numbers as relative comparisons.
  • Tuning scope: hyperparameters were tuned only on the two MNIST benchmarks. CIFAR-10, CIFAR-100 and text5 use the defaults, including the distillation defaults chosen on MNIST.
  • Head assembly in merging: Class-IL merging assembles classifier rows from different experts, leaving logit-scale mismatch between tasks.
  • Distillation cost: both distillation methods first train one expert per task. Their wall time and FLOPs include that, but a deployment that already has the experts would only pay the distillation.

Data

The per-run results behind every table on this page are in docs/assets/results/data/: one CSV per experiment, one row per run with all scalar metrics. The same files are the Hugging Face dataset jonpark0/contilab-cpu-study, with a dataset card; load one with datasets.load_dataset("jonpark0/contilab-cpu-study", "cpu_main").

Reproduce 

contilab run configs/experiments/cpu_main.yaml        # ~22 h of training on 4 CPU cores (resumable)
contilab run configs/experiments/cpu_main_seeds.yaml  # seeds 3–5 on the MNIST benchmarks, same directory
contilab run configs/experiments/cpu_tuned.yaml       # 456 runs, ~2 h
contilab run configs/experiments/cpu_student.yaml     # ~10 min
contilab report runs/cpu_main