19 Commits

Author SHA1 Message Date
Joey Eamigh
67beaede45
quantization + onnx sweeps
Phase 10.8: torchao/bnb quant sweep on iter1-independent. bf16 already
optimal; torchao int8-wo gives -19% VRAM at no F1 cost; all 4-bit
variants collapse (ModernBERT-large too quant-sensitive).

Phase 10.9: ONNX export + ORT eval. Legacy exporter only working path
(dynamo adds 56 Memcpy nodes); ORT fp32 -22% latency vs torch via
kernel fusion but bf16+flash-attn-2 still wins; fp16 broken on rotary;
dynamic int8 silently CPU-fallback + 0.5 F1 collapse.

Driver scripts wired to bun run py:quant / py:onnx; full reports at
results/eval/{quant,onnx}/REPORT.md.
2026-04-07 05:10:38 -04:00
Joey Eamigh
07dc3d6133
testing clspool and dapt on new architecture 2026-04-07 00:51:48 -04:00
Joey Eamigh
4f5c88d94a
trying ensenble and nofilter versions of the model 2026-04-06 15:50:15 -04:00
Joey Eamigh
dac00f90db
data: 246 new 2026-04-05 18:44:40 -04:00
Joey Eamigh
2e932bc327
working model!!!!! 2026-04-05 15:37:50 -04:00
Joey Eamigh
42f8849b14
first finetune attempt 2026-04-05 12:16:16 -04:00
Joey Eamigh
7b660fe361
roll back to python 3.13 to fix everything lol 2026-03-30 21:25:46 -04:00
Joey Eamigh
8190950f1a
fix some bugs for tapt 2026-03-30 20:44:10 -04:00
Joey Eamigh
75ab92628b
wrong pytorch somehow installed itself again lol 2026-03-30 20:15:16 -04:00
Joey Eamigh
3292980d33
tapt setup 2026-03-30 19:46:20 -04:00
Joey Eamigh
c0273c9e2e
adding dvc backend so data can be cleanly pulled 2026-03-30 16:53:35 -04:00
Joey Eamigh
1dce1ccb73
updating narrative at checkpoint resume 2026-03-30 15:20:59 -04:00
Joey Eamigh
669632af7b
tweaking checkpoint saves 2026-03-30 11:41:20 -04:00
Joey Eamigh
fe5155ab6d
pretraining config for run 2026-03-29 23:55:49 -04:00
Joey Eamigh
e5f89ffabb
caching in the pipelines 2026-03-29 21:17:50 -04:00
Joey Eamigh
99cf4a606c
thread tokenization and chunking 2026-03-29 21:03:11 -04:00
Joey Eamigh
9d41dd199f
DAPT and precleaning for DAPT 2026-03-29 20:33:39 -04:00
Joey Eamigh
a9a7d59603
DAPT/TAPT scaffolding 2026-03-29 16:12:19 -04:00
Joey Eamigh
78d1f978de
initial scrape and tag 2026-03-28 20:39:36 -04:00