Browser systems · Experimental research
LlamaJEE.js
Browser-first local JEE solving, experimental fine-tuning, and the checks that kept the unchanged base in place.
Desktop experimental preview · Large model downloads · Image recognition remains unreliable.
Problem & motivation
Can a JEE study tool run on the learner’s device, and can fine-tuning make its answers more reliable? India’s Joint Entrance Examination (JEE) questions require both a correct result and defensible working. An extra correct option is not enough if the reasoning gets worse.
The project therefore has two connected parts: a browser solver and a saved record of controlled model experiments.
Technical approach
- Browser interfaceQuestion, optional image, model choice
- Worker & tokenizerTransformers.js tokenization; generation away from the interface
- ONNX Runtime WebGPUCustom 9B graphs, original compressed weights, separate adapter inputs
The 9B runtime preserves the original MLX affine 4-bit weights and supplies low-rank adapter matrices separately. It is a custom ONNX engine, not a standard Transformers.js model loader. Static model files download from Hugging Face; inference runs on the visitor’s device.
The original Apple Silicon lab remains available through a local Python server and MLX. The browser also offers a separate unchanged 0.8B option; the results below concern the original 9B experiments.
Experiments & results
Five archived adapters vary training families, adapted layers, and checkpoint step. A family is one source question with related teaching exercises. The latest comparison uses the same reused 12-question development slice for every row.
| Condition | Strict answers / 12 | Procedure / 24 | Token caps |
|---|---|---|---|
| Unchanged base | 7 | 17 | 5 |
| 30 families · 3 layers · step 30 | 7 | 16 | 4 |
| 30 families · 3 layers · step 60 | 7 | 18 | 5 |
| 30 families · 3 layers · step 90 | 7 | 17 | 5 |
| 30 families · 16 layers · step 60 | 8 | 14 | 2 |
| 60 families · 3 layers · step 60 | 7 | 16 | 5 |
No adapter was promoted as a generally better solver. The wider-layer adapter gained one strict answer but lost three procedure points. Procedure points use valid = 2, partly valid = 1, invalid = 0; these are coordinator judgments, not independent expert certification. Token caps count responses that reached the output limit.
Conversion parity is a different test
The saved conversion report records 18/18 full CPU sequences and 10/10 Chrome sequences matching their controls on synthetic fixtures. These checks establish consistency on those inputs, not JEE accuracy. Public-site execution was also recorded for the unchanged base and one adapter.
Failures & limitations
The faithful conversion preserves mistakes: synthetic images of “2 + 3 = ?” and “8 − 1 = ?” produced 4 and 2 in both the original and converted model. Screenshot recognition remains unreliable.
The reused development questions cannot establish general exam performance. Earlier training runs also showed short-output failures and later-checkpoint regressions; a training curve alone was not a reliable selection rule.
The desktop preview downloads up to 5.7 GB for text weights and adapters, plus 1.84 GB for optional vision. Reported browser tests used a 64 GiB Mac. Minimum memory requirements and phone support remain unverified.
Experiment history · Conversion failures and hardware limits.
What I learned & next steps
I learned to keep answer counts, procedure quality, and completion failures together. A narrow improvement can hide a regression. Exporting a model also requires checking numerical details and stopping behavior, not only whether the interface loads.
I would require fresh confirmation questions, no-regression checks, and reproducible export parity before promoting an adapter. The 4B curriculum work is a data milestone: the recorded program has not run a 4B optimizer update.
Inspect & reproduce
Start with history.csv and the chart notes; neither requires model downloads. The local lab guide documents the MLX runtime. The browser release report separates synthetic checks from public-site execution.
The public archive excludes private questions and optimizer state. Reading these artifacts does not reproduce training.