S0 → S1: Fine-Tuning

The translations got slightly better at looking like the reference, but not at actually meaning the right thing. Runs A and B improved BLEU (how similar the output looks to the reference), but COMET (whether the meaning is correct) barely moved — only +0.0012 on test. The model learned to mimic the style of the training data without deeper understanding.

The model got worse at using the right technical terms. Even after training on 24K translation pairs, the model didn’t learn to use our approved glossary terms like “パスワード” for “password”. Terminology accuracy actually dropped in Runs A and B. The training data taught general fluency, not domain-specific vocabulary.

We found you can’t have both. In Run D, we changed the training so the model only learned from the Japanese output (ignoring the English input during loss calculation). This was the only run that improved terminology — but it broke everything else. BLEU dropped from 22 to 5. The model got better at producing Japanese words but forgot how to connect them to the English meaning.

Training a model this small is like walking a tightrope. Run B crashed partway through training with NaN errors (the math literally broke). With only 500 million parameters, there’s very little room for the model to learn new things without destabilising what it already knows.

Bottom line: fine-tuning a small model gave us small improvements. The real gains came from giving the model access to external knowledge (S2 retrieval) and letting a bigger model make decisions (S3 agentic).

• Changed: nothing — this was the first stable run using the base config
• Result: best dev metrics (BLEU +2.17, COMET +0.0051), stable training, no NaN
• But: terminology accuracy dropped -0.0067
• A separate final stable rerun (using the same base config) was used for the test_v1 S1 evaluation

• Changed: learning rate 5e-6 → 3e-6 (how fast the model learns per step — lower = slower, more cautious updates)
• Why: Run A had worked, but earlier attempts had crashed mid-training. We thought slowing down the learning might prevent that
• Result: worse on every metric (BLEU +1.18 vs +2.17) and still crashed with NaN anyway
• Takeaway: learning slower didn’t help — the model just absorbed less from the training data in the same number of steps

• Changed: how the model learns from each example. Normally it learns from the entire conversation (English input + Japanese output). In Run D, we told it to only learn from the Japanese output and ignore the English side during training
• Why: we thought focusing purely on the Japanese would teach it better Japanese vocabulary and terminology
• Win: it worked for terminology — the only run where the model used more correct glossary terms (+0.0095)
• Cost: but the translations became nearly unusable. BLEU dropped from 22 to 5 and COMET from 0.78 to 0.61
• Takeaway: the model could still see the English text, but since it wasn’t being trained on that connection, it lost the ability to translate coherently. It learned better Japanese words but forgot what they were supposed to mean in context

• What is NaN? It stands for “Not a Number”. During training, the model does millions of math operations. If one of those calculations produces an impossible result (like dividing by zero), it outputs NaN instead of a real number
• Why is it a problem? Once NaN appears in one calculation, it spreads like a virus. Every subsequent calculation that touches it also becomes NaN. The model’s learned knowledge gets corrupted and it starts producing nonsense
• What happened in our runs? Training would look completely normal for most of the run — loss going down, metrics improving. Then suddenly, without warning, the loss would spike to NaN and the model would be broken
• Why is this tricky? Because the early checkpoints (snapshots we save during training) look fine. You think the run succeeded until you check the later checkpoints and realise they’re corrupted
• Why does this happen with small models? We compressed the model to 4-bit precision to fit it on our hardware. This means every number is stored with less accuracy — like rounding to fewer decimal places. With less room for precision, small errors can snowball into NaN
• How did we deal with it? We saved checkpoints frequently (every 100 steps), set a strict early-stopping rule, limited training to 300-600 steps, and used gradient clipping (capping how big each update can be). If a run hit NaN, we threw away the broken checkpoints and kept the last good one

• LoRA rank (16): rank controls how much the adapter can learn. On a small 0.5B model, making the adapter too big (high rank) risks the model just memorising the training data instead of learning general patterns
• Batch size (32 effective): this is how many examples the model sees before updating its weights. 32 was the most our hardware could handle — going higher would have required a bigger GPU
• Max sequence length (2048): the maximum length of text the model can process at once. Most of our translation pairs were short (~100 characters), but some longer ones needed up to 500. We set a generous ceiling to be safe
• Training length (1 epoch, 300-600 steps): we deliberately kept training short because longer runs crashed with NaN. One pass through half the data was safer than multiple passes through all of it
• Dropout (0.05), clipping (0.1): these are safety mechanisms — dropout randomly disables parts of the model during training to prevent memorisation, clipping limits how big each weight update can be. Both were already set conservatively
• Our approach: we changed one or two things per run so we could tell exactly what caused each result, rather than changing everything at once and guessing