I link LLM AI to game localization, and prove it with eval. 14 years in game studio localization, 8 of them inside the publisher.
Existing MT metrics cannot see what breaks in game text. In two of the three games I measured, the human-written golden scored worse than a plain machine draft on a generic metric. So I build the evaluation per house, in the client's own terms, and use a high-low model mix so the cost lands where it should.
Cross-posted from LinkedIn, archived here.
Not a screenshot. An interactive table you can sort and inspect, with the numbers baked in at publish time.
| Arm | Glossary terms written as the glossary says |
House register matched |
Prompt tokens |
|---|---|---|---|
| bare model | 39 of 80 | 135 | 15,720 |
| + TM neighbours | 58 of 80 | 164 | 24,805 |
| full stack | 79 of 80 | 177 | 221,291 |
| the human golden | 70 of 80 | — | — |
Register out of 200 rows. The last line is the human translation the arms were scored against: on terminology the wired pipeline is above it, and that is the number a client can check. The token column is the other half of the story, because the last arm costs fourteen times the first. Open the full view, twelve arms with per-check breakdowns. Numbers only, no source text and no translations.
Mail sent to that address arrives. Replies come from another address for now.