fix(tokenizers): align Mistral and Gemma end-token control with deprecation warning (#2977)
Sole Author: Patrick Ribbsaeter (@patrickswedish) · Immutable Commit SHA: 2f2401a40664973ff8187bb9fef44c244f9092c2
LLM fine-tuning tokenizer fix aligning add_eos sequence termination controls across Mistral and Gemma tokenizers so sequence packing obeys explicit end-token configuration.
PR #2977
100% CI GREEN · 3/3
2f2401a
Immutable Git SHA
Python
Ai Infrastructure
3/3 CI Green
CI & Test Matrix
01 · Engineering problem
The failure mode
Mistral and Gemma chat tokenizers handled turn-end EOS/EOT token generation inconsistently at prompt boundaries during supervised fine-tuning.
02 · Technical response
What changed
- 01Aligned end-of-turn token emission and masking invariants across Mistral and Gemma tokenizer templates in TorchTune.
- 02Added tokenizer parity unit tests verifying exact token ID sequences and loss masks.
03 · Verified outcome
Evidence-backed result
- ✓Passed 100% of TorchTune CI checks with zero failures on commit 2f2401a40664973ff8187bb9fef44c244f9092c2.
04 · Cryptographic Git record
Immutable source evidence
Independent open-source engineering authored solely by Patrick Ribbsaeter (@patrickswedish). Company and project names identify the public repository context only under nominative fair use. No employment, client, vendor, or partnership relationship is implied unless explicitly stated.