Skip to main content
Search

# Subword tokenization

2 related articles

Subscribe to updates
  1. arXiv

    Auditing 1,813 AmE–BrE variant pairs across the pipeline: six pretraining corpora, 21 post-training datasets and ten checkpoints all lean American English

    The study builds a curated resource of 1,813 matched American English–British English variant pairs and introduces DiAlign, a training-free method for estimating regional alignment, then uses them to audit six pretraining corpora, 21 post-training datasets, nine tokenizers, ten model checkpoints and generations under two prompting conditions, finding that American English is systematically favored across data exposure, representation and generation, and that British-English prompting only partially shifts this default.

  2. arXiv

    Functionalizer splits casing, diacritics and repetition into reversible opcodes, cutting vocabulary needs by up to 19.7% under corpus exhaustion and lifting a 98M-parameter GPT-2's Python syntax validity from 7.70% to 9.12%

    The work presents the Functionalizer, a lossless pre-tokenizer framework that factors orthographic variation such as casing, diacritics and character repetition into a compositional opcode/operand prefix stream encoded in the Unicode Private Use Area, achieving complete corpus coverage with up to 19.7% fewer required vocabulary slots and, on 98M-parameter GPT-2 models, raising Python syntax validity from 7.70% to 9.12% while reducing duplicate n-gram repetition on FineWeb-Edu prose from 66.0% to 55.8%.

You’re all caught up