Skip to main content
Search

# Tokenization Disambiguation

1 related articles

Subscribe to updates
  1. arXiv

    Tokenization ambiguity no longer sinks single-history steganography: coin projection keeps over 90% of string entropy at full vocabulary

    Provably secure linguistic steganography assumes the receiver recovers the sender's tokens, but a text channel delivers only text, and one text admits many tokenizations, so retokenizing can lose the message. SyncPool kept the distribution exact by encoding pools of prefix-related tokens, yet its capacity collapsed at full vocabulary as whitespace and UTF-8 lead-byte hubs merged subtrees. Coin projection fixes each prefix's stop decision with shared randomness and leaves only root and child choices to the coder; on 3–4B models in English, Chinese and Japanese it leaves 91–93% of string entropy to the coder and turns 2.16–6.85 times as much entropy into message bits as SyncPool at full vocabulary, with zero error.

You’re all caught up