TIDE reuses inference-time hidden states to adapt draft models online, reaching up to 1.66x throughput and recovering performance when static drafts degrade it
TIDE is a serving-engine-native framework that reuses the target model's intermediate hidden states produced during inference as training signals for online draft adaptation, avoiding additional target model computation and serving-time overhead, and combines this with adaptive runtime control that activates speculation and draft training only when beneficial and with mapping of inference and training onto appropriate GPU classes; across diverse real-world workloads TIDE achieves up to 1.66x throughput over no-speculation baselines, recovers performance on misaligned workloads where static draft models degrade throughput, reduces training time by up to 3.02x and storage requirements by 24x compared to existing draft training approaches, and improves system throughput by up to 1.