Overview
A measurement and prediction framework for infrastructure-induced output drift in LLM serving systems.
Problem
Production teams routinely change accelerators, numerical precision, and serving frameworks, yet aggregate quality checks can miss prompt-level changes in correctness and safety behavior.
Approach
DriftBench compares matched prompt-response evaluations across serving configurations and models portability risk from hardware, precision, framework, model, and workload characteristics.
Results & current status
- 236,985 prompt-response pairs were evaluated across 105 configurations, five models, four GPU platforms, three serving frameworks, and three precisions.
- The Portability Risk Index generalized to unseen hardware with R² = 0.909 and unseen precision with R² = 0.763.
- The validation was extended to TPU v5e and v6e, where predictability held across a new numerical path with R² values from 0.72 to 0.96.
- The study distinguishes changes that can be predicted from those that require fresh measurement, and demonstrates the operational consequences of infrastructure-induced safety flips.
Technical details
The work covers code, mathematics, safety, chat, and long-context workloads using task-appropriate evaluators. It reports held-out-dimension validation rather than relying only on random train/test splits.
Reproducibility
The archived artifact received the ACM Available, Functional, and Reproduced badges. The paper, artifact, and conference talk are publicly available.