Google hands datacenter repairs to a graph foundation model: GPU downtime falls 18.3% and new hardware ships four months earlier
Synopsis
Google's deployed HardwareDx replaces per-platform hand-maintained scoring rules for datacenter machine repair with a foundation model pretrained on physical hardware topology graphs, so one model diagnoses faulty components and prescribes repair actions across GPU and TPU platforms. In production A/B tests, mean time to repair fell 18.3% on H100 and 16.1% on TPU, and transfer learning brought new platforms to production readiness before hardware general availability.
Interpretation
Datacenter hardware diagnosis moves from a rule system maintained separately for each platform to a graph foundation model that transfers across platforms, with one model producing faulty-component judgments and repair actions on both GPU and TPU compute architectures. The prior production system consisted of hundreds of engineer-maintained scoring rules that accumulated a badness score per component and selected a repair strategy from it, and every new hardware product required a hand-curated new rule set. In a machine-level randomized A/B test in Google's production environment, HardwareDx cut mean time to repair by 18.3% on H100 and 16.1% on TPU relative to that rule system, across 225 H100 repair windows and 2,567 TPU repair windows.
The model represents each machine as a graph whose nodes are components and subcomponents, with sensor readings, test symptoms and log errors attached as feature nodes to the component that produced them, and outputs a binary faulty-or-not label for each field replaceable unit. The architecture interleaves graph attention layers with graph Transformer layers, the former passing messages over local topology and the latter letting any two nodes attend directly, covering failure syndromes spanning up to eight hops and non-physical signals attached to a software node. On the TPU v7x platform this combination reaches a component F1 of 0.990, above GATv2 alone at 0.985, GIN alone at 0.943 and Transformer-only at 0.929; removing LLM text features drops F1 to 0.444, and a GBDT on flattened features reaches 0.540.
Perspective
The demonstrated scope is a machine-level A/B test on H100 and one new TPU platform in Google's production environment, plus offline evaluations on TPU v7x, H100, H200 and B200; the system has run for six months across more than 50 ML clusters and six AI accelerator platforms.
The authors state that foundation model here means a domain-specific graph architecture pretrained across heterogeneous machine topologies, not an open general-domain model, and that it is not fully self-supervised: downstream fine-tuning still depends on platform-specific topology, telemetry schemas and curated technician repair trajectories. The success label for the third task marks every intermediate step as failure, which can penalize necessary partial repairs and introduces causal confounding across actions taken together.
