HVAC digital twins pass the run test but not the physics test: seven of eight LLMs build and converge, yet behavioral equivalence spans 50.0% to 98.6%
Synopsis
For HVAC digital-twin topology specifications that large language models write from plain-language descriptions, a four-level executable validation protocol moves the judgment past parsing, building and convergence to whether the converged state matches a hidden reference model. Across eight LLMs and fourteen topologies, seven models built in 98.6% to 100% of trials and converged in at least 80.0%, but behavioral equivalence reached only 50.0% to 98.6%; about half of the failed trials did converge, to a steady state different from the reference, mostly in desiccant rotor systems.
Interpretation
Judging an LLM-written HVAC digital-twin specification now runs all the way to whether it is physically right, not just whether the file parses, the model builds, or the simulation finishes. Earlier evaluations mostly stopped at parse success, internal consistency, completed runs, or textual similarity to a reference, and so could not reject a model that converges in simulation but is physically wrong. Over 560 evaluation cells, eight models on fourteen topologies with five repetitions each, seven models built in 98.6% to 100% of cells and converged in at least 80.0%, while behavioral equivalence ranged only from 50.0% to 98.6%, so the earlier levels barely separate the models.
The protocol sends each specification through static validity, model build, simulation convergence and behavioral equivalence, and grades it by the highest level reached. The equivalence level compares the generated model with a reference model never shown to the LLM at monitored nodes, on dry-bulb temperature, dew point and mass flow, with tolerance bands set by the envelope of variants that survive plus and minus 20% sizing perturbations of the reference. The fourteen topologies span conventional outdoor-air handling, recirculation, multi-zone conditioning and desiccant dehumidification, five of them describing a low-humidity test chamber; a failing level returns its error message verbatim for up to three repair rounds.
Failures concentrate in the physics that is hardest to model, the desiccant rotor, rather than in text formatting. Of 188 cells that did not reach behavioral equivalence, 93 built and converged but settled away from the reference, and among these the regeneration exhaust temperature was outside its band in 69 cells and the supply dew point in 65. In the five topologies without an operating rotor, 169 of 200 cells reached equivalence, against 203 of 360 rotor-active cells; excluding three topologies with a regeneration-heater sizing ambiguity, the three highest-passing models reached 100%, 100% and 96.7% on the six remaining rotor topologies.
Perspective
The demonstration covers one domain, HVAC, fourteen topologies and a single prompt assembly, and equivalence is measured against reference models rather than measurements; the reference models are assembled from the same component library, and five topologies describe the low-humidity test chamber whose generated physics was validated against measurements earlier. The human-engineer control covers six of the fourteen topologies and four authors, so the authorship comparison holds at that scale.
The output budget and repair count were fixed for all models at 16,384 tokens and three repairs, and 73 of the 93 near-miss cells contain an attempt cut off at the output limit; whether a larger budget or more repairs would bring them to equivalence was not tested. The 66.4% pass rate shifts with the scoring convention: 75.0% without topologies 4 to 6, 73.4% with guide-exception near misses counted as passes, and 64.1% with passes that changed a described operating value counted as failures; halving or increasing the band envelopes by half gives 58.0% and 68.6%, while the order of the three highest-passing models does not change. Results hold for the listed model versions and will change as models change.
