Skip to main content
Back to timeline
arXivSource publication:

Verified SQL history need not be rebuilt each time: MEM-SQL lifts execution accuracy from 37.1% to 50.0% on HARE while cutting input tokens by 76.6%

Synopsis

When users query the same database repeatedly, previously verified SQL often already contains the joins or computations a new question needs, yet standard benchmarks score each question in isolation and never see that reusable history. MEM-SQL stores history in three layers, whole-query templates, reusable SQL blocks and schema concepts, and admits a layer only when every requirement assigned to it clears that layer's threshold, consulting the database schema only when no layer qualifies or the generator judges the evidence insufficient. Across six language models on the 1,600 HARE requests, mean execution accuracy rises from 37.1% for direct generation without history to 50.0%, with 76.6% fewer input tokens and 68.8% lower latency on average.

Source-provided article image: MEM-SQL: Memory-Empowered Text-to-SQL
arXiv

Interpretation

Text-to-SQL systems can now treat verified queries over the same database as a retrievable evidence source instead of reconstructing joins and computations from the schema each time. Benchmarks such as Spider and BIRD evaluate questions independently, so the setting where a user repeatedly queries one database, and an earlier verified query may already answer a new question or supply part of its computation, had not been measured on its own. On the 1,600 HARE requests across six language models, mean execution accuracy rises from 37.1% for direct generation without history to 50.0% for MEM-SQL, while average input tokens fall by 76.6% and latency by 68.8%.

Memory is split into three granularities, and a new question is first decomposed into the retrieval units each layer needs before the system decides whether the evidence suffices. Whole-query templates keep a complete skeleton with literal values, sorting, aggregates and grouping keys replaced by placeholders, reusable SQL blocks keep a SELECT block with its FROM, JOIN and clause structure, and schema concepts map domain terms such as lap time to columns and join relationships. A layer contributes evidence only if every requirement assigned to it has a match above that layer's threshold, so a strong match cannot hide an uncovered requirement; on-demand schema linking consults the schema only when no layer qualifies or the generator judges the evidence insufficient, and MEM-SQL completes 95.4% of requests from memory on average.

Perspective

Evaluation is confined to the 1,600 HARE requests, derived from 823 verified queries drawn from the Spider, BIRD, Spider 2.0 and BEAVER benchmarks, and memory is built only from each database's verified question-SQL history and held fixed throughout evaluation.

HARE does not model schema drift, which is common in enterprise databases, such as a change in the month a fiscal year starts; MEM-SQL is evaluated with static memory, so how adding, updating and retiring memory affects dynamic settings remains open; and execution accuracy rejects answers that differ only in format and cannot detect SQL that returns the right rows for the wrong reason.