Because natural language is ambiguous and real schemas are messy. The best systems on academic benchmarks do not reach 100% either. If a tool claims 100%, ask about its method; it is probably counting text similarity.

Does the model see the test questions in advance?

No. Questions go through the same path as production: the model sees the question and schema fresh, the generated SQL runs, and the result is compared.

What accuracy should I expect on my own schema?

With meaningful names, it behaves like the typical business suites (94%+); full of cryptic abbreviations, like the legacy suites (60 to 80%). The good news: adding column descriptions moves you from the second group toward the first.

How often do you re-measure?

The full set re-runs whenever the model or the generation path changes. The current summary always lives on the accuracy page; the numbers in this article are from the July 2026 run.

← All postsThe PerSight Team · Last updated: July 25, 2026 · 7 min read

How Do You Measure Text-to-SQL Accuracy? Our Own Numbers, Failures Included

How text-to-SQL accuracy should be measured and why a query that "looks right" is not enough. The method behind our 661-question test set, suite-by-suite results, and where we fail most.

When a text-to-SQL product page says "97% accurate", the first question to ask is: what did you count? Two very different things get measured as "correct" in this field, and the gap between them is the gap between deciding on a right number and deciding on a wrong one.

Why a query that "looks right" is not enough

The cheap method compares the generated SQL text against a reference SQL. But dozens of different queries can answer the same question correctly, and a query that closely resembles the reference can return wrong data over a one-word difference. The query runs, a number comes back, the number is wrong, and there is no error message. So we do not look at the text at all.

Our method: count the result, not the text

Every test question has a hand-written reference query whose correctness is known.
The model's generated query actually runs on a live database.
The returned data is compared row by row against what the reference returns. Different column order with correct values passes; extra context columns pass as long as the requested values are present.
The set deliberately includes unanswerable questions: when a question has no answer in the data, we test whether the model says so instead of inventing SQL.

Current results, unfiltered

The latest full run covered 661 real questions across four engines (PostgreSQL, SQL Server, MySQL, Oracle) and schemas from different domains: sales, hospital, HR, university. The overall result was 593 of 661, which is 90%. On typical business schemas, the ones with meaningful table and column names, suites score between 94% and 100%; the 94% figure on our site summarizes that class.

So where does the rest go? Into deliberately hardened "legacy" suites: old schemas full of cryptic abbreviations and no descriptions. There the score drops to the 60 to 80 percent band. We do not hide this, because the most useful practical lesson lives exactly there: the number one variable in text-to-SQL accuracy is not the model, it is how readable your schema is. Adding descriptions to cryptic columns is the single highest-impact move.

What are the most common failures?

Extra rows: a join multiplies more rows than expected, such as 6 rows coming back where the reference expects 5. Result comparison catches this instantly.
Wrong column choice: using the wrong one of two similarly named columns. The classic cryptic-schema failure.
A different reading of an ambiguous question: interpreting "busiest ward" through a different metric. Technically defensible, but different from the reference, so we still count it as a failure.

None of these failure classes can damage data, because generation is read-only; the risk is wrong information. The antidote is the measurement itself plus a query that is always visible: you can see what you are looking at before trusting the number.

Frequently asked questions

Why not 100%?: Because natural language is ambiguous and real schemas are messy. The best systems on academic benchmarks do not reach 100% either. If a tool claims 100%, ask about its method; it is probably counting text similarity.
Does the model see the test questions in advance?: No. Questions go through the same path as production: the model sees the question and schema fresh, the generated SQL runs, and the result is compared.
What accuracy should I expect on my own schema?: With meaningful names, it behaves like the typical business suites (94%+); full of cryptic abbreviations, like the legacy suites (60 to 80%). The good news: adding column descriptions moves you from the second group toward the first.
How often do you re-measure?: The full set re-runs whenever the model or the generation path changes. The current summary always lives on the accuracy page; the numbers in this article are from the July 2026 run.

The short version of the method and the current score live at persight.ai/en/accuracy. The healthiest test is seeing how it behaves on your own data; the beta queue is at persight.ai/beta.