The best text-to-SQL LLM models don't start by writing SQL. They start by reading your database — pulling schema metadata, sampling live column values, and mapping foreign key relationships before generating a single line of query code. This distinction separates a parlor trick from a production system, and it's the paradigm shift that enterprise teams at Snowflake, Databricks, and Google are quietly building around right now.

The stakes are concrete. Natural language to SQL interfaces are moving from internal demos to customer-facing data products, embedded analytics layers, and autonomous reporting pipelines. Yet most of the benchmark scores circulating in vendor decks measure performance on sanitized academic datasets that bear little resemblance to the schema sprawl inside a real enterprise warehouse. A system that scores 87% on Spider 1.0 may score 55% on your actual Snowflake environment. The gap between those two numbers is the story this article unpacks.

The central question: what does it actually take for an LLM to query a database it has never seen before — and which current approaches come closest to solving it?

The Problem Is Harder Than "Write SQL"

The text-to-SQL task sounds deceptively simple: a user types a natural-language question, the system returns an executable SQL query against a specific database schema. The difficulty lies in what the model doesn't know at inference time. Schemas are unseen during training. Column names like acct_eff_dt, gl_cd, or txn_amt_usd_equiv carry zero semantic signal without business context. Business logic — "active customers" means status = 'A' AND churn_flag = 0 — lives nowhere in the DDL.

Two benchmark families define the current conversation. Spider 1.0, developed at Yale, contains 10,181 questions across 200 databases with clean, well-named schemas. Top models like GPT-4o and Claude 3.5 Sonnet achieve roughly 85–87% execution accuracy here. BIRD (Big Bench for Real-world Database) raises the floor: real-world dirty data, external knowledge requirements, and ambiguous column values. Top scores on BIRD sit at 65–70%, a drop of 15–20 percentage points. That gap isn't a benchmark artifact — it's a direct proxy for the accuracy degradation you'll observe moving from a proof-of-concept to a production deployment.

The architecture of production systems has shifted accordingly. Single-pass generation — prompt the model with the schema and question, receive SQL — has largely given way to agentic pipelines: multi-step systems that retrieve schema metadata, sample example rows, generate a candidate query, execute it against the database, and loop on errors. Every meaningful accuracy advance in the last 18 months has come from this architecture, not from scaling base model size.

Schema Inspection: The Step Most Benchmarks Skip

Before writing a single line of SQL, a capable system calls a tool or API to pull INFORMATION_SCHEMA metadata — table names, column types, foreign key relationships, nullable constraints — and samples actual row values from high-cardinality columns. Without this step, the model is guessing. It may hallucinate column names that almost match, infer the wrong data type for a date field stored as VARCHAR, or miss a join path that only becomes visible through foreign key inspection.

Two approaches to schema handling dominate current implementations:

Context-stuffing dumps the full DDL into the prompt. This works adequately for small schemas — 10 to 20 tables — but breaks on enterprise databases with hundreds of tables and thousands of columns. At that scale, the relevant schema information drowns in noise, token limits become a constraint, and the model's attention is spread too thin to reliably identify the two tables that actually matter for the query at hand.

Retrieval-augmented schema selection embeds table and column descriptions, then retrieves only the schema subset relevant to the user's question before constructing the prompt. This RAG + LLM hybrid is now the production standard. Vanna.ai, Dataherald, and the schema-routing layers inside Snowflake Cortex all operate on this principle. The model sees a focused, question-relevant slice of the schema rather than a wall of DDL.

The practical impact is visible in the benchmark gap. GPT-4 achieves ~85–87% on Spider 1.0 but drops to ~60% on BIRD. A significant portion of that drop traces directly to BIRD's messy real-world column values and domain-specific knowledge that aren't deducible from schema names alone. Schema inspection — specifically, sampling live values to understand what a column actually contains — partially closes this gap by giving the model ground truth about data distributions rather than forcing it to infer from column names.

Benchmark Reality Check: Spider vs. BIRD vs. Spider 2.0

Benchmark	Database Count	Questions	Top Model Score (Execution Accuracy)	Primary Challenge
Spider 1.0	200	10,181	~87% (GPT-4o, Claude 3.5 Sonnet)	Cross-domain schema generalization
BIRD	~100	~12,751	~65–70%	Real-world data, external knowledge
Spider 2.0	Enterprise-scale	Complex analytical	Current frontier	Cross-database, time-series, analytical depth

Think of these three as a ladder of difficulty calibrated to production reality. Spider 1.0 is largely a solved problem for frontier proprietary models — it tests whether a model can generalize across domains with clean schemas and well-formed questions. BIRD tests whether a model can handle the friction of real deployments: dirty data, implicit business rules, and questions that require external knowledge to resolve. Spider 2.0, released in 2024 by Yale, pushes further into enterprise territory with cross-database queries, time-series analysis, and analytical complexity that mirrors the questions a finance or operations analyst would actually ask.

The commercially relevant gap is between open-source and proprietary models on the harder benchmarks. On Spider 1.0, DeepSeek-Coder, CodeS, and fine-tuned Llama 3 variants trail GPT-4o by a manageable margin — 5 to 8 percentage points. On BIRD, that gap widens to 10–20 points, which is the difference between a system that works and one that requires human review on every third query.

One evaluation metric deserves particular attention. Execution accuracy — does the query return the correct rows? — is the right measure for production systems. Exact match — does the query match the reference SQL character-for-character? — penalizes semantically correct queries that use different syntax, aliases, or join ordering. Vendors citing exact match scores are, knowingly or not, understating their system's actual utility. Always ask which metric is being reported.

The Agentic Approach: Multi-Step Planning and Self-Correction

The agentic text-to-SQL loop that powers current production systems follows a consistent pattern:

1. Parse the user's question and identify entity references
2. Retrieve relevant schema elements via embedding search over table/column metadata
3. Generate a candidate SQL query
4. Execute the query against the live database
5. Inspect the result set or error message
6. Revise and retry if the result is empty, an error was returned, or the result shape doesn't match the question's intent

This mirrors what a competent human analyst does. Run the query, read the error message, fix the join condition, run it again. The difference is that a well-designed agentic loop can do this in seconds and is consistent about it — it never skips the validation step because it's in a hurry.

Self-correction is the key performance differentiator. Models that receive actual execution feedback — the database engine's error message, not a synthetic penalty signal — and iterate on it outperform single-pass generation significantly on complex queries. The execution error is often more informative than the original question: "column customer_id does not exist in table orders" tells the model exactly where the hallucination occurred.

Commercial implementations of this loop include Vanna.ai (which trains its RAG index on your own query history, making the retrieval step progressively more accurate as the system sees more of your real queries), Dataherald (API-first, enterprise deployment focus with schema metadata caching), and MindsDB (federated data access across multiple databases with LLM inference layer). Each of these sits on top of a frontier base model — the differentiation is entirely in the pipeline architecture, not the underlying LLM.

The open limitation is cross-database and multi-turn queries. Asking a question that requires joining data across two separate databases — a Postgres transactional system and a Snowflake analytical warehouse, for example — causes accuracy to drop sharply even for GPT-4o-class models. Multi-turn queries with implicit back-references ("show me the same breakdown but for last quarter") compound the problem because the model must maintain context about prior schema usage across turns. These are the cases where Spider 2.0 scores remain at the frontier of unsolved problems.

Enterprise Deployment: What Vendors Are Actually Shipping

The four major cloud platforms have all productized natural language to SQL, each with a different architectural bet:

Snowflake Cortex integrates natural language querying directly into the data warehouse, routing questions through a proprietary schema-selection layer before invoking a frontier model. The advantage is tight integration with Snowflake's metadata catalog; the constraint is that you're within the Snowflake ecosystem.

Databricks AI/BI (which raised a landmark $5 billion funding round at a $190 billion valuation in 2026, signaling confidence in AI-native data infrastructure) offers a natural language layer over Unity Catalog, with schema grounding derived from catalog metadata. The Unity Catalog integration is the meaningful architectural detail — it means the system has access to data lineage and column-level access controls, not just DDL.

Google Gemini in BigQuery uses schema-aware SQL generation with grounding against BigQuery's INFORMATION_SCHEMA. The Gemini integration adds multimodal context — chart interpretation alongside SQL generation — which matters for analysts who work in mixed text-and-visual workflows.

Microsoft Copilot in SQL Server provides inline query suggestions within the IDE and management studio, which positions it as a developer productivity tool rather than an end-user analytics interface. Different use case, different accuracy requirements.

The open-source and startup tier offers something the cloud giants don't: control over the RAG index and query history that grounds the model. For enterprises with sensitive schemas and proprietary business logic, the ability to fine-tune the retrieval layer on your own labeled query examples — rather than relying on a cloud vendor's generic schema-routing — is a meaningful architectural advantage. The model itself is increasingly commoditized. GPT-4o and Claude 3.5 Sonnet perform comparably on SQL generation tasks. The differentiation is in schema retrieval quality, query history fine-tuning, and error handling.

Implications for Decision-Makers

Engineering teams building internal tooling should not evaluate text-to-SQL models on Spider 1.0 scores alone. Run a BIRD-style evaluation on a representative sample of your actual schema — real table names, real column values, real query patterns from your analytics history. This takes two to three days to set up and will tell you more about production suitability than any public leaderboard.

Data and analytics leaders evaluating vendors should ask one specific question before signing: does the system perform schema inspection at query time, or does it rely on a static schema snapshot ingested at setup? Static snapshots go stale the moment a column is renamed, a table is added, or a data type is changed. Live inspection at query time is a hard requirement for any schema that evolves — which is every production schema.

Teams choosing between open-source and proprietary models face a real accuracy tradeoff on hard queries, but it's not insurmountable. The 10–20 point gap on BIRD is real, but fine-tuned open-source models — CodeS, domain-specific Llama 3 fine-tunes trained on your own labeled query examples — can close it substantially. The decision variable is whether you have 500 to 2,000 labeled question-to-SQL pairs from your own database to use as fine-tuning data. If yes, open-source becomes viable. If not, you're starting from the base model's general SQL knowledge, and the accuracy gap is likely to manifest in production.

The next meaningful accuracy jump won't come from a larger base model. It will come from better tool-use loops: smarter schema retrieval that understands data lineage, execution feedback that distinguishes between a syntax error and a semantic error, and multi-step planning that can decompose a cross-database analytical question into executable sub-queries. Spider 2.0 performance improvements will emerge from this tier first, and the vendors who invest in pipeline architecture rather than model size are the ones to watch.

What This Means for the Field

The question is no longer "can LLMs write SQL?" — they clearly can, and they do it reliably on well-formed schemas with clean data. The operative question is "how well can a system reason about an unfamiliar database at query time, under real-world conditions, without human intervention on errors?" That question reframes text-to-SQL as a database-aware reasoning problem, not a code generation problem.

Systems that treat it as code generation — prompt in, SQL out — will plateau around 60% execution accuracy on real enterprise workloads. Systems that treat it as an agentic reasoning loop — inspect, retrieve, generate, execute, revise — are where the production-grade accuracy gains are happening. The benchmark scores confirm it. The commercial architecture of every serious vendor in the space confirms it. The engineering question is no longer which model to use; it's how well your pipeline is built.