Dataset Overview

SQL-IQ is a comprehensive benchmark that evaluates LLMs across the full spectrum of SQL intelligence. It spans 4 dimensions and 7 tasks, covering generation, comprehension, debugging, and cross-dialect adaptation.

SQL-IQ Benchmark Overview

Tasks by Dimension

Generation Evaluating models' ability to produce correct SQL from various inputs

Text-to-SQL

Translate natural language questions into executable SQL queries given a database schema. This is the most fundamental SQL generation capability.

NL → SQL SQLite

Conversational SQL

Generate SQL in multi-turn dialogue settings where context from previous turns must be understood and carried forward to produce correct queries.

Multi-turn Context-aware
Comprehension Evaluating models' understanding and reasoning about SQL semantics

SQL Equivalence Judge

Given two SQL queries, determine whether they are semantically equivalent — i.e., whether they always produce the same result on any valid database instance.

Equivalence Pair-wise

SQL Judge

Given a natural language question and two candidate SQL queries, select the one that correctly answers the question. Tests comparative SQL comprehension.

Correctness Pair-wise
Debugging Evaluating models' ability to identify and fix SQL errors

SQL Error Classification

Given a buggy SQL query, detect and classify the type of error present (e.g., syntax error, logic error, schema mismatch, etc.).

Error Type Diagnosis

SQL Debugging

External

Fix buggy SQL queries given error messages and database context. This task is sourced from the BIRD-CRITIC benchmark and evaluated using their framework.

Bug Fixing BIRD-CRITIC Repository →
Adaptation Evaluating models' ability to work across different SQL dialects

SQL Translation

Translate SQL queries between different database dialects. Models must understand dialect-specific syntax and functions to produce equivalent queries in the target dialect.

Oracle PostgreSQL ClickHouse Druid MSSQL

Access the Data

The SQL-IQ benchmark data and evaluation code are hosted on GitHub. SQL Debugging uses an external repository (BIRD-CRITIC).

SQL-IQ Repository BIRD-CRITIC (SQL Debugging)