Submission Guide

Submit your model's prediction files and we will run the official evaluation pipeline to produce scores. This ensures fairness and reproducibility across all submissions.

How It Works

Evaluation Model

6 tasks (Text-to-SQL, Conversational SQL, SQL Judge, SQL Equivalence Judge, SQL Error Classification, SQL Translation) — you submit prediction files; we run official evaluation scripts and produce the scores.
SQL Debugging† — evaluated externally via BIRD-CRITIC. You run their evaluation yourself and report the score.

Generate Predictions

Clone the SQL-IQ repository and run your model on each task's test set. Save the raw predictions (generated SQL queries, classification labels, etc.) in the format specified below.

Package Submission

Organize your prediction files into the standard directory structure and include a metadata.json with model information. For SQL Debugging, include your BIRD-CRITIC score in the metadata.

Submit via GitHub Issue

Open an issue on our GitHub repository using the "Leaderboard Submission" issue template. Attach your prediction files (as a zip) and fill in the required fields (model name, organization, BIRD-CRITIC score, etc.).

Official Evaluation & Publish

We run the official evaluation pipeline on your predictions in our controlled environment. Once verified, your scores will appear on the leaderboard within 3–5 business days.

Submission Format

Your submission should contain a metadata.json and a predictions/ directory with one JSON file per task. This mirrors the output structure of the SQL-IQ evaluation pipeline.

Directory Structure TREE

submission/
├── metadata.json
└── predictions/
    ├── text2sql/predictions.json
    ├── conversational_sql/predictions.json
    ├── sql_judge/predictions.json
    ├── sql_equ_judge/predictions.json
    ├── sql_err_class/predictions.json
    ├── sql_trans_oracle/predictions.json
    ├── sql_trans_postgresql/predictions.json
    ├── sql_trans_clickhouse/predictions.json
    ├── sql_trans_druid/predictions.json
    └── sql_trans_mssql/predictions.json

metadata.json JSON

{
  "model": "Your-Model-Name",
  "organization": "Your Organization",
  "description": "Brief description of the model or method",
  "model_version": "gpt-4o-2025-05-13",
  "sql_debugging_score": 42.5,
  "contact_email": "your@email.com"
}

Metadata Fields

model — Display name for the leaderboard entry.
model_version — Specific model version or checkpoint (for reproducibility).
sql_debugging_score — Your BIRD-CRITIC Reward Score (0–100). Set to null if not evaluated.
contact_email — For verification purposes only; will not be published.

Prediction File Formats

Each predictions.json is a JSON array. Every entry must include an id field (matching the test sample) and a prediction field. You only need to submit tasks you want to be evaluated on.

Text-to-SQL

Provide the predicted SQL query string for each sample. Evaluated by Execution Accuracy (comparing query execution results against ground truth on SQLite).

text2sql/predictions.json JSON

[
  { "id": 0, "prediction": "SELECT name FROM schools WHERE type = 'Charter'" },
  { "id": 1, "prediction": "SELECT COUNT(*) FROM frpm WHERE county = 'Los Angeles'" },
  ...
]

Conversational SQL

Multi-turn SQL generation. Each entry includes the turn context. Evaluated by Execution Accuracy.

conversational_sql/predictions.json JSON

[
  { "id": 0, "prediction": "SELECT AVG(score) FROM students WHERE grade = 'A'" },
  { "id": 1, "prediction": "SELECT name FROM students WHERE score > 90" },
  ...
]

SQL Judge

Given two candidate SQLs, predict which one correctly answers the question. Prediction should be "A" or "B". Evaluated by Accuracy.

sql_judge/predictions.json JSON

[
  { "id": 0, "prediction": "A" },
  { "id": 1, "prediction": "B" },
  ...
]

SQL Equivalence Judge

Determine whether two SQL queries are semantically equivalent. Prediction should be "yes" or "no". Evaluated by Geometric Mean of equivalent-accuracy and non-equivalent-accuracy.

sql_equ_judge/predictions.json JSON

[
  { "id": 0, "prediction": "yes" },
  { "id": 1, "prediction": "no" },
  ...
]

SQL Error Classification

Detect whether a SQL is correct and classify error types if present. Prediction is a structured object. Evaluated by Unified F1.

sql_err_class/predictions.json JSON

[
  {
    "id": 0,
    "prediction": {
      "result": "False",
      "error_types": ["Function-Related Errors", "Clause-Related Errors"]
    }
  },
  {
    "id": 1,
    "prediction": {
      "result": "True",
      "error_types": []
    }
  },
  ...
]

result: "True" (no error) or "False" (has errors)
error_types: list of error category strings (empty if result is "True")

SQL Translation

Translate SQLite queries into target dialects. Submit one file per target dialect (sql_trans_oracle/, sql_trans_postgresql/, sql_trans_clickhouse/, sql_trans_druid/, sql_trans_mssql/). Evaluated by Execution Accuracy on the target database engine.

sql_trans_postgresql/predictions.json JSON

[
  { "id": 0, "prediction": "SELECT CAST(free_meal_count AS FLOAT) / enrollment FROM frpm LIMIT 10" },
  { "id": 1, "prediction": "SELECT school_name FROM schools WHERE county = 'San Diego'" },
  ...
]