Export an eval set¶
Platform (dbtCharts.com)
This page describes dbt Charts Platform, the
hosted product at dbtCharts.com. These features are not part of the
open source dct engine.
Query review is in alpha and is turned on per organization. On a query's review page, a reviewer approves the SQL and writes the questions that query answers. The eval export hands those pairs back as JSON Lines: one question, the SQL a person approved as its answer, and who approved it. Use it to score an agent against answers your own team signed off on.
The same bytes are served by the Platform API:
curl -H "Authorization: Bearer $DCT_CLOUD_TOKEN" \
https://dbtcharts.com/api/orgs/acme/projects/analytics/eval.jsonl
Keep the export away from the agent you are testing¶
The file is an answer key. An agent that can read it is not being tested.
- Do not commit it to the project repository, or to any repository, folder, or bucket the agent under test can read. An agent working in a project reads that project's files, so an export saved there hands it every answer. Write it outside the project, as in the example above.
- Do not give the agent under test a credential that can call the export.
A
dct cloud loginsession or a token withdashboards:readon the project can run it. Run the agent without one, or with a token for a user who cannot view the reviewed boards. - The Claude and ChatGPT connector tools and the Platform chat never return review questions. They read board files and your warehouse, not the review record.
What is exported¶
By default, a pair is exported only while its approval still holds: the query's SQL on the project's serving branch matches the SQL that was approved. Edit the SQL and the pair drops out until someone approves it again.
--include-unapproved (API: ?include_unapproved=true) also writes pairs
whose query was never approved, or was changed after its approval. Those
lines have a null approved_at, so a scorer can keep or drop them.
A question whose board or query no longer exists has no SQL and is never exported.
Lines are ordered by board, then query name, then the order the questions appear on the review page.
Who can export¶
The caller needs the dashboards:read scope and must be a member of the
organization. Pairs from a board the caller cannot view are left out. The
CLI reports how many boards were left out on stderr, and the API reports it
in the X-Dct-Skipped-Boards response header, so a partial export is never
mistaken for a complete one.
If a board with pairs to export does not compile, the export fails instead
of leaving its pairs out: the API answers 409 with a message naming the
board, and the CLI prints that message, exits non-zero, and writes nothing.
Fix the board, or the pair's source and dialect cannot be resolved.
The record¶
Each line is one JSON object with these fields, in this order:
| Field | Type | Meaning |
|---|---|---|
question |
string | The question, as the reviewer left it. |
sql |
string | The SQL template as authored on the board, Jinja included. |
dialect |
string or null | SQL dialect of source. Null when the source's type names none, such as a file source. |
source |
string or null | The source the query runs on after inheritance: its own source:, the board's, or the folder's meta.yml. |
variables |
object | The board's variables at their defaults, which is what sql binds when the board renders with no overrides. |
board_path |
string | The board file, relative to the project. |
query_name |
string | The key under queries:, or charts/<chart_id> for SQL written inline on a chart. |
question_origin |
string | Who wrote the question. See below. |
approved_by |
string or null | Email of the approver. Null for an unapproved pair, and when the approver's account has been deleted. |
approved_at |
string or null | The UTC date of the approval, as YYYY-MM-DD. Null exactly when the pair is unapproved. |
sql_digest |
string | A digest of sql with comments dropped and whitespace collapsed: the value the approval is pinned to. |
{"question":"How many orders came in over the last 12 months?","sql":"SELECT count(*) AS orders\nFROM analytics.orders\nWHERE ordered_at >= current_date - INTERVAL '{{ months }}' MONTH\n","dialect":"bigquery","source":"warehouse","variables":{"months":12},"board_path":"charts/finance/revenue.yml","query_name":"monthly_orders","question_origin":"authored","approved_by":"sam@example.com","approved_at":"2026-10-22","sql_digest":"v1:sha256:9f2c..."}
To tell approved pairs from unapproved ones, test approved_at, not
approved_by: an approval outlives its approver's account.
question_origin¶
| Value | Meaning |
|---|---|
authored |
A reviewer typed the question. |
generated |
The question was generated from the query and a reviewer left it as is. |
edited |
The question was generated, then rewritten by a reviewer. |
A generated question is derived from the SQL it is paired with, so it tends
to restate the query. Filter to authored when you want questions phrased
the way a person asks them.
sql_digest¶
Export again later and compare digests for the same board_path and
query_name: a different digest means the SQL changed and the pair you
saved is stale.