Skip to content

Export an eval set

Platform (dbtCharts.com)

This page describes dbt Charts Platform, the hosted product at dbtCharts.com. These features are not part of the open source dct engine.

Query review is in alpha and is turned on per organization. On a query's review page, a reviewer approves the SQL and writes the questions that query answers. The eval export hands those pairs back as JSON Lines: one question, the SQL a person approved as its answer, and who approved it. Use it to score an agent against answers your own team signed off on.

dct cloud export-eval --org acme --project analytics > ~/evals/analytics.jsonl

The same bytes are served by the Platform API:

curl -H "Authorization: Bearer $DCT_CLOUD_TOKEN" \
  https://dbtcharts.com/api/orgs/acme/projects/analytics/eval.jsonl

Keep the export away from the agent you are testing

The file is an answer key. An agent that can read it is not being tested.

  • Do not commit it to the project repository, or to any repository, folder, or bucket the agent under test can read. An agent working in a project reads that project's files, so an export saved there hands it every answer. Write it outside the project, as in the example above.
  • Do not give the agent under test a credential that can call the export. A dct cloud login session or a token with dashboards:read on the project can run it. Run the agent without one, or with a token for a user who cannot view the reviewed boards.
  • The Claude and ChatGPT connector tools and the Platform chat never return review questions. They read board files and your warehouse, not the review record.

What is exported

By default, a pair is exported only while its approval still holds: the query's SQL on the project's serving branch matches the SQL that was approved. Edit the SQL and the pair drops out until someone approves it again.

--include-unapproved (API: ?include_unapproved=true) also writes pairs whose query was never approved, or was changed after its approval. Those lines have a null approved_at, so a scorer can keep or drop them.

A question whose board or query no longer exists has no SQL and is never exported.

Lines are ordered by board, then query name, then the order the questions appear on the review page.

Who can export

The caller needs the dashboards:read scope and must be a member of the organization. Pairs from a board the caller cannot view are left out. The CLI reports how many boards were left out on stderr, and the API reports it in the X-Dct-Skipped-Boards response header, so a partial export is never mistaken for a complete one.

If a board with pairs to export does not compile, the export fails instead of leaving its pairs out: the API answers 409 with a message naming the board, and the CLI prints that message, exits non-zero, and writes nothing. Fix the board, or the pair's source and dialect cannot be resolved.

The record

Each line is one JSON object with these fields, in this order:

Field Type Meaning
question string The question, as the reviewer left it.
sql string The SQL template as authored on the board, Jinja included.
dialect string or null SQL dialect of source. Null when the source's type names none, such as a file source.
source string or null The source the query runs on after inheritance: its own source:, the board's, or the folder's meta.yml.
variables object The board's variables at their defaults, which is what sql binds when the board renders with no overrides.
board_path string The board file, relative to the project.
query_name string The key under queries:, or charts/<chart_id> for SQL written inline on a chart.
question_origin string Who wrote the question. See below.
approved_by string or null Email of the approver. Null for an unapproved pair, and when the approver's account has been deleted.
approved_at string or null The UTC date of the approval, as YYYY-MM-DD. Null exactly when the pair is unapproved.
sql_digest string A digest of sql with comments dropped and whitespace collapsed: the value the approval is pinned to.
{"question":"How many orders came in over the last 12 months?","sql":"SELECT count(*) AS orders\nFROM analytics.orders\nWHERE ordered_at >= current_date - INTERVAL '{{ months }}' MONTH\n","dialect":"bigquery","source":"warehouse","variables":{"months":12},"board_path":"charts/finance/revenue.yml","query_name":"monthly_orders","question_origin":"authored","approved_by":"sam@example.com","approved_at":"2026-10-22","sql_digest":"v1:sha256:9f2c..."}

To tell approved pairs from unapproved ones, test approved_at, not approved_by: an approval outlives its approver's account.

question_origin

Value Meaning
authored A reviewer typed the question.
generated The question was generated from the query and a reviewer left it as is.
edited The question was generated, then rewritten by a reviewer.

A generated question is derived from the SQL it is paired with, so it tends to restate the query. Filter to authored when you want questions phrased the way a person asks them.

sql_digest

Export again later and compare digests for the same board_path and query_name: a different digest means the SQL changed and the pair you saved is stale.