Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,17 +98,20 @@ but the printed result is a truncated preview). Re-fetch
past results with `hotdata databases results get <result-id>`; browse history
with `hotdata databases queries list`.

## Join across sources
## Join across databases

Attach another catalog to an instant database and join its live tables directly,
no copying:
Attach another instant database and join its live tables directly, no copying:

```sh
hotdata databases attach prod-replica --alias prod
hotdata query "SELECT t.id, o.total FROM demo.public.tickets t
JOIN prod.public.orders o ON o.ticket_id = t.id"
```

The attached database is read-only here: loads still go to your own database,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the command table at README.md:168 still reads "Attach a catalog so its tables are queryable", and line 169 still reads "Detach a previously attached catalog" (not blocking). A reader who scans the table sees the old model, two sections below the new one. Reword both rows to match this section.

and `detach` withdraws visibility without deleting anything. `--alias` is
required when the other database kept the stock `default` catalog name.

## Search

Create an index once, then search server-side. Vector search auto-embeds the
Expand Down Expand Up @@ -163,8 +166,8 @@ The full command surface. The top level has nine groups — `auth`, `workspaces`
| `databases create` | Create a new instant database |
| `databases fork` | Fork a database into a new, independent database |
| `databases lineage` | Show a database's whole fork family tree |
| `databases attach` | Attach a catalog so its tables are queryable |
| `databases detach` | Detach a previously attached catalog |
| `databases attach` | Attach another database so its tables are queryable |
| `databases detach` | Detach a previously attached database |
| `databases use` | Set the current (default) database |
| `databases unset` | Clear the current database |
| `databases remove` | Delete a database and all its tables |
Expand Down
49 changes: 28 additions & 21 deletions skills/hotdata/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ A workspace's query worker scales to zero after inactivity. The **first** comman

**Agents — list before show.** Run `hotdata databases context list` (optionally `--prefix DATAMODEL`) first; run `hotdata databases context show DATAMODEL` *only if* the stem is listed. A missing stem makes `show` exit 1 — normal for a fresh database, not a failure: don't retry in a loop or run speculative `show` in parallel with other tools. Proceed without context:DATAMODEL until one exists.

**context:DATAMODEL is the durable, shared store** — entities, keys, cross-catalog joins, and the naming/query conventions the whole team relies on. Keep task-scoped exploration (scratch SQL, hypotheses, one-off join checks) in the conversation or local notes; **promote** to context:DATAMODEL only when findings should outlive the session and guide everyone — reconcile against `databases context show DATAMODEL` (if listed), write `./DATAMODEL.md`, then `hotdata databases context push DATAMODEL`. No need to update it after every ad-hoc query. What to write inside the document: [references/DATA_MODEL.template.md](references/DATA_MODEL.template.md) and [references/MODEL_BUILD.md](references/MODEL_BUILD.md).
**context:DATAMODEL is the durable, shared store** — entities, keys, cross-database joins, and the naming/query conventions the whole team relies on. Keep task-scoped exploration (scratch SQL, hypotheses, one-off join checks) in the conversation or local notes; **promote** to context:DATAMODEL only when findings should outlive the session and guide everyone — reconcile against `databases context show DATAMODEL` (if listed), write `./DATAMODEL.md`, then `hotdata databases context push DATAMODEL`. No need to update it after every ad-hoc query. What to write inside the document: [references/DATA_MODEL.template.md](references/DATA_MODEL.template.md) and [references/MODEL_BUILD.md](references/MODEL_BUILD.md).

## Multi-step workflows

Expand Down Expand Up @@ -110,9 +110,9 @@ hotdata databases unset
hotdata databases <id> [--workspace-id <workspace_id>] [--output table|json|yaml]
hotdata databases remove <id> [--workspace-id <workspace_id>]

# Attach a catalog so its tables are queryable (enables cross-catalog queries — see below)
hotdata databases attach <catalog|name> [--database <id>] [--alias <alias>]
hotdata databases detach <catalog|name|alias> [--database <id>]
# Attach another database so its tables are queryable (enables cross-database queries — see below)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit: hotdata databases attach --help still describes the old model (not blocking). src/commands/databases.rs:128 reads "Attach a catalog to an instant database", and the --attach help at src/commands/databases.rs:69 uses --attach github as its example. The skill docs and the CLI help now disagree. Updating those doc comments needs the matching assertions in tests/databases_cli.rs checked.

hotdata databases attach <database> [--database <id>] [--alias <alias>] # <database>: name, catalog alias, or id
hotdata databases detach <database> [--database <id>] # or the alias it was attached under

# Preferred: load by catalog alias (server declares the table/schema if missing).
# Loads csv, json, or parquet — format read from the extension.
Expand Down Expand Up @@ -140,9 +140,9 @@ hotdata databases tables remove <table> [--database <id>] [--schema public] [--w
- `tables add` — declares a table **with its key and storage layout**, which a load cannot infer. `--key` (repeatable) is what enables the `delete`/`update`/`upsert` load modes on that table. `--sorted-by <col>` or `<col>=desc` sets sort order; `--partition-by <col>` partitions on the value, `<col>=month` (or `year`/`day`/`hour`) on a calendar part — one partition per calendar month needs **both** `<col>=year` and `<col>=month`, or every March shares a partition. Sort and partition are fixed once the table exists. `--key-determines` (repeatable) asserts a column's value is fixed by the key: it prunes keyed loads harder, and is **correctness-affecting** — declare it only where the invariant really holds, or a keyed load can leave a duplicate key behind. Re-adding an existing table is a conflict (409), and `tables remove` does not clear the declaration — the table leaves the listing but the name stays declared and still conflicts. So **a key cannot be retrofitted onto a table declared without one**: declare it with `--key` up front, or use a new table name.
- `tables load` — publishes to an instant-database table from a local file (`--file`), a remote URL (`--url`), a pre-staged upload (`--upload-id`), or a saved query result (`--result-id`, must belong to the target database). Same `--mode`, `--format`, and `--key` flags as the top-level `load` above.
- `tables remove` — drops a table from the instant database.
- `attach` — attaches a **catalog** to an instant database, so the catalog's **live** tables become visible inside that database's query scope. Defaults to the active database; target another with `--database`. `--alias` sets the SQL name the catalog answers to (defaults to the catalog's name). This is how you query an attached catalog's tables and **join across catalogs** — see [Querying across catalogs](#querying-across-catalogs-attach).
- `detach` — removes an attached catalog. Accepts the catalog name/id **or** the alias you attached it under. Defaults to the active database.
- `create --attach <catalog>[=<alias>]` — attach one or more catalogs at creation time (repeatable), e.g. `--attach github --attach salesdb=sales`.
- `attach` — attaches **another instant database** to this one, so its **live** tables become visible inside this database's query scope. Name the other database by name, catalog alias, or id. Defaults to the active database; target another with `--database`. `--alias` sets the SQL name it answers to (defaults to the attached database's own catalog alias). **Required when that alias is `default`** — the stock name for a database created without `--catalog` — since `default` cannot be attached under its own name. This is how you **join across databases** — see [Querying across databases](#querying-across-databases-attach). Read-only: loads still target your own database, and attaching is not transitive — you see what you attached, not what it attached.
- `detach` — removes an attachment, withdrawing visibility without deleting any data. Accepts the attached database's name/id **or** the alias you attached it under. Defaults to the active database.
- `create --attach <database>[=<alias>]` — attach one or more databases at creation time (repeatable), e.g. `--attach reference --attach salesdb=sales`.

Example:

Expand Down Expand Up @@ -175,30 +175,37 @@ hotdata databases load --catalog airbnb --table bookings --file removed.csv --mo
`removed.csv` carries only the key columns. On a table already declared without
a key, name one per load instead: `--mode upsert --key booking_id`.

#### Querying across catalogs (attach)
#### Querying across databases (attach)

**A `hotdata query` runs inside exactly one instant database** — the active database (`hotdata databases use <id>`) or the one named by `--database`. With none set, the query fails with *"a database is required."* That database's query scope sees **only its own catalog plus any catalogs explicitly attached to it** — a workspace catalog is **not** visible just because it exists. Referencing an unattached catalog fails with *"table '\<catalog\>.\<schema\>.\<table\>' not found."*
**A `hotdata query` runs inside exactly one instant database** — the active database (`hotdata databases use <id>`) or the one named by `--database`. With none set, the query fails with *"a database is required."* That database's query scope sees **only its own catalog plus whatever is explicitly attached to it** — another database is **not** visible just because it exists. Referencing something unattached fails with *"table '\<catalog\>.\<schema\>.\<table\>' not found."*

To query an attached catalog's tables, or **join a managed table against an attached catalog's table in one query**, attach the catalog to the database first. The catalog's data stays **live** (synced) — this is not a copy:
To read another database's tables, or **join your own tables against them in one query**, attach it first. The data stays **live** — this is not a copy:

```
# Attach the 'github' catalog (live) to the active database under alias 'gh'
hotdata databases attach github --alias gh
# Attach the 'reference' database to the active one under alias 'ref'
hotdata databases attach reference --alias ref

# Now both the database's own tables and the attached catalog are in scope:
hotdata query "SELECT * FROM gh.github.issues WHERE state = 'OPEN' LIMIT 10"
# Now both this database's own tables and the attached one are in scope:
hotdata query "SELECT * FROM ref.public.regions LIMIT 10"

# Cross-catalog join: a managed table JOINed against the live attached-catalog table
# Cross-database join: your table JOINed against the attached one, live
hotdata query "
SELECT t.id, i.title
SELECT t.id, r.name
FROM mycatalog.public.tickets t
JOIN gh.github.issues i ON i.number = t.gh_issue
JOIN ref.public.regions r ON r.id = t.region_id
"

hotdata databases detach gh # when finished (optional)
hotdata databases detach ref # when finished (optional)
```

Without `--alias`, the catalog answers to its own name (`github.github.issues`). Do **not** export a catalog to parquet just to query it — attach is the live, sync-preserving path.
Without `--alias`, the attached database answers to its own catalog alias
(`reference.public.regions`). Notes worth knowing:

- **Read-only.** Loads always target your own database's catalog, never an attached one.
- **Not transitive.** You see the database you attached, not the ones *it* has attached.
- **The source cannot be deleted while you hold it** — that delete is refused until you detach. An *expiring* source is still removed on its `expires_at`, so check that date before relying on one.

Do **not** export a database to parquet just to query it — attach is the live path.

### Tables

Expand All @@ -216,7 +223,7 @@ hotdata databases tables show <catalog.schema.table|schema.table> [--output tabl

**`databases tables show`**
- Fetches column definitions (`COLUMN`, `DATA_TYPE`, `NULLABLE`) for a single table.
- **`catalog.schema.table`** — three-part form; the catalog resolves to an instant database or an attached source by name.
- **`catalog.schema.table`** — three-part form; the catalog resolves to an instant database or an attached database by name.
- **`schema.table`** — two-part form; uses the active database (errors if none is set).
- Copy the name directly from `databases tables list` output — both forms match what `list` prints.
- **Always use `databases tables show` to inspect columns before writing queries.**
Expand Down Expand Up @@ -247,7 +254,7 @@ hotdata query status <query_run_id>
```

- Default output is `table` (row count and execution time).
- **A query runs inside one instant database** (active database or `--database`); with none set it fails *"a database is required."* The scope sees the database's own catalog **plus any attached catalogs only**. To query an attached catalog's tables or join across catalogs, attach the catalog first — see [Querying across catalogs (attach)](#querying-across-catalogs-attach).
- **A query runs inside one instant database** (active database or `--database`); with none set it fails *"a database is required."* The scope sees the database's own catalog **plus whatever is attached to it only**. To read another database's tables or join across databases, attach it first — see [Querying across databases (attach)](#querying-across-databases-attach).
- Use `hotdata databases tables list` and `hotdata databases tables show` for discovery — not `information_schema` via `query`. (Discovery lists every workspace table; queryability still requires the table's catalog to be in the active database's scope.)
- **PostgreSQL dialect.** Quote non-lowercase columns with double quotes. To write DuckDB/Postgres/Snowflake SQL instead, pass `--dialect` (server-side transpile, read-only queries) — details in **`hotdata-analytics`**.
- Async runs return `query_run_id` → poll with `query status <id>` (do not re-run the same heavy SQL). `query status` exit codes: `0` succeeded, `1` failed, `2` still running (poll again), `3` succeeded but the result is a truncated/incomplete preview.
Expand Down
4 changes: 2 additions & 2 deletions skills/hotdata/references/DATA_MODEL.template.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,11 @@ For each business entity:
- **Primary tables:** `catalog.schema.table`
- **Key columns:**

## Cross-catalog joins
## Cross-database joins

Document safe join paths and caveats (fan-out, timing, different refresh cadence, type mismatches).

> A cross-catalog join runs inside one instant database; each catalog it touches must be **attached** to that database (`hotdata databases attach <catalog>`) so its live tables are in query scope. Note here which catalogs a join requires attached, and the alias each is attached under. See **`hotdata`** skill → Querying across catalogs.
> A cross-database join runs inside one instant database; every other database it touches must be **attached** to that one (`hotdata databases attach <database>`) so its live tables are in query scope. Note here which databases a join requires attached, and the alias each is attached under. See **`hotdata`** skill → Querying across databases.

## Search & index summary (optional)

Expand Down
6 changes: 3 additions & 3 deletions skills/hotdata/references/MODEL_BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Optional **deep pass** for a single authoritative markdown document stored as **

## 1. Discover catalogs and tables

List the catalogs you can query — instant databases you own and any attached catalogs — and the tables they expose:
List the catalogs you can query — your instant database and anything attached to it — and the tables they expose:

```bash
hotdata databases list # instant databases (catalogs you own)
Expand Down Expand Up @@ -74,7 +74,7 @@ For each table, capture where reasonable:
2. **Primary keys** — `id`, `<entity>_id`, or composite patterns from names + types.
3. **Foreign keys** — `_id` / `_fk` / name matches to other tables; confirm with connector docs when possible.
4. **Parent–child** — Flattened API/JSON tables (often nested names) and dlt parent keys.
5. **Cross-catalog** — Same logical entity in two catalogs (keys, type mismatches, caveats).
5. **Cross-database** — Same logical entity in two databases (keys, type mismatches, caveats).

For **small** schemas (e.g. ≤5 tables in a domain), a short **ASCII diagram** helps. For larger ones, group by domain in prose (e.g. billing, identity, product).

Expand Down Expand Up @@ -114,7 +114,7 @@ This Markdown body is what you store as **context:DATAMODEL** (`hotdata database
- **Overview** — Domains and what the workspace is for.
- **Per catalog** — Optional subsection per source; for **deep** models, **repeat** one block per `catalog.schema.table` (grain, column table with name/type/nullable/PK-FK/notes, relationships, queryability, caveats)—the template’s single `####` heading is a pattern to copy for each table.
- **Instant databases** — Same treatment as catalog tables where relevant.
- **Cross-catalog joins** — Keys, semantics, type caveats.
- **Cross-database joins** — Keys, semantics, type caveats.
- **Search / index summary** — Table, column, index status, intended use.

If the workspace has **many** tables (e.g. 50+), add a **table of contents** after the overview (catalog → table counts).
Expand Down
Loading
Loading