docs(streaming): one --topic vocabulary for kafka and iggy - #312
Conversation
…ontinuous behaves --topic serves every log engine: an entry carries the engine's hierarchy as a path (kafka 'orders', iggy 'events/orders'), so no family-specific flag exists. The help and the skill say so, and the skill gains the streaming facts an agent needs: tail-by-batch on --every, a committed position per topic, keyed upsert on partition+offset, a loud failure at a retention gap, and start=latest as a continuous-only seed. Describes dlthubworker #289 and #290; merge after they deploy.
9c63a07 to
cf339b4
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| /// path: Kafka has one level (`--topic orders`), Iggy two | ||
| /// (`--topic events/orders`). Each topic lands in a destination | ||
| /// table named from the whole path (`events_orders`); put them under | ||
| /// a common prefix with --dest-table-prefix. |
There was a problem hiding this comment.
nit: add Iggy to the --dest-table-prefix doc at line 231 (not blocking).
Line 231 lists the several-table families as SQL --table/--sql, Iceberg, DuckLake, Kafka, REST. An Iggy user follows this line to --dest-table-prefix, then reads a family list that excludes Iggy. The same list in skills/hotdata/SKILL.md:393 also omits iggy.
| # --stream shorthand for --type continuous (still needs --every). | ||
| # Offered only where `hotdata ingest sources fields | ||
| # <family>` says `continuous: yes` — today filesystem, | ||
| # derived, kafka, iggy; the server refuses it elsewhere. |
There was a problem hiding this comment.
nit: drop derived here, or add derived to the family list at line 324 (not blocking).
The word derived appears in no other file of this repo. The family list at line 324 names sql, filesystem, kafka, iggy, iceberg, delta, ducklake, rest. An agent reading this line may call --family derived and take the 422 as a service fault.
| # --config also accepts a bare config object, @- (stdin), or inline JSON. | ||
| # --credentials takes the secret half separately. Keep secrets out of argv. | ||
| # Families: sql, filesystem (buckets), kafka, iceberg, delta, ducklake, rest. | ||
| # Families: sql, filesystem (buckets), kafka, iggy, iceberg, delta, ducklake, rest. |
There was a problem hiding this comment.
nit: add iggy to the --family help in src/commands/datasource.rs (not blocking).
Lines 67 and 96 of that file list sql, filesystem, iceberg, delta, ducklake, kafka, rest, and the error at line 512 repeats the same list. hotdata ingest sources add --help therefore contradicts this skill line. The CLI does not validate --family, so iggy already works.
| return Err( | ||
| "nothing to read — pass --table <name> (SQL, Iceberg, DuckLake), --table-path \ | ||
| <path> (Delta), --topic <name> (Kafka), --format with an optional --glob \ | ||
| <path> (Delta), --topic <name> (Kafka; Iggy as stream/topic), --format with an optional --glob \ |
There was a problem hiding this comment.
super nit: rewrap this string line (not blocking). The line reaches 113 characters, while the other lines of the literal stop near 90.
There was a problem hiding this comment.
Documentation-only change. Topic strings pass through to the selector untouched, so a path-form topic such as events/orders needs no client-side change. Four non-blocking comments on family-list consistency are inline. CI / test was still queued at review time.
1 participant
Stacked on #311. Draft until hotdata-dev/dlthubworker#289 and #290 deploy, since it documents a family and a capability production does not serve yet.
What
--topichelp: one flag for every log engine, entries carry the engine's hierarchy as a path (ordersfor Kafka,events/ordersfor Iggy), table named from the whole path. No new flags, which is the point.iggyjoins the family list; the--topic,--stream, andwrite_modeshorthand notes cover both engines; three agent tips on streaming: tail-by-batch on--everywith a committed position per topic and keyed upsert, retention gaps fail loudly and hold the position,start: latestis a continuous-only seed.Verified
CLI tests pass (the help still carries the "Kafka sources" phrase the contract tests pin). Skill gate passes: 66 commands, 161 flags documented. The commands shown are the exact ones run against the local cluster for both engines.