Document conservative DB Connect / serverless env upgrade rule - #6407
Conversation
baf5cb3 to
507d261
Compare
Integration test reportCommit: a2925dc
Top 10 slowest tests (at least 2 minutes):
|
|
@yiweidai-db @Hangzhi — follow-up to #6378 addressing your review requests: this documents the conservative DB Connect upgrade rule and the env-version / Python / DB Connect consistency invariant. Couldn't add you as formal reviewers (GitHub requires repo collaborator access), so flagging here — would appreciate your eyes. cc @lennartkats-db |
507d261 to
2746fb8
Compare
anton-107
left a comment
anton-107
left a comment
There was a problem hiding this comment.
Docs-only follow-up to #6378, and the direction is right — #6378's aggressive bump is exactly the kind of thing worth encoding as a durable rule. No rendered output changes (the comment lives in a {{/* */}} block), so untouched goldens are correct.
That said, I think the new rule file has a few factual problems that would misdirect the next person doing a bump, plus one convention break. Requesting changes on the first three.
1. Rule 3 enshrines an accident as intentional design
Rule 3 says each SQL template "pins its own latest_lts_db_connect_version_spec" and lists it among values that "must NOT be synced." But at this branch's head:
dbt-sql:>=16.4,<16.5, carrying an inline{{/* TODO: unused — no dbt-sql template references this macro. Remove it. */}}default-sql:>=15.4,<15.5
A repo-wide grep finds zero references to latest_lts_db_connect_version_spec outside those two define blocks — it's dead in both templates, and the divergence is a leftover from #6378 bumping one dead copy. The doc reframes that as a deliberate invariant to preserve, which is the opposite of the truth.
The PR description defers dead-macro cleanup in dbt-sql to a separate PR, which is fine — but then this rule shouldn't cite the macro as evidence of intentional divergence. Either drop that clause or say plainly it's dead pending removal.
2. Rules 1 and 2 conflict at the exact decision point the doc exists for
Rule 1: bump DB Connect "only when the pinned DBR release falls out of support — never just to match the newest serverless environment version."
Rule 2: keep the env version, Python pins, and DB Connect "mutually compatible."
When serverless env 6 lands requiring a Python that DB Connect 16.4 doesn't support, Rule 2 forces the bump Rule 1 forbids. That's precisely the next bump this doc is meant to guide, and it gives two contradictory answers. Worth noting #6378's own justification invoked both reasons at once.
Suggest stating the precedence explicitly — e.g. compatibility (Rule 2) is a hard constraint, and Rule 1 governs discretionary bumps within it.
3. Missing .cursor/rules/template-versions.mdc symlink
All 9 existing .agents/rules/*.md files have a matching symlink in .cursor/rules/ (10 .mdc entries = 9 symlinks + 00-agents-context.mdc). This new rule has none, so Cursor users never load it:
ln -s ../../.agents/rules/template-versions.md .cursor/rules/template-versions.mdc4. The doc hardcodes the numbers it governs
It embeds 5, 16.4, 15.4, and 3.12 in prose, so bumping any pin now requires remembering to edit this doc too, with nothing enforcing it — the rule file goes stale in exactly the scenario it's written for. Prefer describing the invariant and pointing at the .tmpl files as the source of truth. The (`5`) parenthetical in Rule 3 is the clearest case: the shape of the rule ("same value in both") survives a bump, the literal doesn't.
5. Two inaccuracies in the intro paragraph
- It calls the listed five macros "the full set" for
default/, but that template also defineslatest_databricks_bundles_version. - "Each bundle template pins the runtime versions ... in
library/versions.tmpl" isn't true fordefault-scala, which pinsdbr_version(17.3),databricks_cli_version,scala_version, andjava_versioninlibrary/template_variables.tmpl. That file isn't matched by the frontmatter globs either, so a version-pin rule silently excludes a template full of version pins. Either widen the globs or scope the opening sentence to the Python/SQL templates.
Smaller notes
- The rewritten
.tmplcomment keeps the substantive content (forward-compat, the DBR-15/Python-3.11 rationale) — good, and in line with.agents/rules/comments.mdon not dropping context from comments you didn't write. One small regression: the URL loses its "for DB Connect release notes and version compatibility" descriptor and is now a bare trailing link. - The rule paragraphs run long — 4–6 sentences each after the bolded rule. Existing rules (
changelog.md,dresources.md) are tighter, andcomments.md's "AI-generated comments trend long and explanatory" applies to rule prose too. - No
.nextchanges/fragment needed (not user-visible) — correct as-is.
dbe8b16 to
9374164
Compare
|
Thanks @anton-107 — addressed all of these:
Smaller notes: restored the "for DB Connect release notes and version compatibility" descriptor on the comment's URL, and tightened the rule paragraphs. PTAL. |
Follow-up to #6378. Tightens the comment above conservative_db_connect_version_spec into crisp upgrade guidance, and adds an .agents/rules doc capturing the conservative DB Connect upgrade rule and the env-version / Python / DB Connect consistency invariant so the next serverless environment bump is smoother. Co-authored-by: Isaac <no-reply@databricks.com>
…omment - Drop the claims about default-sql's versions.tmpl and the SQL templates pinning an older DBR, both invalidated by #6420 and #6418 - Attribute the 16.4 DB Connect floor to the Python/serverless-env compatibility constraint instead of DBR 15's support status - Remove the comment's absolute "never bump to match the newest serverless environment", which contradicted the hard compatibility constraint - Extend the sync rule and the globs to the hardcoded environment_version and requires-python occurrences outside library/versions.tmpl - Shrink the versions.tmpl comment to the invariant plus a pointer, so the rule has a single home - Reference the rule file by repo-root path and name ./task test-update-templates Co-authored-by: Isaac <no-reply@databricks.com>
9374164 to
56ed261
Compare
|
Ready for review. Rebased onto current Corrected stale facts. Two claims in the rule file were invalidated by commits that landed after this branch was cut: #6420 deleted Re-attributed the 16.4 floor. It was justified by DBR 15 being out of support. That is not the binding constraint — Python is: 16.4 is the lowest DB Connect release whose Python (3.12) satisfies Removed a contradiction between the two halves. The comment said bump "never to match the newest serverless environment version", while the rule makes env → Python → DB Connect compatibility a hard constraint that does force such a bump (e.g. a future env moving to 3.13). The absolute is gone. Extended the sync rule past the macros. Gave the rule one home. The rule file and the comment were near-verbatim restatements, which is what produced the contradiction above. The comment is now the invariant plus a pointer, so it is a net -2 lines against Also: cross-reference uses a repo-root path (the relative link resolved to a nonexistent Verified: |
anton-107
left a comment
anton-107
left a comment
There was a problem hiding this comment.
Third round. Verified against 56ed261d5 (rebased on a161e5d56).
Round-1 follow-through first: items 1 (dead latest_lts_db_connect_version_spec cited as intentional divergence — the macro now has zero references repo-wide), 2 (Rules 1↔2 contradiction — compatibility is now explicitly the hard constraint), 3 (.cursor/rules/template-versions.mdc symlink) and 5 (intro inaccuracies) are resolved. Item 4 is partial — see finding 1.
The glob extension is a real fix and I checked it exhaustively: sweeping libs/template/templates/ for hardcoded version literals outside the two library files returns exactly four files, and all four are covered.
Two things I'd like changed before this lands, then a set of one-line edits.
1. The re-attributed 16.4 floor is contradicted by the file it documents
Rule 2 (.agents/rules/template-versions.md:21) now says the pin sits at 16.4 "because that is the lowest DB Connect release whose Python (3.12) satisfies python_version_spec". But libs/template/templates/default/library/versions.tmpl:36 — in this same PR's file — states:
Serverless environment versions 4 and 5, DBR 16, and DBR 17 all use Python 3.12.
If the whole DBR 16 line is Python 3.12, then DB Connect 16.0–16.3 also satisfy >=3.12,<3.13 and are lower than 16.4. (acceptance/localenv/dbconnect-compatible-kept-check uses databricks-connect~=16.0 as a realistic pin, so those releases exist.)
The actual floor is support status: 16.4 is the LTS — latest_lts_dbr_version is 16.4.x-scala2.12 — and 16.0–16.3 are out of support. That is what the comment text this commit deleted said, in a looser form.
This inverts the failure mode the commit set out to fix. Rule 2's headline is "keep the pin as low as compatibility allows" and the body names Python as the binding constraint, so an agent applying it literally can lower the pin to 16.1. Out-of-support appears only as a trigger to raise, never as a floor.
Suggest: the floor is the lowest release that is both still supported (today the 16.4 LTS) and whose Python satisfies the pin — two constraints, and Python alone is not what puts it at 16.4. Dropping the 16.4/3.12 literals while you're there also closes round-1 item 4 and the file's own "the .tmpl files are the source of truth for the current values" promise, which Rule 2 is currently the only paragraph to break.
2. The new glob turns Rule 2 on default-scala, where the policy is the opposite
:5 and :10 add libs/template/templates/**/library/template_variables.tmpl. That file is default-scala's only pin site, and it pins dbr_version 17.3 — the newest LTS — deriving its DB Connect dependency from it:
libs/template/templates/default-scala/template/{{.project_name}}/build.sbt.tmpl:9
libraryDependencies += "com.databricks" %% "databricks-connect" % "{{template `dbr_version` .}}.+"
while default sits at 16.4. So the round-1 glob fix now loads "keep conservative_db_connect_version_spec as low as compatibility allows" onto a template that deliberately does the reverse. Python 3.12 spans 16.x and 17.x, so nothing in Rule 2 stops an agent from reading it as license to downgrade 17.3. Worth scoping Rule 2 to default's macro, or stating the Scala template's policy explicitly.
Relatedly, build.sbt.tmpl:9 is the repo's second DB Connect pin and is open-ended (.+) — the one pin that contradicts Rule 2 outright — and it is named nowhere in the doc. Editing dbr_version does load the rule, so this is a prose gap rather than a coverage gap.
3. ./task test-update-templates does not regenerate everything a pin bump touches
:27. That task runs -run '^TestAccept/bundle/templates' (Taskfile.yml:593), and acceptance subtests are named by directory (acceptance/acceptance_test.go:638, t.Run(dir, …) at :500). Twelve subtests have goldens carrying the pinned values; three are outside that prefix:
acceptance/pipelines/init/python/output/my_python_project/pyproject.toml:19—"databricks-connect>=16.4,<16.5", and:5—requires-python = ">=3.12,<3.13"acceptance/pipelines/init/sql/output/my_sql_project/resources/sample_job.job.yml:32—environment_version: "5"acceptance/pipelines/e2e/output/lakeflow_project/pyproject.toml— both pins
These are generated (acceptance/pipelines/init/python/script runs $CLI pipelines init --output-dir output), so a bump changes them and the named command leaves them stale — the agent follows the rule and still gets red CI. .agents/rules/auto-generated-files.md:93 qualifies the same command as "(templates only)" and lists ./task test-update above it for the full sweep; this rule drops the qualifier. Point at ./task test-update, or name both.
This is also the one place the doc restates rather than cites: .agents/rules/testing.md:264 already carries RULE: Use ./task test-update-templates to regenerate materialized templates, and auto-generated-files.md:93 lists it too. Three copies that already disagree on scope is how this finding arose — a cross-reference beats a third restatement.
Smaller
:19— Rule 1's "hard constraint" triple omitslatest_lts_dbr_version, the DB Connect pin's closest neighbour, and the two have demonstrably drifted:latest_lts_dbr_versionreached 16.4 in #3558 whileconservative_db_connect_version_specstayed at 15.4 until #6378. An agent bumping the DBR LTS reads Rule 1 (DBR absent) and Rule 2 ("raise it only when the compatibility constraint above forces it") and concludes nothing forces a DB Connect move.:21— Rule 2 describes the pin as a floor and never explains the<16.5upper bound.>=16.4is a conforming edit under "as low as compatibility allows", and rendered intopyproject.tomlit resolves 17.x into every freshly initialized project — the opposite of conservative, and the golden diff still starts with16.4so it does not read as alarming.:23— Rule 3 omitsdefault_python_version. A future env on Python 3.13 needspython_version_spec,default_python_version(versions.tmpl:47, consumed atsrc/sample_notebook.ipynb.tmpl:149as the notebook kernel version) and the hardcodedrequires-python; the rule names onlyenvironment_versionandrequires-python, leaving notebook metadata stale. "Grep both" also has no clear antecedent.:23— "several templates hardcodeenvironment_version" is two templates (lakeflow-integrations×2 files,default-scala×1). "Two" is more useful than "several" when the instruction is "update every hit".:17— The per-template framing omits the mechanism that makes the sync rule tractable:default-python,default-minimal,lakeflow-pipelinesandpydabsare alias dirs —"template_dir": "../default"in theirdatabricks_template_schema.json— sodefault/library/versions.tmplis the single pin site for five shipped templates. (default-sqlowns atemplate/dir but renders no version pins since #6420.) A reader told "most pins are macros in a template'slibrary/versions.tmpl" will grep for alakeflow-pipelines/library/that does not exist.versions.tmpl:27— theserverless_environment_versioncomment links to the DBR runtime release notes (docs.databricks.com/release-notes/runtime/index.html), where no environment version is listed.dbt-sql/library/versions.tmpl:6and the new rule file both have the right URL. Four lines below the hunk this PR edits, inside the block the PR exists to make legible — worth fixing here.:25— now thatdbt-sqlmatchesdefaultat 16.4, Rule 4 has no concrete example left. Fine as-is, just noting the grounding round 1 asked about is gone rather than replaced.- The comment relocation is defensible — the duplication is what produced the round-1 contradiction.
.agents/rules/comments.md:31is why finding 1 matters: the "why 16.4, not older" context left the comment and arrived misstated.
Verified clean: frontmatter matches the globs + paths convention, symlink matches the other nine, file ends with a newline and has no trailing whitespace, no .nextchanges/ fragment needed, ./task test-update-templates and .agents/rules/auto-generated-files.md both exist, and referenced PRs #3897/#6378 are apt.
One process note, since it has now caught two reviewers of this PR (including me in round 1): claims about dbt-sql's pins go stale fast. At this base, dbt-sql pins 16.4.x-scala2.12 (#6418) and the dead latest_lts_db_connect_version_spec is gone entirely (#6420) — worth re-grepping rather than trusting either review's history.
Findings 1 and 2 are the blocking ones; the rest are one- or two-line edits.
- The floor is the lowest release that is both still supported (in practice the DBR LTS) and Python-compatible. Attributing it to Python alone was wrong: a whole DBR major line shares one Python version, so 16.0-16.3 also satisfy python_version_spec and are lower than the pin - Drop the 16.4/3.12 literals so the rule stops restating current values - Explain the upper bound: widening it resolves a much newer client than the floor advertises - Scope Rule 2 to default's macro and state default-scala's opposite policy (tracks the newest LTS, derives an open-ended client from build.sbt.tmpl) - Add latest_lts_dbr_version to the compatibility set; it drifted from the DB Connect pin between #3671 and #6378 - Sync rule now covers default_python_version (notebook kernel version) and names the two hardcoding templates instead of saying "several" - Point at ./task test-update: acceptance/pipelines renders the pins too, so test-update-templates leaves those goldens stale - Note the alias templates (template_dir ../default) that share default's pins - Fix the serverless_environment_version comment to link the environment version release notes instead of the DBR runtime notes Co-authored-by: Isaac <no-reply@databricks.com>
|
Round 3 addressed in 1 — the re-attributed floor was wrong, and it was my regression. Confirmed from 2 — Rule 2 is now scoped, and 3 — now points at Smaller items — all applied: One correction to the review: the DBR-LTS/DB-Connect drift is real, but the citation is off by a PR. Searching the file's history for when Rule 4 is left without a concrete example, as you noted — I'd rather it have none than have one that goes stale again, which is what round 1 was about. Verified: |
anton-107
left a comment
anton-107
left a comment
There was a problem hiding this comment.
Round 4. Everything from round 3 is addressed — both blocking items and all the smaller ones. The citation pushback is right, too: c30c45611 is #3671, and at that commit latest_lts_dbr_version was already 16.4.x-scala2.12 while the DB Connect pin was still >=15.4,<15.5. Re-verified clean as well: Rule 3's counts, the glob coverage of all four hardcode sites, the ./task test-update scope claim (Taskfile.yml:588 vs :582), the symlink, and default_python_version → sample_notebook.ipynb.tmpl:149. No staleness this round either — the only template diff between this branch and current main is the file it edits.
Two blocking items remain, both in the same sentence: .agents/rules/template-versions.md:21.
The doc exists to answer one question: which DB Connect version should default pin? The answer today is 16.4, which is what ships. The problem is that following the rule as written does not produce 16.4.
The rule is: pick the lowest version that is (a) still supported and (b) Python-compatible. Applied to the candidates:
| candidate | still supported? | Python 3.12? |
|---|---|---|
| 14.3 | yes, until Feb 2027 | no — 3.10 |
| 15.4 | yes, until Aug 2027 | cluster runs 3.11 |
| 16.0–16.3 | no — short-lived non-LTS | yes |
| 16.4 | yes | yes |
Two tests, each knocking out a different group, landing on 16.4. That structure is right. What is wrong is the wording attached to each test — one makes the rule too aggressive, the other too lax.
1. Blocking — the shortcut for test (a) resolves to DBR 18
:21 glosses the floor as "in practice the current DBR LTS". The current LTS is 18 (released Jun 10 2026), not 16.4. 17.3 LTS is out too, and default-scala already pins dbr_version: 17.3, so the phrase is ambiguous inside this repo alone.
An agent taking that shortcut pins >=18.0,<18.1 — the exact value this PR's parent rejected. In #6378: 18.0 was pushed, @yiweidai-db flagged "Bumping the db-connect version from 15 to 18 seems aggressive", and it was reverted to >=16.4,<16.5 on @lennartkats-db's reasoning. The document written to prevent that bump now instructs it.
The gloss's justification is also inaccurate: "the non-LTS minor releases below it fall out of support first" implies support status is what rules out everything under 16.4, but 14.3 LTS (Feb 2027) and 15.4 LTS (Aug 2027) are both still supported and both lower. Support only rules out 16.0–16.3.
Suggest dropping the "in practice" clause, or replacing it with the accurate shape: support rules out only the short-lived non-LTS releases; older LTSes stay supported ~3 years and are ruled out by Python instead; the floor is the oldest still-supported release whose Python matches — never the newest LTS.
2. Blocking — the wording for test (b) lets 15.4 through
:21 says the release's "Python satisfies python_version_spec". But python_version_spec is the rendered project's requires-python — a client-side constraint — and DB Connect 15.4 satisfies it. #3897, cited by this rule as prior history, says so outright:
DB Connect 15 supports all DBR versions above and supports Python 3.12
15.4 LTS is supported until Aug 2027, so it passes both tests, and the rule ends up telling you to pin lower than what ships.
What actually disqualifies 15.4 is the other Python — the DBR 15 cluster runs 3.11. #3897's next line: "DB Connect 15 also supports DBR 15, which uses Python 3.11 at runtime". @lennartkats-db's wording in #6378 was "the oldest possible DB Connect version that fully uses Python 3.12", and the comment this PR deletes said "DBR 15 ... used Python 3.11, whereas 16.4 uses Python 3.12". Neither survives: git grep '3\.11' under libs/template/ and .agents/ now returns nothing, so the one fact that explains the floor is recorded nowhere in the repo. That is the comments.md:31 risk — context encoded in a comment, lost on rewrite — actually materializing.
Suggest saying the runtime Python of the matching DBR must satisfy python_version_spec, and keeping DBR 15 / Python 3.11 as the worked example.
…ime Python Round 3 defined the floor as "the lowest release that is both still supported (in practice the current DBR LTS) and whose Python satisfies python_version_spec". Both halves misfire: - "in practice the current DBR LTS" points at the newest LTS, not a floor. default-scala already pins dbr_version 17.3, so the phrase resolves above 16.4 inside this repo alone, and an agent following it lands on the bump #6378 reverted after review. Support status only rules out the short-lived non-LTS releases; an LTS stays supported for years, so 14.3 and 15.4 are still supported and still lower - "whose Python satisfies python_version_spec" is a client-side test, and DB Connect 15.4 passes it (#3897: "DB Connect 15 supports all DBR versions above and supports Python 3.12"). What disqualifies it is the cluster runtime: DBR 15 runs Python 3.11 (#3897, #6378). That fact left the deleted comment and is now recorded nowhere, so restore it as the worked example The floor is now the oldest release passing both tests, with 15.4 as the example and an explicit warning against reading it as "pin the current LTS". Co-authored-by: Isaac <no-reply@databricks.com>
|
Round 4 addressed in 1 — "in practice the current DBR LTS" does point at the newest release. Confirmed without needing the DBR 18 date: 2 — the client-side test does let 15.4 through, and the fact that rules it out was gone. #3897's summary states both halves verbatim: "DB Connect 15 supports all DBR versions above and supports Python 3.12" and "DB Connect 15 also supports DBR 15, which uses Python 3.11 at runtime". So the disqualifier is the cluster runtime, not The floor now reads: the oldest release passing two tests — still supported, and the DBR version it targets runs One limitation worth stating: I could not independently verify that DBR 18 is the current LTS (no web access on my side), so I took that on your word. It does not affect the fix — 17.3 in-repo is already enough to break the old gloss, and the new wording never refers to "the current LTS" as a target. Also merged current Verified: |
anton-107
left a comment
anton-107
left a comment
There was a problem hiding this comment.
Round 5. Both round-4 blocking items are fixed, and the test that matters now passes: applying the rule as written lands on 16.4, which is what ships.
| candidate | still supported? | targeted DBR runs 3.12 on the cluster? |
|---|---|---|
| 14.3 | yes | no — client is on 3.10 |
| 15.4 | yes | no — DBR 15 clusters run 3.11 |
| 16.0–16.3 | no — short-lived non-LTS | yes |
| 16.4 | yes | yes |
Two tests, each knocking out a different group, landing on what versions.tmpl:23 pins. The "never read this as pin the current DBR LTS" guard closes item 1, and "not merely that the client supports that Python" plus the DBR-15/3.11 worked example closes item 2 — with the 3.11 fact back in the repo, which was the comments.md:31 loss at round 4.
Verified clean this round: frontmatter matches the globs + paths convention (dresources.md, style-guide-go.md, telemetry.md); the symlink matches the other nine; Rule 3's counts (default-scala job :34, lakeflow-integrations send_slack_message.job.yml.tmpl:22 + wait_for_run_sensor.job.yml.tmpl:18, and the one hardcoded requires-python at its pyproject.toml.tmpl:5); default_python_version → sample_notebook.ipynb.tmpl:149; :29's scope claim (Taskfile.yml:565 ^TestAccept$ vs :592 ^TestAccept/bundle/templates, and all three acceptance/pipelines/ golden sets still carry the pins); trailing newline, no trailing whitespace; no .nextchanges/ fragment needed.
No staleness either: main has changed no template pin since the merge base — the only drift is #6549's plan_version normalization in the default-python plan goldens, which this PR doesn't touch.
Approving so this can land. One substantive gap left and two one-liners; I'd fold in #1 before merge.
1. The DBR pin has a second, hardcoded site that the sync rule doesn't reach
Rule 3 (:25) covers the serverless env version and Python. The identical hazard exists for DBR, and default-scala is where it lives:
libs/template/templates/default-scala/library/template_variables.tmpl:10— thedbr_versionmacrolibs/template/templates/default-scala/template/{{.project_name}}/resources/{{.project_name}}.job.yml.tmpl:41—spark_version: 17.3.x-scala2.13, a literal, in the classic branch
Every other template renders DBR through a macro (default/…/sample_job.job.yml.tmpl:110, sample_job.py.tmpl:136, dbt-sql/…/{{.project_name}}.job.yml.tmpl:47), so this is the tree's only hardcoded DBR version. It carries the .x-scala<major.minor> shape that dbr_version (bare 17.3) lacks, which is presumably why it was inlined rather than composed from the two macros that already hold both halves.
The failure mode is stated by this PR's own text. Bump dbr_version alone and build.sbt.tmpl:9 resolves a client on the new line while the job cluster stays on the old one; since the client "reaches compute of its own version and higher" (:21), a freshly initialized project can no longer reach the cluster it ships with. Note the asymmetry — moving only spark_version is harmless, so the drift is silent in one direction and breaking in the other.
Nothing catches it. compute_type defaults to serverless (default-scala/databricks_template_schema.json), the only default-scala acceptance test pins "compute_type": "serverless" (acceptance/bundle/templates/default-scala/input.json:3), and record-deployment-history/input.default-scala.json omits the key, so it takes the same default. git grep 17.3.x-scala2.13 -- acceptance/ returns nothing: no golden renders that branch, so ./task test-update (:29) can't surface the drift either.
Cheapest fix is a clause on the default-scala paragraph (:23) — dbr_version is duplicated as a literal in the classic branch of the job template, no golden renders it, move both together — plus spark_version in Rule 3's grep list.
2. "five shipped templates", four named
:17 names default-minimal, default-python, lakeflow-pipelines and pydabs. The fifth is cli-pipelines: hidden and deprecated, but it renders lakeflow-pipelines through Reader: &builtinReader{name: string(LakeflowPipelines)} (libs/template/template.go:87), so it inherits default's pins. It is invisible to the grep the sentence hands you, because that aliasing lives in Go rather than in a databricks_template_schema.json.
Half a sentence is also worth spending on default itself not being shipped — it's absent from databricksTemplates (libs/template/template.go:43) — since that's what makes the count five rather than four-plus-the-base.
3. The globs miss build.sbt.tmpl
:5–:12 reach both library/ files, resources/*.job.yml.tmpl and pyproject.toml.tmpl — every site the rules ask you to edit except one. :23 gives a direct instruction about build.sbt.tmpl ("do not … pin that dependency"), and that file matches no glob, so an agent editing the repo's second DB Connect pin loads no rule. One line: libs/template/templates/**/build.sbt.tmpl.
Smaller
:21is now ~330 words in one paragraph, carrying two tests, a worked example, a "never read this as", the upper-bound rule and the bump trigger. The content is right; the two tests as a short list would make the derivation scannable. It's the paragraph that has grown every round.- Rule 4 (
:27) still has no concrete example. Agreed that none beats a stale one.
|
@anton-107 will address in a follow-up. |
Integration test reportCommit: 6a1530b
830 interesting tests: 786 MISS, 43 FAIL, 1 SKIP
Top 50 slowest tests (at least 2 minutes):
|
4 participants
Changes
Follow-up to #6378 — documents the conservative DB Connect upgrade rule reviewers asked for, so the next serverless env bump goes smoothly.
conservative_db_connect_version_specinto a crisp upgrade rule.agents/rules/template-versions.md: keep the DB Connect pin conservative (bump only when the pinned DBR falls out of support, not to match the newest serverless env), keep env-version / Python / DB Connect mutually compatible, and only keepserverless_environment_versionin sync across templatesWhy
Tests
{{/* */}}block, so rendered output and acceptance goldens are unchangedFollow-ups in separate PRs: a
bump-serverless-env-versionskill, and dead-macro cleanup indbt-sql.This pull request and its description were written by Isaac.