Add bump-serverless-env-version skill

.agents/rules/template-versions.md

@@ -20,7 +20,7 @@ Bundle templates pin the runtime versions they render. Most pins are macros in `
 
 **RULE: Keep the serverless environment version, the Python pins, the DB Connect pin, and `latest_lts_dbr_version` mutually compatible. This is a hard constraint.** A serverless environment version dictates a runtime Python version, and the DB Connect pin must support that Python. The DBR LTS pin belongs in the same check rather than being treated as independent — the two have drifted before: `latest_lts_dbr_version` reached 16.4 in #3671 while the DB Connect pin stayed on 15.4 until #6378. Cross-check the [serverless environment version release notes](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/) and the [DB Connect requirements](https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements) whenever you change any of the four.
 
-**RULE: Within that constraint, keep `default`'s `conservative_db_connect_version_spec` as low as compatibility allows.** The DB Connect client is only forward-compatible (it reaches compute of its own version and higher), so the lowest compatible pin maximizes the range of DBR versions a customer can connect to; a higher pin rules out customers on older DBR, and they can move to a newer version themselves after initializing the template. The floor is the *oldest* release that passes two tests: it is still supported, *and* the DBR version it targets runs `python_version_spec`'s Python on the cluster — not merely that the client supports that Python. Support status only rules out the short-lived non-LTS releases; an LTS stays supported for years, so older LTS releases are ruled out by the runtime test instead. DB Connect 15.4 is the worked example: it is still supported and its client supports Python 3.12, but DBR 15 clusters run Python 3.11, so it fails the second test and 16.4 is the floor. Never read this as "pin the current DBR LTS" — that is the newest release, the opposite of a floor, and #6378 rejected exactly that bump. Keep the upper bound at the next minor release, so the spec stays inside one minor: widening it lets a freshly initialized project resolve a client far newer than the floor advertises, which is the opposite of conservative. Do not raise the pin merely to match a newer serverless environment version — raise it only when the constraint above forces it, or when the pinned release falls out of support (see the [DBR release notes](https://docs.databricks.com/aws/en/release-notes/runtime/) for supported versions). See PR #3897 and PR #6378 for prior history.
+**RULE: Within that constraint, keep `default`'s `conservative_db_connect_version_spec` as low as compatibility allows.** The DB Connect client is only forward-compatible (it reaches compute of its own version and higher), so the lowest compatible pin maximizes the range of DBR versions a customer can connect to; a higher pin rules out customers on older DBR, and they can move to a newer version themselves after initializing the template. The floor is the *oldest* release that passes two tests: its DBR line has more than six months of support remaining, *and* the DBR version it targets runs `python_version_spec`'s Python on the cluster — not merely that the client supports that Python. The support buffer prevents freshly generated projects from immediately approaching end of support. DB Connect 15.4 is the worked example: its client supports Python 3.12, but DBR 15 clusters run Python 3.11, so it fails the second test and 16.4 is the floor. Never read this as "pin the current DBR LTS" — that is the newest release, the opposite of a floor, and #6378 rejected exactly that bump. Keep the upper bound at the next minor release, so the spec stays inside one minor: widening it lets a freshly initialized project resolve a client far newer than the floor advertises, which is the opposite of conservative. Do not raise the pin merely to match a newer serverless environment version — raise it only when the constraint above forces it, or when the pinned DBR line has six months or less of support remaining. Use the [DBR release notes](https://docs.databricks.com/aws/en/release-notes/runtime/) to choose the oldest compatible replacement with more than six months of support remaining. See PR #3897 and PR #6378 for prior history.
 
 This rule governs `default`'s pin. `default-scala` deliberately does the opposite: it tracks the newest LTS in `dbr_version`. Its client in `template/{{.project_name}}/build.sbt.tmpl` uses that macro as an open-ended `.+` dependency, while its classic-cluster `spark_version` combines `dbr_version` with `scala_major_minor_version`. Keep `dbr_version` and `scala_major_minor_version` compatible with the selected DBR release. Do not lower `dbr_version`, or pin the client dependency, to satisfy the rule above.
 

.agents/skills/bump-serverless-env-version/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: bump-serverless-env-version
+description: "Use when bumping or upgrading the default serverless environment version shipped by Databricks bundle templates, including its coupled Python and DB Connect pins."
+user-invocable: true
+allowed-tools: Read, Edit, Write, Bash, Glob, Grep, WebFetch, AskUserQuestion
+---
+
+# Bump the serverless environment version
+
+Read `.agents/rules/template-versions.md` first for compatibility and
+synchronization policy. Never hand-edit generated acceptance output.
+
+## 1. Resolve a compatible version set
+
+Use the requested environment version, or the newest published version when none
+was specified. Confirm its runtime Python version from the
+[environment release notes](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/)
+and [databricks/environments](https://github.com/databricks/environments).
+
+Apply `.agents/rules/template-versions.md` to the environment, Python, and DB
+Connect pins. Check Python compatibility in the
+[DB Connect requirements](https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements).
+Advance `conservative_db_connect_version_spec` when compatibility requires it or its
+DBR line has six months or less of support remaining. Use the supported-LTS table in
+the [Databricks Runtime release notes](https://docs.databricks.com/aws/en/release-notes/runtime/)
+to choose the oldest compatible replacement with more than six months of support remaining.
+Do not advance it merely to match the environment version.
+
+## 2. Update every template source
+
+Find both macro definitions and hardcoded literals; trust the search results:
+
+```bash
+grep -rn 'environment_version\|environment-version\|requires-python\|default_python_version' libs/template/templates/
+```
+
+Inspect every hit and change only values that pin a version. Re-run the command
+after editing and confirm every version-bearing template hit uses the target.
+If all sources and coupled pins already match the resolved set and the worktree has
+no partial bump, report that no changes are needed and stop.
+
+Current sources include:
+
+- `serverless_environment_version` definitions in `default/library/versions.tmpl`
+  and `dbt-sql/library/versions.tmpl`;
+- a hardcoded value in the `default-scala` job template;
+- three hardcoded values in `lakeflow-integrations`, including its
+  `--environment-version` argument.
+
+In `default/library/versions.tmpl`, update `python_version_spec` and
+`default_python_version` when the runtime Python version changes. Update
+`conservative_db_connect_version_spec` only under the compatibility and support rule
+above. Keep the version-specific compatibility comments accurate even when their pin
+is unchanged. Do not synchronize unrelated DBR or SQL-template DB Connect macros.
+
+Update version-specific examples in `.agents/rules/template-versions.md` so its
+policy remains accurate; do not change the policy itself as part of the bump.
+
+Also inspect `defaultServerlessVersion` in `libs/localenv/envkey.go`. It is the
+product-spec fallback for `databricks environments setup-local`, not a template
+pin; do not infer that it should move with the templates. Change it only when the
+user or current product specification also requires the fallback to move, and first
+confirm `python/serverless/serverless-v{N}/pyproject.toml` exists in
+`databricks/environments`. Then update default-version help, error, and test
+expectations under `libs/localenv` and `cmd/environments`; verify with
+`go test ./libs/localenv ./cmd/environments`, then update and verify with:
+
+```bash
+go test ./acceptance -run '^TestAccept/localenv' -update -timeout=60m
+go test ./acceptance -run '^TestAccept/localenv' -timeout=60m
+```
+
+Otherwise leave it unchanged and record why. Do not change the intentionally older
+SSH fixtures in `acceptance/ssh/connect-serverless-*`.
+
+## 3. Regenerate and verify goldens
+
+Update and verify the acceptance suite:
+
+```bash
+./task test-update
+go test ./acceptance -run '^TestAccept$' -timeout=60m
+```
+
+The non-update command must pass. Update mode selects covering `EnvMatrix` variants;
+the non-update run verifies every variant against the regenerated goldens.
+
+## 4. Add the changelog fragment
+
+Add `.nextchanges/bundles/serverless-environment-version-v{N}.md`. Follow the
+`pr-checklist` skill's changelog conventions. Describe a benefit stated in the
+target version's release notes, link the actual version page, and mention the DB
+Connect version only if it changed.
+
+Cross-check the final source and generated-output footprint against the prior
+template bumps in [PR #3897](https://github.com/databricks/cli/pull/3897) and
+[PR #6378](https://github.com/databricks/cli/pull/6378). Explain material
+differences in the final handoff or PR description. Prior PRs are not sources of
+truth: revalidate their compatibility decisions, wording, URLs, and file lists.
+
+## 5. Finish only when requested
+
+**Required sub-skill:** use `pr-checklist`, run its checks, and inspect the final
+diff. If formatting or linting changes generated acceptance files, fix the source
+and regenerate them.
+
+Commit, push, or create/update a PR only when the user explicitly requests that
+operation. When requested, follow `pr-checklist` rather than duplicating its commit
+and PR-body instructions here.
+
+When creating a PR for the bump, suggest `lennartkats-db` as a reviewer.