Skip to content

Bump default serverless environment version to 5 #6378

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rugpanov merged 7 commits into from
Aug 27, 2026
Merged

Bump default serverless environment version to 5 #6378

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
af3f027
Bump default serverless environment version to 5
rugpanov Aug 25, 2026
5b1d7b1
Regenerate pipelines init/e2e goldens for env version 5
rugpanov Aug 25, 2026
3f41ff3
Address review: bump DB Connect floor to 16.4, pin Python to 3.12
rugpanov Aug 25, 2026
d8c43f5
Address review: match DB Connect to serverless v5 (18.0), link releas…
rugpanov Aug 26, 2026
989250c
dbt-sql: align unused DB Connect macro to 18.0 and flag for removal
rugpanov Aug 26, 2026
8d3cfc2
Revert DB Connect floor to 16.4 (conservative, forward-compatible)
rugpanov Aug 27, 2026
adf3fd6
Simplify changelog fragment per review
rugpanov Aug 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .nextchanges/bundles/serverless-environment-version-v5.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Bundle templates now use serverless [environment version 5](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/five), which offers better performance, and `databricks-connect` 16.4.
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/dbt-sql/output/my_dbt_sql/resources/my_dbt_sql.job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,6 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "4"
environment_version: "5"
dependencies:
- dbt-databricks>=1.8.0,<2.0.0
4 changes: 2 additions & 2 deletions acceptance/bundle/templates/default-minimal/python/output/my_default_minimal/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name = "my_default_minimal"
version = "0.0.1"
authors = [{ name = "[USERNAME]" }]
requires-python = ">=3.10,<3.13"
requires-python = ">=3.12,<3.13"
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand All @@ -16,7 +16,7 @@ dev = [
"pytest",
"ruff",
"databricks-dlt",
"databricks-connect>=15.4,<15.5",
"databricks-connect>=16.4,<16.5",
"ipykernel",
]

Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/default-python/classic/out.compare-vs-serverless.diff
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@
- environments:
- - environment_key: default
- spec:
- environment_version: "4"
- environment_version: "5"
- dependencies:
- # By default we just include the .whl file generated for the my_default_python package.
- # See https://docs.databricks.com/dev-tools/bundles/library-dependencies.html
Expand Down
4 changes: 2 additions & 2 deletions acceptance/bundle/templates/default-python/classic/output/my_default_python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name = "my_default_python"
version = "0.0.1"
authors = [{ name = "[USERNAME]" }]
requires-python = ">=3.10,<3.13"
requires-python = ">=3.12,<3.13"
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand All @@ -16,7 +16,7 @@ dev = [
"pytest",
"ruff",
"databricks-dlt",
"databricks-connect>=15.4,<15.5",

@yiweidai-db yiweidai-db Aug 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[question]
environment_version 5 actually requires datbaricks-connect>=18.0 to work now.
Is updating the db-connect version for the Python version compatibility rather than the db-connect client <> server compatibility?

@rugpanov rugpanov Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point — bumped to >=18.0,<18.1 to match v5's DB Connect 18 (client↔server compat). Verified: uv sync resolves databricks-connect==18.0.9, v5's shipped version. 18.0 is also Python 3.12, so the Python pin is unchanged.

@yiweidai-db yiweidai-db Aug 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rugpanov

Bumping the db-connect version from 15 to 18 seems aggressive. I heard from @lennartkats-db that the db-connect version should only be bumped up when it reaches LTS. Can you please help confirm with @lennartkats-db that this upgrade is good?

@lennartkats-db lennartkats-db Aug 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this is the conservative_db_connect_version_spec value; see the comments there.

Looking at DBR versions more closely, we should just bump to 16:

With the environment tooling we should be able to let customers update to a more recent version, but out of the box we should be conservative to maximize the number of supported DBR versions.

@rugpanov rugpanov Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted to >=16.4,<16.5 per this. Thank you!

@Hangzhi Hangzhi Aug 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document this conservative db-connect client upgrade rule in .agents and .claude to let the next upgrade be smoother.

@lennartkats-db lennartkats-db Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 please review if we can give agents some more guidance here. That might take the form of an upgrade skill focused on DBR+client version+db connect+Python? This could be a followup PR, though.

For just conservative_db_connect_version_spec, it might suffice to just copy-edit the comment above the field with more crisp guidance. Which could be worth it for the present PR unless we want to add the skill.

@rugpanov rugpanov Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like both ideas, will address in the follow-up PR and tag you.

"databricks-connect>=16.4,<16.5",
"ipykernel",
]

Expand Down
2 changes: 1 addition & 1 deletion ...undle/templates/default-python/classic/output/my_default_python/src/sample_notebook.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@
"dependencies": [
"--editable .."
],
"environment_version": "4"
"environment_version": "5"
},
"language": "python",
"notebookMetadata": {
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/default-python/serverless/out.plan_after_deploy_dev.direct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"dependencies": [
"/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1-py3-none-any.whl"
],
"environment_version": "4"
"environment_version": "5"
}
}
],
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/default-python/serverless/out.plan_after_deploy_prod.direct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"dependencies": [
"/Workspace/Users/[USERNAME]/.bundle/my_default_python/prod/artifacts/.internal/my_default_python-0.0.1-py3-none-any.whl"
],
"environment_version": "4"
"environment_version": "5"
}
}
],
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/default-python/serverless/out.plan_dev.direct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"dependencies": [
"/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1-py3-none-any.whl"
],
"environment_version": "4"
"environment_version": "5"
}
}
],
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/default-python/serverless/out.plan_prod.direct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"dependencies": [
"/Workspace/Users/[USERNAME]/.bundle/my_default_python/prod/artifacts/.internal/my_default_python-0.0.1-py3-none-any.whl"
],
"environment_version": "4"
"environment_version": "5"
}
}
],
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/default-python/serverless/out.requests.dev.direct.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@
"dependencies": [
"/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1-py3-none-any.whl"
],
"environment_version": "4"
"environment_version": "5"
}
}
],
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/default-python/serverless/out.requests.dev.terraform.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@
"dependencies": [
"/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1-py3-none-any.whl"
],
"environment_version": "4"
"environment_version": "5"
}
}
],
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/default-python/serverless/out.requests.prod.direct.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@
"dependencies": [
"/Workspace/Users/[USERNAME]/.bundle/my_default_python/prod/artifacts/.internal/my_default_python-0.0.1-py3-none-any.whl"
],
"environment_version": "4"
"environment_version": "5"
}
}
],
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/default-python/serverless/out.requests.prod.terraform.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@
"dependencies": [
"/Workspace/Users/[USERNAME]/.bundle/my_default_python/prod/artifacts/.internal/my_default_python-0.0.1-py3-none-any.whl"
],
"environment_version": "4"
"environment_version": "5"
}
}
],
Expand Down
4 changes: 2 additions & 2 deletions ...ptance/bundle/templates/default-python/serverless/output/my_default_python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name = "my_default_python"
version = "0.0.1"
authors = [{ name = "[USERNAME]" }]
requires-python = ">=3.10,<3.13"
requires-python = ">=3.12,<3.13"
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand All @@ -16,7 +16,7 @@ dev = [
"pytest",
"ruff",
"databricks-dlt",
"databricks-connect>=15.4,<15.5",
"databricks-connect>=16.4,<16.5",
"ipykernel",
]

Expand Down
2 changes: 1 addition & 1 deletion ...templates/default-python/serverless/output/my_default_python/resources/sample_job.job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "4"
environment_version: "5"
dependencies:
# By default we just include the .whl file generated for the my_default_python package.
# See https://docs.databricks.com/dev-tools/bundles/library-dependencies.html
Expand Down
2 changes: 1 addition & 1 deletion ...le/templates/default-python/serverless/output/my_default_python/src/sample_notebook.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@
"dependencies": [
"--editable .."
],
"environment_version": "4"
"environment_version": "5"
},
"language": "python",
"notebookMetadata": {
Expand Down
2 changes: 1 addition & 1 deletion ...bundle/templates/default-scala/output/my_default_scala/resources/my_default_scala.job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,6 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "4"
environment_version: "5"
java_dependencies:
- ${workspace.artifact_path}/.internal/my_default_scala-assembly-0.1.jar
4 changes: 2 additions & 2 deletions ...ce/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name = "my_lakeflow_pipelines"
version = "0.0.1"
authors = [{ name = "[USERNAME]" }]
requires-python = ">=3.10,<3.13"
requires-python = ">=3.12,<3.13"
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand All @@ -16,7 +16,7 @@ dev = [
"pytest",
"ruff",
"databricks-dlt",
"databricks-connect>=15.4,<15.5",
"databricks-connect>=16.4,<16.5",
"ipykernel",
]

Expand Down
2 changes: 1 addition & 1 deletion ...lates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/resources/sample_job.job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,4 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "4"
environment_version: "5"
2 changes: 1 addition & 1 deletion ...emplates/lakeflow-pipelines/sql/output/my_lakeflow_pipelines/resources/sample_job.job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,4 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "4"
environment_version: "5"
4 changes: 2 additions & 2 deletions acceptance/bundle/templates/pydabs/init-classic/output/my_pydabs/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name = "my_pydabs"
version = "0.0.1"
authors = [{ name = "[USERNAME]" }]
requires-python = ">=3.10,<3.13"
requires-python = ">=3.12,<3.13"
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand All @@ -16,7 +16,7 @@ dev = [
"pytest",
"ruff",
"databricks-dlt",
"databricks-connect>=15.4,<15.5",
"databricks-connect>=16.4,<16.5",
"ipykernel",
"databricks-bundles==0.0.0",
]
Expand Down
4 changes: 2 additions & 2 deletions acceptance/pipelines/e2e/output/lakeflow_project/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name = "lakeflow_project"
version = "0.0.1"
authors = [{ name = "[USERNAME]" }]
requires-python = ">=3.10,<3.13"
requires-python = ">=3.12,<3.13"
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand All @@ -16,7 +16,7 @@ dev = [
"pytest",
"ruff",
"databricks-dlt",
"databricks-connect>=15.4,<15.5",
"databricks-connect>=16.4,<16.5",
"ipykernel",
]

Expand Down
2 changes: 1 addition & 1 deletion acceptance/pipelines/e2e/output/lakeflow_project/resources/sample_job.job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,4 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "4"
environment_version: "5"
4 changes: 2 additions & 2 deletions acceptance/pipelines/init/python/output/my_python_project/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name = "my_python_project"
version = "0.0.1"
authors = [{ name = "[USERNAME]" }]
requires-python = ">=3.10,<3.13"
requires-python = ">=3.12,<3.13"
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand All @@ -16,7 +16,7 @@ dev = [
"pytest",
"ruff",
"databricks-dlt",
"databricks-connect>=15.4,<15.5",
"databricks-connect>=16.4,<16.5",
"ipykernel",
]

Expand Down
2 changes: 1 addition & 1 deletion acceptance/pipelines/init/python/output/my_python_project/resources/sample_job.job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,4 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "4"
environment_version: "5"
2 changes: 1 addition & 1 deletion acceptance/pipelines/init/sql/output/my_sql_project/resources/sample_job.job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,4 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "4"
environment_version: "5"
5 changes: 3 additions & 2 deletions libs/template/templates/dbt-sql/library/versions.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,14 @@
15.4.x-scala2.12
{{- end}}

{{/* TODO: unused — no dbt-sql template references this macro. Remove it. */}}
{{define "latest_lts_db_connect_version_spec" -}}
>=15.4,<15.5
>=16.4,<16.5
{{- end}}

{{/* The serverless environment version to use.
* See https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/
*/}}
{{define "serverless_environment_version" -}}
4
5
{{- end}}
2 changes: 1 addition & 1 deletion ...mplates/default-scala/template/{{.project_name}}/resources/{{.project_name}}.job.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "4"
environment_version: "5"
java_dependencies:
- ${workspace.artifact_path}/.internal/{{.project_name}}-assembly-{{template `version` .}}.jar
{{- else}}
Expand Down
30 changes: 15 additions & 15 deletions libs/template/templates/default/library/versions.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,40 +5,40 @@
16.4.x-scala2.12
{{- end}}

{{/* A safe version of DB Connect that is compatible with at least half the
* clusters running in production.
{{/* A conservative version of DB Connect for local development.
*
* We need to be very conservative in updating this, since a newer version can
* only connect to compute of that same version and higher. If the version is
* deemed too old, customers can update the version themselves after initializing
* the template.
* DB Connect is only forward-compatible: a client connects to compute of the
* same version and higher. We keep this conservative (low) so a freshly
* initialized project can reach the widest range of DBR versions; a high
* version would rule out customers on older DBR. We use 16.4 rather than an
* older release because DBR 15 is no longer supported and used Python 3.11,
* whereas 16.4 uses Python 3.12 (matching the serverless environment and the
* template's Python pin). Customers can move to a newer version themselves
* after initializing the template.
*
* See https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements
* for DB Connect release notes and version compatibility.
*/}}
{{define "conservative_db_connect_version_spec" -}}
>=15.4,<15.5
>=16.4,<16.5
{{- end}}

{{/* The serverless environment version to use.
* See https://docs.databricks.com/release-notes/runtime/index.html for available versions.
*/}}
{{define "serverless_environment_version" -}}
4
5
{{- end}}

{{/* Python version specification for the template.
* This needs to be compatible with the version of DB Connect and the serverless_environment_version.
*
* Serverless environment version 4 (and upcoming v5), DBR 16, and DBR 17 all use Python 3.12.
Comment thread
lennartkats-db marked this conversation as resolved.
* DB Connect 15 supports all of these and is compatible with Python 3.10-3.12.
*
* We use >=3.10,<3.13 rather than pinning to 3.12 specifically because overly narrow pinning
* means users always need to use uv or similar tools to install the exact version. A broader
* range improves usability while maintaining compatibility with both DB Connect and serverless.
* Serverless environment versions 4 and 5, DBR 16, and DBR 17 all use Python 3.12.
* DB Connect 16.4 requires Python 3.12, so we pin the project to 3.12 to match both
* DB Connect and the serverless runtime.
*/}}
{{define "python_version_spec" -}}
>=3.10,<3.13
>=3.12,<3.13
{{- end}}

{{/* The default Python version used in notebook metadata.
Expand Down
Loading