Add libraries field to the clusters resource - #6365
Open
Sankalp-Mittal wants to merge 47 commits into
Open
Conversation
Collaborator
Integration test reportCommit: 612add7
Top 45 slowest tests (at least 2 minutes):
|
Sankalp-Mittal
force-pushed
the
sankalp-mittal/all-purpose-cluster-libraries
branch
from
August 25, 2026 09:36
e378a6a to
e4e688e
Compare
Sankalp-Mittal
changed the title
Sankalp-Mittal
marked this pull request as ready for review
August 26, 2026 08:19
Contributor
Approval status: pending
|
Collaborator
Author
|
Check if a library is changes and all others are unchanged, does only reinstalling the change library work or can others have a dependency on it and everything needs to be reinstalled |
Sankalp-Mittal
commented
Aug 26, 2026
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/cluster-libraries/default/files... | ||
| Updated clusters.mycluster.libraries | ||
| Files: 3 uploaded, 0 deleted | ||
| Resources: 0 created, 1 changed, 0 deleted, 1 unchanged |
Contributor
There was a problem hiding this comment.
nit: Should we mark it deleted instead of changed?
Collaborator
Author
There was a problem hiding this comment.
this actually counts as an update to the libraries resource, hence marked as changed, delete is only when the entire node is removed
| @@ -34,6 +35,9 @@ package terraform_dabs_map | |||
| // TerraformToDABsFieldMap maps DABs group name → nested TF segments → DABs segment name. | |||
| // Navigate using TF field name segments; DABs is the corresponding DABs name when it differs. | |||
| var TerraformToDABsFieldMap = map[string]RenameTree{ | |||
Contributor
There was a problem hiding this comment.
If this is direct only change, why is this changed?
Collaborator
Author
There was a problem hiding this comment.
Apparently terraform always had a library block, it was just never supported in DABs, which now causes this file to register a rename to libraries. Terraform is explicitly blocked in validate_cluster_libraries.go. I don't think terraform should be supported since we have moved to direct only
Wire the direct engine to treat resources.clusters.*.libraries as a child-resource node, the same way permissions and grants are handled: node/type resolution, reference splitting, and plan node discovery. Co-authored-by: Isaac <no-reply@databricks.com>
Implement ResourceLibraries: installs/uninstalls cluster libraries via the Libraries API, reconciling removed libraries on update and polling for install completion on a running cluster. Registered in all.go. Note: TestAll/clusters.libraries fails until the testserver models the libraries install/uninstall/cluster-status endpoints (next step). Co-authored-by: Isaac <no-reply@databricks.com>
Add stateful fakes for the Libraries API (install, uninstall, cluster-status) so clusters.libraries runs against the in-process server. Add the TestAll fixture and classify libraries as a no-op delete alongside permissions/grants; this greens TestAll/clusters.libraries. Co-authored-by: Isaac <no-reply@databricks.com>
Add clusterLibrariesPattern (resources.clusters.*.libraries) to glob expansion, local-library collection/upload, duplicate-name checking, and patched-wheel swapping, mirroring the job task library wiring. Local whl/jar globs now build, upload to artifact_path/.internal, and rewrite to absolute workspace paths; pypi/maven entries pass through unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Add ValidateClusterLibraries and register it in PreDeployChecks so a libraries block on a cluster errors under the terraform engine instead of being silently dropped. Cluster libraries are direct-only. Mirrors the existing lifecycle.started guard. Co-authored-by: Isaac <no-reply@databricks.com>
Adding the clusters libraries field makes DABs libraries map to the terraform databricks_cluster.library field; regenerate the mapping so reference translation and the tf-only field audit stay correct. Co-authored-by: Isaac <no-reply@databricks.com>
- clusters/libraries: direct-engine deploy installs pypi + local wheel (rewritten to its uploaded path), and removing the wheel and redeploying uninstalls it. - clusters/libraries-terraform-error: bundle plan/deploy reject cluster libraries under the terraform engine. Co-authored-by: Isaac <no-reply@databricks.com>
Enumerate the remaining LibraryInstallStatus cases in the install-wait poll and set EmbeddedSlice explicitly in DoRead's state literal. Co-authored-by: Isaac <no-reply@databricks.com>
clusters/libraries-drift (Cloud = true, direct engine): deploy a cluster with a pypi library, then assert the immediate re-plan is a no-op (0 to change). Verified on a real AWS workspace: the Libraries status API round-trips the library without drift, so no normalization is needed. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Pipelines have a native libraries field that is a plain field, not a child resource. The sub-resource wiring matched resources.*.*.libraries for every resource type, so the direct engine tried to plan pipelines.libraries as a resource and failed with 'unsupported resource type: pipelines.libraries'. Scope both GetNodeAndType and the plan pattern to clusters. Co-authored-by: Isaac <no-reply@databricks.com>
Consistency follow-up to the GetNodeAndType/plan-pattern scoping: ResourcesTypes registered a .libraries key for every resource type with a Libraries field, spuriously adding pipelines.libraries and cluster_policies.libraries. Those keys are unreachable now that GetNodeAndType is scoped, but scope this branch too so the map stays consistent and the entries don't mislead future callers. Co-authored-by: Isaac <no-reply@databricks.com>
Two CI failures in the local acceptance suite: 1. bundle/resources/clusters/libraries deployed a prebuilt wheel from ./dist/*.whl, but dist/ is gitignored so the fixture was never committed. CI's clean checkout hit 'no files match pattern: ./dist/*.whl'. Force-add the dummy wheel as a committed test input. 2. TestInvariantConfigsCoverage requires every config.ResourcesTypes key to be covered. clusters.libraries had no coverage: the scanner only understood .permissions/.grants sub-resources. Teach it .libraries too, add a pypi-only cluster_libraries invariant config, and wire it into INPUT_CONFIG. Cluster libraries are direct-only, so exclude the config from the terraform-seeded migrate subtest like the other direct-only resources. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
force-pushed
the
sankalp-mittal/all-purpose-cluster-libraries
branch
from
August 27, 2026 08:44
6b15637 to
612add7
Compare
Installs are invisible to attached notebooks and uninstalls are deferred until restart, so restart the running cluster after a library change (WaitAfterUpdate/WaitAfterDelete). DoDelete now uninstalls recorded libraries. Co-authored-by: Isaac <no-reply@databricks.com>
Cover full-block removal (Delete path: uninstall + restart) and a stopped cluster (restart skipped when the cluster is not running). Co-authored-by: Isaac <no-reply@databricks.com>
Set all fields (RestartUser, ForceSendFields) explicitly, matching the dresources exhaustruct convention. Co-authored-by: Isaac <no-reply@databricks.com>
The broad //libraries filter degrades to a bare 'libraries' substring on Windows Git Bash, matching the bundle name (cluster-libraries-stopped) in file-upload paths and dumping unrelated requests. Use //libraries/install like the sibling tests so the filter stays specific across platforms. Co-authored-by: Isaac <no-reply@databricks.com>
Cluster libraries postdate v0.293.0, so the old CLI cannot deploy the config; exclude it like the other post-0.293 resources. Addresses PR review feedback. Co-authored-by: Isaac <no-reply@databricks.com>
splitResourcePath treated any 4th-component 'libraries' as a sub-resource, but only clusters expose libraries as one; pipelines have a native top-level libraries field, so a reference into a pipeline's libraries mis-split to a non-existent node. Scope it to clusters, matching makePlan. Addresses Isaac Review finding. Co-authored-by: Isaac <no-reply@databricks.com>
libraryKey ignored pypi/maven/cran repo and maven exclusions, so a change to only the repo was treated as no change and never applied. The backend hashes these into library_id_hash and round-trips them on cluster-status, so include them. Adds a Cloud=true test covering repo round-trip + repo-change detection. Co-authored-by: Isaac <no-reply@databricks.com>
The libraries sub-resource is emitted by both the clusters adapter's input walk (as an INPUT-only block) and the clusters.libraries sub-resource adapter (as an ALL block), duplicating resources.clusters.*.libraries in the refschema dump. permissions and grants avoid this via an explicit skip filter; extend it to libraries, scoped to clusters since pipelines have a native top-level libraries field. Co-authored-by: Isaac <no-reply@databricks.com>
The invariant no_drift suite now runs configs/cluster_libraries.yml.tmpl (added in 612add7), which deploys the identical pypi-requests cluster config and asserts no drift after deploy, plus the delete/destroy/migrate idempotency variants. That makes the standalone libraries-drift cloud test a strict subset. The restart-on-destroy output it captured is already asserted in the local libraries test, and libraries-repo remains as the stronger repo round-trip check. Co-authored-by: Isaac <no-reply@databricks.com>
The three pypi/wheel tests were authored fake-only (hardcoded i3.xlarge, no $UNIQUE_NAME, no instance pool, static databricks.yml), so they could not run against a real workspace. Convert them to cloud: - libraries: build a real wheel via setup.py + an artifacts block (mirroring integration_whl/interactive_cluster) instead of the 20-byte dummy stub, so the /Workspace wheel install is exercised on a real cluster. Cloud + CloudSlow; data_security_mode=USER_ISOLATION. Deploy output routed to LOG; asserts the install request (rewritten /Workspace wheel path), uninstall, and restart. - libraries-remove-all, libraries-stopped-cluster: parameterize databricks.yml as a .tmpl with cloud vars and instance pool; assert summary/restart lines instead of exact request bodies (which the recording proxy makes redundant). - All four tests: map the cloud cluster-id format to [UUID] so cloud output matches the fake run. All four remain Cloud=true and still run locally against the fake server. Co-authored-by: Isaac <no-reply@databricks.com>
The Aug 27 DABs core-eng meeting finalized that cluster libraries are managed as part of the cluster resource (Option 3), not as a `clusters.libraries` sub-resource. This reworks the feature accordingly: - Fold install/uninstall/restart into ResourceCluster: Libraries is now a field of ClusterState and ClusterRemote (DoRead populates it from the Libraries cluster-status API). WaitAfterCreate installs on the freshly-running cluster; DoUpdate reconciles (uninstall removed + install desired) and restarts a running cluster so the change takes effect (a cluster edit already restarts, so we only restart ourselves when no edit fired). KeyedSlices keys libraries by identity so reordering is not drift. - Delete cluster_libraries.go and remove all sub-resource wiring (all.go, resources_types.go, root.go, bundle_plan.go splitResourcePath + makePlan, refschema.go skip). Library changes now surface as `update clusters.<name>`, and destroy no longer wastefully restarts (PermanentDelete removes the cluster). - Rewrite the clusters/libraries* acceptance tests and regenerate refschema for the part-of-cluster shape. Co-authored-by: Isaac <no-reply@databricks.com>
A local whl/jar whose workspace path is unchanged but whose contents changed (same name+version, non-dev mode) does not trigger a restart. Note the gap and why hashing the built wheel is unsafe (non-reproducible zip mtimes); a general fix needs a source hash in state. Co-authored-by: Isaac <no-reply@databricks.com>
Add clusters/libraries-restart (Cloud=true, pypi-only) pinning restart behaviour across the library lifecycle on a single cluster: - install (create): libraries install on the fresh cluster with NO restart - update (add a library): the running cluster is restarted - delete (remove all libraries): the running cluster is restarted Fold in clusters/libraries-remove-all, whose remove-all -> restart case is subsumed by the delete step here, keeping the Cloud=true cluster count flat. Co-authored-by: Isaac <no-reply@databricks.com>
…rpose-cluster-libraries
…inue Regenerate the cluster-libraries acceptance snapshots to carry EnvMatrix.DMS = ["", "true"] after merging main's DMS test matrix, fixing the post-test git-diff drift check. Also remove the now-dead continue at the end of the Grants loop body in resources_types.go, orphaned when the Libraries block was removed in the part-of-cluster rewrite. Co-authored-by: Isaac <no-reply@databricks.com>
In the part-of-cluster model libraries is a field of ClusterState, not a sub-resource, so there is no clusters.libraries key in config.ResourcesTypes and no resource with top-level libraries missing in state. Drop the now-dead ".libraries" scanning in the invariant coverage test and the "libraries" entry in commonMissingInStateType. Co-authored-by: Isaac <no-reply@databricks.com>
The initial deploy step routed its output to LOG and asserted signals via contains.py, but had no title, so the phase was invisible in output.txt. Add a title so the golden documents it, matching the other cluster-libraries tests. Co-authored-by: Isaac <no-reply@databricks.com>
denik
reviewed
Sep 1, 2026
| ) | ||
|
|
||
| // librariesWaitTimeout bounds how long we poll for libraries to finish installing. | ||
| const librariesWaitTimeout = 15 * time.Minute |
Contributor
There was a problem hiding this comment.
Is this chosen to match terraform? What happens when time runs out?
Collaborator
Author
There was a problem hiding this comment.
this was just taken to be half of clusterWaitTimeout, I was not really sure what to keep it, if the timer runs out the deploy fails
| trap cleanup EXIT | ||
|
|
||
| # Deploy output is noisy and differs fake vs cloud; assert only the plan signals below. | ||
| $CLI bundle deploy &> LOG.deploy |
Contributor
There was a problem hiding this comment.
could you add READPLAN variant to these tests?
Collaborator
Author
There was a problem hiding this comment.
There is an issue where the READPLAN variant doesn't work #6490
Sankalp-Mittal
commented
Sep 1, 2026
Run each cluster-libraries test through a saved plan (bundle plan -o json, applied via readplanarg) in addition to the inline deploy, mirroring clusters/readplan-lifecycle. Addresses review feedback. Deploy and plan output route to LOG/tmp.plan.json, so output.txt is unchanged; only the READPLAN matrix is added. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Add support for installing libraries on all-purpose clusters. Libraries are declared as a field of the cluster resource, not a separate resource:
A cluster's libraries are not part of the cluster create/edit spec, so they are managed through the dedicated Libraries API (
install,uninstall,cluster-status) as part of the cluster lifecycle. Installed libraries are read back viacluster-status, so library changes participate in normal drift detection. Local.whlartifacts are built, uploaded, and rewritten to their workspace path before install.Install/uninstall cases:
Why
Resolves #1860.
Tests
Acceptance tests covering create / add / remove, local wheel build-and-upload, repo-hosted wheels, and the stopped-cluster (no restart) case. They run locally against the fake server and additionally against a real workspace (
Cloud = true).Also manually verified on a real Azure workspace: a job with a
python_wheel_taskand no task-level library successfully ran a wheel installed only via the cluster'slibraries:field, confirming a cluster-installed library is actually usable by workloads.