Add job_runs lifecycle.triggers.on_value_change - #6341
Open
radakam wants to merge 115 commits into
Open
Conversation
Re-fire a run when matched file contents change, using stable hash fingerprints so mtime-only updates (touch) do not recreate.
Satisfy the linter on PrepareState literals after adding OnFileChange, and refresh out.fields.txt so validate-generated stays green.
Reject patterns outside the sync root and directory-only matches so the trigger cannot fingerprint escaped paths or silently disarm.
Drop size/mtime fingerprints and the resolve fast path so content identity alone drives recreate, and touch no longer needs a planner exception.
The modernize linter rejects the manual m[k]=v copy loop, failing CI lint.
Honor .gitignore and sync.exclude when hashing matches. Cover mysubdir globs, READPLAN, and keep LF on Windows so restored files hash cleanly.
The glob test edits migrations/ignored.txt, but the test .gitignore excluded it from the commit so CI clones would miss the file.
Plan serialization is already covered by on_file_change; the glob cases only assert the hash map.
Cover deploying glob hashes from a saved plan, not only from a live re-plan.
Keep the gitignore fixture as dot_gitignore so it does not affect the repo while developing, drop unit tests that duplicate acceptance coverage, and allocate trigger state once when a trigger is armed.
The previous simplification omitted OnBundleDeploy from the struct literal, which the linter rejects.
Always persist the same lifecycle.triggers shape so trigger fingerprints compare at the same paths. Wrap on_file_change hashes in files so dropping the trigger updates instead of recreating.
Nest lifecycle by value so structdiff descends to the leaf that changed instead of reporting the whole subtree, and let RemapState supply the empty shape. GetRun never returns the fingerprints, so the remote type no longer advertises a lifecycle it cannot fill.
radakam
marked this pull request as ready for review
August 21, 2026 15:59
Contributor
Approval status: pending
|
Collaborator
Integration test reportCommit: 971126e
Top 3 slowest tests (at least 2 minutes):
|
Drop the files wrapper so plan diffs land on lifecycle.triggers.on_file_change, matching the config path without an extra state-only level.
The trigger, PrepareState and OverrideChangeDesc cases are asserted end to end by on_bundle_deploy and on_file_change, so the unit copies only duplicated goldens. Keep the ones acceptance cannot reach without stubbing GetRun.
Flattening on_file_change to a map made an unarmed trigger drop out of the serialized state, so the three plans that print an unarmed job_run now report an empty triggers object.
The unit test staged an asynchronous cancellation by hand; the acceptance test shows the real request order a user's destroy produces. Also drops two unit tests that the on_file_change acceptance tests already cover.
Re-fire a run when a resolved expression changes, including resource refs that become deploy-graph edges. Removing the trigger is a state-only update.
Include the on_value_change input and state paths required by generated-file validation.
radakam
force-pushed
the
deco-job-runs-lifecycle-on-value-change
branch
from
August 24, 2026 09:36
0db1f4b to
e431088
Compare
Re-fire a run when matched file contents change, using stable hash fingerprints so mtime-only updates (touch) do not recreate.
Satisfy the linter on PrepareState literals after adding OnFileChange, and refresh out.fields.txt so validate-generated stays green.
Reject patterns outside the sync root and directory-only matches so the trigger cannot fingerprint escaped paths or silently disarm.
Drop size/mtime fingerprints and the resolve fast path so content identity alone drives recreate, and touch no longer needs a planner exception.
Directory-only and sync-excluded matches silently disarmed the trigger. Treat them as errors, warn that ** is not recursive, and cover both in acc.
The goldens were captured against a one-line-taller databricks.yml, so CI failed after the source settled on line 24.
Clearing a trigger is now a skip, so the plan reports the run as unchanged and the deploy makes no API call.
Patterns must still land under the sync root. Reject **, warn when a glob matches directories, and skip only when the trigger itself is cleared.
filepath.Join rebases an absolute glob under the bundle root, so reject those up front. Key missing-file placeholders to the sync root, and skip a cleared trigger from path.String() instead of prefix matching.
Every "nothing to hash" case re-arms once a matching file appears, now that the placeholder is always recorded, so none of them justify failing the deploy. Directories matched alongside real files are skipped silently: a glob like migrations/* routinely matches a subdirectory, and warning about it on every command was noise the user cannot act on. Also drop the unreachable "lifecycle.triggers" branch in OverrideChangeDesc (structdiff descends into the value struct and never reports a change at that path) and document the ** and sync-skipping rules in the schema.
Windows CI converted /tmp/seed.txt to %TEMP%/seed.txt, so the golden expecting the POSIX path failed. MSYS_NO_PATHCONV keeps the argument intact.
MSYS_NO_PATHCONV also stops Git Bash converting the path of a shebang helper, so update_file.py could not be found on Windows. Keep the leading-slash pattern out of argv entirely instead.
radakam
force-pushed
the
deco-job-runs-lifecycle-on-file-change
branch
from
August 25, 2026 10:12
d65e640 to
b0437a3
Compare
…s-lifecycle-on-value-change The base was rebased onto a newer main and softened the on_file_change "nothing to hash" error into a warning. Adapt on_value_change: follow the base in dropping the unreachable "lifecycle.triggers" case from OverrideChangeDesc, keeping on_value_change alongside on_file_change on the leaf paths, and take the zero-value emptyJobRunLifecycleState now that it no longer lists fields by hand. Also restore the initialize.go ApplySeqContext formatting that the previous merge reverted.
preventDestroyError took one bool per trigger, so every new trigger widened the signature and risked a swapped argument. Have JobRun report the armed names instead.
extractReferences validated a config path against the state type with the Terraform rule that treats [0] on a struct as a no-op. That let lifecycle.triggers[0].on_value_change match the flattened state map and register a reference that cannot resolve. ValidatePathStrict drops the tolerance for these native DABs paths.
State kept each watched value verbatim and keyed it by the expression, so a long or sensitive resolved value ended up in the plan and in state. Key the entries by the value itself and hash it once it grows past the digest, normalizing through structvar as soon as the last reference resolves. With the expression gone from the key, dropping an entry can only mean the config stopped watching that value, so it skips instead of re-firing the run. DropJobRunValueChangeConfigRefs goes away with the strict path validation.
Keying state by value treated two watches converging as a removal, so the run did not re-fire. A list in trigger order distinguishes a changed fingerprint from a dropped watch.
Keep each watch's identity stable so removing one watch cannot hide a value change in another.
radakam
marked this pull request as draft
August 26, 2026 13:27
… duplicate plan entry Iterate the sync file list with path.Match instead of filepath.Glob so on_file_change honors .gitignore and sync.exclude directly, and drop the whole-map diff when per-file entries already report the change.
Snapshot on_value_change expressions before interpolation so a watch keeps its identity by expression, always fingerprint resolved values, and skip a removed trigger declaratively via skip_when_removed instead of ad-hoc OverrideChangeDesc logic.
Keep resolved hashes off user config, omit empty lifecycle from state, and fold include/sync-root path coverage into one acceptance test.
Match the INPUT type so refschema reports one INPUT STATE row instead of splitting pointer and value.
update_file.py uses text mode, so Windows rewrites committed LF files as CRLF and later plans still see those files as changed.
Group resolved hashes as pattern -> file -> hash so the planner can tell a removed trigger from a file dropping out of a glob, replacing the flat map and the empty-hash placeholder. PrepareState now copies the resolved trigger state instead of rebuilding it.
Store one deterministic fingerprint per pattern so plan paths remain concise while file additions, removals, and content changes still re-fire the run.
…triggers Resolve on_file_change fingerprints after the build/predeploy scripts have run and hash matched files through the SyncRoot VFS, so build outputs and WSFS notebooks are captured. Persist a cleared trigger's state without invoking the Jobs API, so removing then re-adding a trigger re-fires the run.
…s-lifecycle-on-value-change Adapt on_value_change fingerprints to the shared trigger state and preserve removals across saved plans.
Exercise trigger behavior through the full bundle pipeline while retaining focused unit coverage for internal invariants.
radakam
marked this pull request as ready for review
August 27, 2026 17:19
denik
force-pushed
the
deco-job-runs-lifecycle-on-file-change
branch
2 times, most recently
from
September 3, 2026 11:43
725e759 to
5e8b3a9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds
lifecycle.triggers.on_value_changeforresources.job_runs(direct engine).${resources.*}for the planner so the deploy graph still depends on them.lifecycle.triggers_state.on_value_change.prevent_destroy.Why
on_bundle_deployandon_file_changealready re-fire a run for deploy and file events.on_value_changecovers the remaining case: re-run when a watched config value changes (a variable, a workspace path, or another resource's id) without treating trigger removal as a config change that would replace a still-valid run.Tests
Acceptance:
acceptance/bundle/resources/job_runs/on_value_change— first deploy runs; hashes are stored (trimmed var, numeric var, workspace path, long string); unchanged redeploy is a no-op; changing one value re-fires; removing a watch via a saved plan does not run; re-adding it does; changing a remaining watch to a previously removed value re-fires; clearing the rest is unchanged;on_file_changestill re-fires after all value watches are gone.acceptance/bundle/resources/job_runs/on_value_change_ref— watches a pure${resources.jobs.*.id}and a mixedid-var-idexpression; the plan depends on both jobs; first deploy (and saved-plan deploy) runs; hashes stay stable on redeploy; deleting a referenced job and recreating it re-fires.acceptance/bundle/resources/job_runs/on_value_change_errors— validate rejects an empty expression, duplicates that only differ by whitespace,on_file_changepluson_value_changeon one entry,prevent_destroy, and an undefined variable.