feat(jobs): retry, run history, notify, and manual runs with inputs for scheduled jobs

[vishr]

What this changes

Completes the scheduled-job contract without adding a resident process or a new store, as specified in the issue: bounded retry inside one timer firing, one run record per activation in the host journal, notifications per outcome, and declared inputs with an operator-initiated ob schedule run.

• Project schema: schedule.retry, schedule.notify, inputs, validated by the loader and constrained in the published JSON Schema.
• Runner: writes a state file in both lock modes, retries with capped doubling backoff, consumes a manual-run inputs file as -e arguments (never shell text), records a skip with its reason and exits 0 before any container starts, and honours the application lock only within its TTL.
• Notifier (ExecStopPost): finalises one JSON record per run with logger --journald, carrying ONEBOX_UNIT/ONEBOX_JOB fields, then sends webhooks for the outcomes the job selected, with the run id as deploy_id.
• CLI: ob schedule list, ob schedule history, ob schedule logs, ob schedule run --input NAME=VALUE --wait; ob status reads outcome, attempts, duration, consecutive failures and skips from the records.
• Host floor: systemd 252 for every scheduled job (TRIGGER_UNIT), checked in preflight and in schedule reconciliation.
• Docs: scheduling guide, capabilities page, generated CLI and field references, schema files.

Closes #155

Why this is correct

• The generated runner and notifier shell is executed by the unit tests, not only inspected: the notifier runs with stub logger and curl binaries for every outcome (success, failure, timeout, skip, container exit 75) and the inputs-consume block runs against a real file. Each behaviour was added test-first and the test was watched to fail.
• The server e2e suite on the Ubuntu 24.04 Lima guest passed on the final tree (635s): a recorded run, a two-attempt retry, a manual run with an input override, the journal for that run, the audit row, a recorded timeout, and a manual run clearing that timeout from ob status.
• The record is written with explicit journal fields rather than relying on journald's cgroup attribution, because on the real guest a systemd-cat record arrived with no _SYSTEMD_UNIT (the process exits before journald reads /proc), and journalctl -u never found it. The field query was probed on the guest before the e2e rerun.
• A local high-effort review of the branch produced nine confirmed findings; all are fixed in the last commit (operation-id matching for --wait, inputs-file cleanup on failure, skip semantics, systemd floor scope and placement, retry arithmetic parity, schema constraints, logs run id, and dropping systemd's Result as a verdict).
• just check passes on the final tree.

Effect on the safety envelope

Changes what Onebox does on a host, deliberately and within the documented boundary:

• Timer firings: a lock-conflict skip now exits 0 with a recorded reason instead of a failed unit with exit 75; three skips in a row are an ob status issue. A retry sleeps under the locks the run already holds, bounded by timeout at validation. Nothing else about unattended firings changes.
• New operator path: ob schedule run starts a unit the timer already runs unattended, only for data_effect: none jobs, with inputs limited to the declared enum or pattern. Jobs with migration or destructive effects keep the sealed-plan gates of ob job run.
• New host requirement: systemd 252 or newer for any scheduled job; ob preflight and deploy refuse older hosts before staging.
• /status/capabilities gains one shipped bullet covering retry, run history, notify, inputs and manual runs. Nothing moved from "schema accepts it" to "binary does it" without landing in the binary.

Checklist

• just check passes locally.
• Tests cover the new behaviour, including the failure paths.
• Generated documentation is current (just check verifies this).
• I have accepted the CLA, or will when the bot asks on my first pull request.

https://claude.ai/code/session_013MqwrF5NA179khDtEQdib5

[Copilot]

🟡 Changes recommended

Confirmed issues in the new scheduling implementation can cause incorrect skip behavior under sub-minute lock TTLs and can silently hide journald read failures, which should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Implements the “scheduled-job contract” end-to-end by extending the project schema, generated systemd runner/notifier scripts, and CLI to support bounded retry, per-run journald history, per-outcome notifications, and operator-initiated manual runs with declared/validated inputs—without adding a resident host process or new store.

Changes:

• Adds schedule.retry, schedule.notify, and inputs to the spec (validation + published JSON Schema) and renders input defaults into Compose env.
• Updates generated scheduled-job runner/notifier to record one structured run record per activation into journald, support in-firing retry with capped exponential backoff, and consume manual-run inputs safely.
• Adds CLI commands (ob schedule list|history|logs|run) and updates ob status/audit/e2e/tests to surface outcomes, attempts, durations, skip streaks, and run correlation.

File summaries

File	Description
site/src/content/docs/status/capabilities.mdx	Documents new scheduled-job capabilities at a high level.
site/src/content/docs/reference/policies.mdx	Updates output-policy classification for new schedule commands.
site/src/content/docs/reference/fields/workloads.mdx	Adds reference docs for inputs, schedule.notify, schedule.retry.
site/src/content/docs/reference/errors.mdx	Documents new schedule-related error codes.
site/src/content/docs/reference/cli.mdx	Adds CLI reference sections for schedule subcommands.
site/src/content/docs/guides/schedule-a-job.mdx	Expands guide with retry, journald history, notifications, and manual runs with inputs.
site/public/onebox.run-v1.schema.json	Publishes updated JSON Schema for new schedule/input fields.
docs/onebox.run-v1.schema.json	Publishes updated JSON Schema for new schedule/input fields (docs copy).
internal/onebox/operation_types.go	Adds schedule_run operation kind.
internal/onebox/operation_errors.go	Registers schedule read/flag validation failure definitions.
internal/onebox/execution_types.go	Extends ExecuteRequest/OperationResult for schedule run requests/results.
internal/onebox/execute.go	Routes schedule_run to engine execution.
internal/onebox/binding.go	Includes schedule run in inspection-runtime selection.
internal/engine/status.go	Extends ob status schedule line output (attempts, next run, journal volatility, last outcome).
internal/engine/status_snapshot_test.go	Updates snapshot expectations for schedule status changes.
internal/engine/schedule.go	Major runner/notifier generation changes: inputs consume, skip semantics, journald records, retry loop, systemd floor.
internal/engine/schedule_test.go	Adds/updates unit-script/notifier contract tests and behavior tests.
internal/engine/schedule_status.go	Changes schedule status model to be record-driven (outcomes, streaks, next run, attempts).
internal/engine/schedule_run.go	Implements operator-initiated schedule run with inputs file + journaling + optional wait for record.
internal/engine/schedule_run_test.go	Adds tests for schedule run behavior, refusal paths, and --wait record matching.
internal/engine/schedule_history.go	Adds history/list/logs primitives backed by journald run records.
internal/engine/schedule_history_test.go	Adds tests for history parsing/list/logs behavior.
internal/engine/preflight.go	Enforces scheduled-job host requirements during preflight.
internal/engine/deploy_test.go	Updates fake host to satisfy new systemd floor check.
internal/engine/audit.go	Adds audit labeling for schedule run operations.
internal/app/validate.go	Wires inputs + retry validation into workload validation.
internal/app/types.go	Adds typed model fields for inputs/retry/notify.
internal/app/schedule.go	Resolves scheduled jobs including retry/notify defaults and inputs.
internal/app/schedule_test.go	Adds tests for retry/notify resolution, bounds, inputs constraints, and compose env rendering.
internal/app/schedule_retry.go	Implements retry policy defaults, worst-case backoff computation, and validation.
internal/app/schedule_inputs.go	Implements input constraints + workstation-side value validation.
internal/app/names.go	Adds path helper for per-job .inputs file.
internal/app/jsonschema.go	Adds JSON Schema constraints and marks inputs as job-only.
internal/app/generate.go	Renders inputs defaults into Compose environment.
internal/app/constraints.go	Adds enum values for schedule notify outcomes.
e2e/testdata/postgres/ob.yml.tmpl	Adds e2e fixtures for retry/input/manual run behaviors.
e2e/server_test.go	Extends e2e coverage for run history, retries, manual runs with inputs, and record-driven status clearing.
cmd/ob/schedule.go	Introduces `ob schedule list
cmd/ob/schedule_test.go	Adds tests for --input NAME=VALUE flag parsing.
cmd/ob/output.go	Extends CLI output matrix for new schedule commands.
cmd/ob/output_test.go	Updates output matrix test expectations.
cmd/ob/ops.go	Hooks schedule commands into ops command wiring.

Review details

• Files reviewed: 42/43 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[vishr]

Fixed in b4762c8. The runner now reads the lock's age with lockAgeCmd, the same shell AcquireLock uses, and compares it against the TTL in whole seconds rather than rounding up to a minute with find -mmin. That also inherits its fail-closed behaviour: a dangling symlink or an unreadable parent directory reads as age 0, so the runner still defers to a lock it cannot inspect.

[vishr]

Fixed in b4762c8. The command now tests for the file first and exits 73 for that case, so an existing pending run and a host that refuses the write are reported separately, the latter with the shell's own stderr. Covered by TestScheduleRunTellsAPendingFileFromAWriteFailure, which also asserts neither path starts the unit.

[Copilot]

🟡 Changes recommended

A couple of verified issues in the new journald record parsing and skipped-notification messaging can lead to incorrect status/reporting or confusing operator output.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 43/44 changed files
• Comments generated: 2
• Review effort level: Lite

[vishr]

Fixed in 5bf92a1. parseScheduleRunRecords now takes the job name and keeps a decoded line only when it names that job, carries a 32-hex systemd invocation id, and reports an outcome from the closed set (success, failure, timeout, skipped). Anything else is dropped, so a stray JSON line cannot become the newest record and clear a failure or a streak. Covered by TestParseScheduleRunRecordsRejectsLinesThatAreNotThisJobsRuns.

[vishr]

Fixed in 5bf92a1. The skip error is now just the directive, so the line reads ⏭️ app: scheduled job X did not run on host — inspect trusted host diagnostics instead of repeating itself. The reason itself stays on the host, in the run record's reason field and in ob status, which is where it can be specific without crossing the trust boundary.

[Copilot]

🟡 Changes recommended

A job-lock conflict “skip” can currently be mis-recorded (and can clobber the in-flight run’s state), so the journald history/status semantics are not reliable under overlap.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

internal/engine/schedule_history.go:141

• ScheduleList suppresses systemctl show failures with 2>/dev/null || true, which forces an overall exit code of 0 and prevents the later res.ExitCode != 0 check from ever surfacing a real read failure. This contradicts the intent in the comment below (distinguishing “empty state” vs “host would not tell us”) and will make ob schedule list silently print dashes when systemctl is unavailable/permission-denied/unit missing.

• Files reviewed: 44/45 changed files
• Comments generated: 1
• Review effort level: Lite

[vishr]

Both correct, and both fixed in d8d94e9.

The stand-aside path now leaves a note keyed to its own invocation, <state>.skip.$INVOCATION_ID, carrying the reason and the manual run's operation and inputs. The notifier prefers that note over the state file: it records the skip and never reads or removes $state, so the run still holding the job lock keeps its outcome and its pinned-release evidence. That closes both halves you named, the false success and the clobbering, without per-invocation state files for the ordinary path. TestScheduledJobNotifierReadsAStandAsideNoteAndSparesTheRunningState executes the generated notifier against a laid-out directory and asserts the live state file is byte-identical afterwards.

On the suppressed note about ScheduleList: also right, and it was mine from the same commit that added the exit check. The 2>/dev/null || true forced the batch to exit 0, so the check could never fire. The probe no longer swallows its failure, and TestScheduleListSurfacesAFailedTimerRead covers it.

[Copilot]

🔵 Needs a closer look

The published JSON Schemas define schedule.notify as an array but currently set its default as a string, which should be corrected before merge.

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

docs/onebox.run-v1.schema.json:2602

• schedule.notify is an array, but the JSON Schema default is currently a string ("failure, timeout"), which is not a valid default for this type and can confuse tooling.
  internal/app/types.go:287
• JobSchedule.Notify is a []string, but its struct tag sets default:"failure, timeout" (a single string). This likely drives the generated JSON Schema schedule.notify.default to be a string too, making the schema invalid for an array type (as seen in the committed schema JSON).
  site/public/onebox.run-v1.schema.json:2602
• schedule.notify is typed as an array, but its JSON Schema default is currently a single string ("failure, timeout"). This makes the schema internally inconsistent and can break editors/validators that apply defaults.

• Files reviewed: 44/45 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

There is at least one confirmed build-breaking bug (for attempt := range 10 in awaitScheduleRecord) and a small robustness mismatch in skip-marker naming between runner and notifier that should be corrected.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

internal/engine/schedule.go:303

• The comment above requireScheduleHost says the systemd 252 floor “applies to every scheduled job”, but the implementation (and tests) only enforce it when any job declares inputs. This mismatch makes it harder to reason about host requirements and could confuse future edits.

internal/engine/schedule.go:510

• The stand-aside marker file uses ${INVOCATION_ID:-$$} in the runner, but the notifier looks for ${INVOCATION_ID:-}. If INVOCATION_ID is ever unset (e.g., script invoked outside systemd), the notifier won't find the marker and may fall back to reading/removing the shared $state file, risking clobbering the in-flight run's evidence.

		// never held the job lock, so the state file belongs to whichever run
		// is still going: read the note and leave that file alone.
		"skip_marker=\"$state.skip.${INVOCATION_ID:-}\"",
		"if [ -f \"$skip_marker\" ]; then",
		"  while IFS= read -r line || [ -n \"$line\" ]; do",

• Files reviewed: 46/47 changed files
• Comments generated: 1
• Review effort level: Lite

[vishr]

Thanks — two of the three are real and are fixed; the headline one is not.

for attempt := range 10 is not build-breaking. Ranging over an integer has been legal since Go 1.22; this module targets go 1.27.0, go build and go vet pass, and CI's Check job is green on the same commit. It is also not a stylistic choice: .golangci.yml enables intrange, and that linter is what required this form over the three-clause loop.

The skip-marker mismatch is real. The runner wrote $state.skip.${INVOCATION_ID:-$$} while the notifier looked for $state.skip.${INVOCATION_ID:-}. Under systemd INVOCATION_ID is always set so the two agreed in practice, but anywhere else they did not, and a note the notifier cannot see sends it back to the state file belonging to the run in flight, which is exactly the clobbering the note exists to prevent. Both sides now use one expression, and TestScheduleSkipMarkerIsNamedIdenticallyOnBothSides extracts the assignment from each generated script and fails if they differ.

The suppressed note on requireScheduleHost is right too. The doc comment still said the floor applies to every scheduled job. That was true for one commit and stopped being true when the floor was scoped back to jobs declaring inputs, so it also contradicted the comment inside the function. Corrected.

Both fixed in the commit above.