branch-4.1: [feature](lance) Add RESOLVE LANCE INDEX JOB (FORCE_RELEASE) and job retention GC by u70b3 · Pull Request #67754 · apache/doris

u70b3 · 2026-09-10T02:27:34Z

Target: branch-4.1. Builds on #67201, #67235 and #67630 (all merged). Implements slice 3E of design v5.1 (design comment; slice map and status: roadmap comment, issue #66497): RESOLVE LANCE INDEX JOB <id> AS FORCE_RELEASE COMMENT '<note>' — the operator escape hatch that durably releases an UNKNOWN job — plus the retention GC that removes resolved job records. There is no dispatcher yet (#67978, open): no real UNKNOWN job can be produced so far, so FORCE has unit-test coverage only and the regression suite is deliberately negative/static (see Tests).

What lands

RESOLVE SQL (design 2.3/7.1): grammar (RESOLVE and FORCE_RELEASE are new non-reserved keywords, COMMENT is mandatory), ForwardWithSync to master. The release flow:
1. Load the job — a missing job and an unauthorized job share the same fixed 5103 (non-disclosing).
2. Resolve the persisted target three ways and authorize: names resolve and the catalog's current durable dataset locator still matches the job's (the same revalidation SHOW LANCE INDEX JOBS applies, so a repointed dataset reusing the same names never turns a stale name into table-level authorization) → table-level ALTER; verifiably absent names, or a locator that positively points elsewhere → the global-ADMIN orphan branch; a resolution that errors out, or a locator that cannot be resolved right now → global ADMIN, and the typed 5105 after the checks below, never an orphan verdict.
3. Idempotent short-circuit: an already released job returns OK + the late-commit warning, even with the pre-release revision and even when the target can no longer be resolved — a retry during a provider outage still observes the existing release record.
4. UNKNOWN-only state gate (5104).
5. Resolution-failure rejection (5105) — deliberately after steps 3-4, so an already released or terminal job never sees a 5105 that misstates its state.
6. Note validation (non-empty, ≤1024 UTF-8 bytes, 5100).
7. For a live catalog: one authoritative latest-metadata read and one external-table refresh with current credentials, both outside every lock, then the durable release inside the admission critical section (captureLanceIndexTarget → read/refresh → withLanceIndexAdmission recheck), serialized against DROP CATALOG / identity ALTER exactly like admission.
The durable release is one ordinary job upsert with the five FORCE audit fields set; the fence, the unresolved quota charge and the possible-live slot flip off through the existing field semantics in applyToMemory — no new teardown code.
Failure semantics (7.1 step 4): any target-resolution, read or refresh failure is the typed 5105 (ERR_LANCE_INDEX_JOB_RESOLUTION_INCOMPLETE); nothing is journaled, the job stays UNKNOWN holding fence/quota/slot, and the operator fixes the cause (including the 4.3 credential rotation case) and retries the same statement. The cause is logged server-side (fe.log) at the point of failure; failure details returned to the client pass through the catalog's sanitized root-cause chain only — no locator, credential or dataset URI is echoed. Like the 5104 state rejection, the 5105 is only visible to an already authorized caller.
Retention GC: resolved records (force-released UNKNOWN and terminal COMMITTED/NOT_COMMITTED alike) are audit-only; a master-only LanceIndexJobCleaner daemon (interval lance_index_job_clean_interval_second, default 1h) removes records resolved longer than lance_index_job_keep_max_second (default 7d, aligned with history_job_keep_max_second) through one batch edit-log record (OP_LANCE_INDEX_JOB_REMOVE = 501) per round, so every FE serves the same SHOW LANCE INDEX JOBS view. Unresolved records are never removed regardless of age (fail-closed, including corrupt identity-less records); the retention clock is the durable update time, which the force release bumps onto the force time.
Config: the two retention items are mutable, masterOnly, positive-validated (same handler as the existing quotas).

Two design readings that need reviewer confirmation

RESOLVE is not behind enable_lance_index_mutation. The gate controls mutation admission; FORCE is the escape hatch and must stay usable exactly when the gate is off — otherwise PENDING/UNKNOWN jobs could never be released once the gate is closed, and the catalog DDL guard would freeze that catalog forever. Nothing in 2.3 or 9.7 scopes the mutation gate to RESOLVE.
Full-orphan jobs skip the authoritative read and the refresh. Section 7.1 step 1 explicitly opens the global-ADMIN branch for a target that no longer resolves, but steps 3-4 ("with current credentials") cannot execute when the catalog itself is gone — there are no credentials and nothing to invalidate. Requiring them mechanically would make an orphan job unreleasable, contradicting step 1. So: catalog gone → release directly after ADMIN authorization; catalog alive but db/table unresolvable, or the current dataset locator positively pointing elsewhere (half-orphan) → skip the read, refresh with ignoreIfNotExists=true as best-effort invalidation. A non-null exception during target resolution is never treated as an orphan verdict — it fails with 5105 so the fence is kept when "table gone" cannot be told apart from "network down". An unresolvable locator is treated the same way: absence of evidence is not an orphan verdict. SHOW fails that uncertainty closed by hiding the row behind the ADMIN rule; RESOLVE fails it closed by not releasing. (Slice 3C's guard makes orphans unreachable on the normal path; they can only come from pre-guard journals/images.)

Operational notes

A FORCE retry is a success, not an error: the idempotent short-circuit deliberately runs before the expected-revision check and before the resolution-failure rejection, so a retry carrying the pre-release revision — or one issued during a provider outage — still observes the existing release record (unlike the stale-callback convention of the dispatch paths). The manager-side short-circuit honors only a released record in a null/UNKNOWN state; a force-released flag on any other state can only come from a corrupt journal/image and falls through to the revision CAS and the state gate (fail-closed).
The release does not clear possibleLiveOwned/terminationProof; the derived slot is released by forceReleased=true and the raw values stay for audit.
Success returns an OK packet with one warning row carrying the late-commit warning, persisted verbatim as the job's forceWarning: the old worker may still overwrite, remove, or reintroduce the index name; the mutation outcome remains UNKNOWN. (SHOW WARNINGS is a stub in the new planner; the OK packet is the only working warning channel.)
Audit needs no extra code: the coordinator FE's standard audit log carries the full SQL (note included), the durable force* fields replay everywhere, and SHOW LANCE INDEX JOB(S) already renders the force columns.
The GC daemon is registered in startMasterOnlyDaemonThreads (MasterDaemon itself does no master check); both retention configs are re-read every round, so ADMIN SET takes effect without restart.

Tests

FE UT (JUnit5, no FE startup): manager FORCE transition (success with atomic fence/quota/slot release and exactly one journal record, idempotent retry with stale revision, four non-UNKNOWN states rejected without a journal, revision CAS, bounded-text rejection, same-name re-admission, corrupt identity-less unblock, corrupt force-released flag on a non-UNKNOWN state does not short-circuit, refresh-queue filter); retention GC (both resolved families removed, unresolved never removed at any age, one batch record per round, per-round cap, replay convergence/idempotence, image shrink, 501 JournalEntity round-trip, update-time clock base); cleaner daemon (seconds→millis config conversion, per-round cap, round failure swallowed); opcode wiring (501 uniquely assigned); config validator (defaults, wiring, rejection, setMutableConfig end-to-end); RESOLVE parser (full forms, resolve/force_release stay usable as identifiers/aliases, rejection family); RESOLVE command (5103 byte-identical for missing vs unauthorized, ALTER vs ADMIN authorization, proxy identity, note validation, 5104 gate, idempotent replay — including under a provider outage after a successful release, a terminal job sees 5104 rather than a state-misstating 5105, full/half-orphan paths, resolution failures and an unresolvable locator at authorization time keep everything with zero manager calls, a positively repointed locator takes the ADMIN half-orphan branch, read/refresh failures keep everything with zero manager calls, non-external relation guard, REST defensive 5101, FORWARD_WITH_SYNC). Full FE suite (mvn clean test -pl fe-common,fe-core -am, checkstyle bound into the build): fe-common all green and fe-core 9602 tests with 19F+2E in 14 classes (date/timezone literals, huggingface network unreachable, native-lib link, disk-rebalance scheduling) that reproduce on the pre-change base — environmental, unrelated to this PR (DiskReblanceWhenSchedulerIdle re-verified directly on the base commit; two module-aborting environmental classes, the timezone-bound PropertySchemaTest and the oidc-dependent AuthenticationPluginManagerTest, are excluded from the run and fail identically on the base tree). The 598 tests across the 43 Lance-matching classes (fe-common + fe-core, this slice's nine among them) are all green; the RESOLVE command class alone runs 26.
Regression (external_table_p0/lance/test_lance_index_resolve.groovy, nonConcurrent, per-run suffix, master-config capture/restore): syntax rejection family, missing job 5103, unauthorized 5103 under a second identity, PENDING job rejected with 5104, retention config SHOW/SET smoke (defaults, zero/negative rejected), GC never touches an unresolved job. Needs the external MinIO/REST docker env — runs in pipeline, not locally.
Not covered here (needs the dispatcher slice, branch-4.1: [feature](lance) Add Lance index job dispatcher with thrift dispatch boundary #67978): FORCE happy-path e2e, same-name reuse e2e, quota reclaim e2e, expiry-GC e2e — there is no way to produce a real UNKNOWN job in this stack.

Deliberately not done

No dispatch/worker (branch-4.1: [feature](lance) Add Lance index job dispatcher with thrift dispatch boundary #67978); no gate flip (9.7 evidence gates).
FORCE does not touch the refresh state machine (markRefresh*): UNKNOWN owes no refresh (6.3), and the refresh of the release protocol is the command's own synchronous action. The only adjacent change is that getJobsNeedingRefresh now excludes force-released records — they owe no refresh, and picking them up would only add audit noise to the next slice's driver.
No "persisted locator + storage options" bypass read for half-orphans (optional enhancement; the late-commit warning already covers the risk).
No count-based retention cap (the per-round removal batch is fixed at 1024; the time bound plus the global unresolved quota already bounds the table).
RESOLVE takes one job id per statement (the 2.3 form); no WHERE/batch form.
SHOW output unchanged (the force columns already render since slice 3C).

Release note

Add the RESOLVE LANCE INDEX JOB <id> AS FORCE_RELEASE COMMENT '<note>' statement that lets an operator durably release a Lance index job whose mutation outcome is UNKNOWN, plus a master-only retention cleaner that garbage-collects resolved Lance index job records after lance_index_job_keep_max_second (default 7 days).

hello-stephen · 2026-09-10T02:27:39Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

u70b3 · 2026-09-16T01:47:26Z

run buildall

hello-stephen · 2026-09-16T03:41:54Z

FE UT Coverage Report

Increment line coverage 80.39% (861/1071) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-16T06:39:55Z

FE Regression Coverage Report

Increment line coverage 42.87% (481/1122) 🎉
Increment coverage report
Complete coverage report

u70b3 · 2026-09-16T08:54:39Z

run p0

u70b3 · 2026-09-16T11:39:54Z

run check_coverage_fe

u70b3 · 2026-09-16T12:15:04Z

run p0

hello-stephen · 2026-09-16T14:43:11Z

FE Regression Coverage Report

Increment line coverage 41.54% (481/1158) 🎉
Increment coverage report
Complete coverage report

…n SQL and catalog DDL guard (#67630) ## What problem does this PR solve? Issue: #66497 (design v5.1, slice 3). Target: `branch-4.1`. This is PR3C, building on #67201 (PR3A, DDL surface) and #67235 (PR3B, durable jobs), both already merged into branch-4.1. It adds the FE admission path for top-level Lance CREATE, CREATE OR REPLACE, and DROP INDEX, returns durable job IDs, and exposes job inspection SQL. **`enable_lance_index_mutation` remains false by default.** This slice does not execute index mutations: admitted jobs remain PENDING until dispatch/worker support and FORCE_RELEASE arrive in later slices. Enabling it now leaves unresolved jobs that prevent target-changing ALTER CATALOG and DROP CATALOG. ## Behavior - Admission reads dataset version, schema fields, logical indexes, and physical index families from one opened latest Dataset. It does not call `countRows()` or `getIndexStatistics()`. Schema contract v1 preserves Arrow field identity, nullability, vector dimensions/element types, and relevant type parameters for later worker revalidation. For fixed-size lists the pinned SDK reconstructs no child field — the element type lives only in the manifest logical type and element nullability is synthesized as nullable (a manifest-format fact that also binds existing datasets) — so the contract sources element facts from the reconstructed Arrow view and is pinned by a real on-disk dataset fixture. - Reserved names and ambiguous case-only collisions are rejected (reserved names fail before any metadata read). REPLACE and DROP persist the stored display name of a unique case-insensitive match. Column identities that are ambiguous under the table's case-insensitive lookup relation are rejected before anything is journaled. CREATE IF NOT EXISTS is a no-op only when algorithm, physical family, column (compared in the same path-segment representation on both sides), and exposed whitelist properties match — an omitted `num_bits` compares as the persisted 8; DROP IF EXISTS is a no-op when the authoritative name is absent. - Metric comparison ignores case. Numeric properties use strict parsed-long equality for numeric primitives or integer strings; fractions, exponent notation, overflow, and malformed values fail closed. Unexposed properties are skipped, and `num_partitions` is not compared. - An admitted statement returns one `JobId` row after the job and fence are durable; an IF no-op returns the same column with zero rows. The response works for direct and forwarded connections. - `SHOW LANCE INDEX JOBS [FROM [catalog.]db] [WHERE TableName = "t" [AND State = "PENDING"]]` and `SHOW LANCE INDEX JOB <id>` authorize each persisted target. Orphans — including targets whose provider lookup fails (credential expiry, outage) and targets whose currently resolved dataset locator no longer matches the job's persisted locator after a legal identity change — require global ADMIN; unauthorized rows are omitted, and unauthorized/missing job IDs share error 5103, so provider failures never leak through the authorization boundary. Locators, credentials, properties JSON, and schema-contract contents are not exposed. WHERE values accept what the parser actually produces for string literals, including `VarcharLiteral`; the `FROM` database name is canonicalized through the selected catalog's resolution semantics. ## Concurrency and configuration Catalog identity is captured under the CatalogMgr lock before the remote snapshot read. After the read, admission rechecks catalog existence, identity properties, and a local identity-change generation under the same lock used by catalog DDL, then allocates the ID and creates the job. This closes both the guard-check/create race and A → B → A identity changes. IF no-ops also revalidate the target. Identity changes invalidate in-flight reads even when a tentative ALTER is rolled back; same-value rewrites with unchanged key spelling and credential-only changes remain allowed. Metadata I/O runs outside the DDL lock. Final admission uses the lock order CatalogMgr → job manager → journal. DROP and changes to target-identity properties — `lance.catalog.type`, `warehouse`, namespace parent/delimiter/root_database, and the storage-routing endpoint/region keys the Lance property chain actually consumes — reject unresolved jobs; access keys and tokens stay rotatable. Renames and replay are not blocked by the unresolved-job guard. The gate and unresolved-job quotas (table/catalog/global defaults 8/64/256) are mutable and masterOnly. ADMIN SET changes runtime values on the master; it does not persist them to the custom configuration file or automatically synchronize other FEs. Restart loads configured file values/defaults, while promotion uses the promoted FE's own configuration. The two static bounds (`num_partitions` 4096, `num_sub_vectors` 256) are masterOnly — validation runs on the master — so a single-FE ADMIN SET takes effect only there; use ADMIN SET ALL FRONTENDS CONFIG to update the cluster. ADMIN SET callbacks validate positive quotas/bounds. File-loaded quotas are checked again before ID allocation; the job manager independently rejects non-positive limits. Gate-off admission uses error 5102. Job SHOW commands use FORWARD_NO_SYNC: follower requests execute on the master, without waiting for follower journal replay. This does not make SHOW a follower-local stale read. ## Validation - FE reactor compilation passed (JDK 17, Maven 3.9.9). - 262 tests passed across 16 focused FE test classes, with zero failures/errors/skips: configuration, catalog guard, admission, snapshots, index families, static validation, schema contracts, job queries, loader defenses, parser, command responses, and job authorization. Beyond the admission/concurrency cases (concurrent DROP and identity ALTER versus admission, A → B → A, failed ALTER rollback, replay with null old properties, target changes during IF no-ops, strict numeric parsing, real command-to-CatalogMgr integration), coverage includes WHERE predicate shapes through the real parser (LIKE, reversed literals, `VarcharLiteral` values, which the parser produces for regular string literals), reserved-name rejection without a snapshot read, half-orphan row hiding for non-ADMIN, gate-off persistence assertions, the exactly-one-operation precondition, multi-column stored indexes in the IF preflight, unparsable snapshot properties, the loader's physical-entry defenses (bounds, duplicate UUIDs recorded before the system-entry filter, system-entry filtering, ordering), malformed provider schema facts (negative field ids, non-positive dimensions), the catalog's identity epoch, failure sanitization and REST early rejection, storage-routing key guarding, SHOW locator revalidation and provider-failure non-disclosure, and a real on-disk Lance dataset fixture pinning the fixed-size-list contract semantics. - JaCoCo new-code line coverage: 93% on the new classes (each ≥ 86%), 81% on the new lines of modified files, 90% overall. The remainder is the JNI dataset-open path exercised by the external regression suite. - The full FE test suite (fe-common + fe-core + reactor dependencies, ~9.6k tests) ran to completion on both the pre-review and the review-fix heads: the only failures are this machine's documented timezone-, network- and native-library-related environmental tests — 38 failing methods across 15 classes, verified identical (same classes, same counts) by running the same classes on the merge base. - Maven validate/Checkstyle passed with zero violations; `git diff --check` passed. - Groovy compilation and a harness exercising the actual regression suite's original-value restoration and cleanup-failure propagation passed. - The external MinIO/REST regression was not run locally; the corresponding Docker environment is not running. BE compilation was not run. The external regression suite runs as `nonConcurrent`, captures the original master gate/quota values, and restores them even on failure. Cleanup failures are reported. Admitted PENDING jobs/catalogs remain durable in this slice; per-run names avoid same-name collisions, but repeated runs still consume the global unresolved-job quota. ## Deferred Dispatch, BE selection, worker execution, possible-live slot configuration, and the local/file operator assertion belong to subsequent dispatch/worker slices. FORCE_RELEASE/RESOLVE and retention GC belong to PR3E (#67754). The gate default remains unchanged. REST catalog mutations, ALTER TABLE ADD/DROP INDEX, and BUILD INDEX remain unsupported. ## Release note Experimental, default-disabled FE admission and job inspection for Lance index mutations; actual index execution remains deferred. --------- Co-authored-by: u70b3 <u70b3@users.noreply.github.com>

u70b3 · 2026-09-16T15:56:00Z

run buildall

u70b3 · 2026-09-16T22:22:15Z

run p0

…n SQL and catalog DDL guard (#67630) ## What problem does this PR solve? Issue: #66497 (design v5.1, slice 3). Target: `branch-4.1`. This is PR3C, building on #67201 (PR3A, DDL surface) and #67235 (PR3B, durable jobs), both already merged into branch-4.1. It adds the FE admission path for top-level Lance CREATE, CREATE OR REPLACE, and DROP INDEX, returns durable job IDs, and exposes job inspection SQL. **`enable_lance_index_mutation` remains false by default.** This slice does not execute index mutations: admitted jobs remain PENDING until dispatch/worker support and FORCE_RELEASE arrive in later slices. Enabling it now leaves unresolved jobs that prevent target-changing ALTER CATALOG and DROP CATALOG. ## Behavior - Admission reads dataset version, schema fields, logical indexes, and physical index families from one opened latest Dataset. It does not call `countRows()` or `getIndexStatistics()`. Schema contract v1 preserves Arrow field identity, nullability, vector dimensions/element types, and relevant type parameters for later worker revalidation. For fixed-size lists the pinned SDK reconstructs no child field — the element type lives only in the manifest logical type and element nullability is synthesized as nullable (a manifest-format fact that also binds existing datasets) — so the contract sources element facts from the reconstructed Arrow view and is pinned by a real on-disk dataset fixture. - Reserved names and ambiguous case-only collisions are rejected (reserved names fail before any metadata read). REPLACE and DROP persist the stored display name of a unique case-insensitive match. Column identities that are ambiguous under the table's case-insensitive lookup relation are rejected before anything is journaled. CREATE IF NOT EXISTS is a no-op only when algorithm, physical family, column (compared in the same path-segment representation on both sides), and exposed whitelist properties match — an omitted `num_bits` compares as the persisted 8; DROP IF EXISTS is a no-op when the authoritative name is absent. - Metric comparison ignores case. Numeric properties use strict parsed-long equality for numeric primitives or integer strings; fractions, exponent notation, overflow, and malformed values fail closed. Unexposed properties are skipped, and `num_partitions` is not compared. - An admitted statement returns one `JobId` row after the job and fence are durable; an IF no-op returns the same column with zero rows. The response works for direct and forwarded connections. - `SHOW LANCE INDEX JOBS [FROM [catalog.]db] [WHERE TableName = "t" [AND State = "PENDING"]]` and `SHOW LANCE INDEX JOB <id>` authorize each persisted target. Orphans — including targets whose provider lookup fails (credential expiry, outage) and targets whose currently resolved dataset locator no longer matches the job's persisted locator after a legal identity change — require global ADMIN; unauthorized rows are omitted, and unauthorized/missing job IDs share error 5103, so provider failures never leak through the authorization boundary. Locators, credentials, properties JSON, and schema-contract contents are not exposed. WHERE values accept what the parser actually produces for string literals, including `VarcharLiteral`; the `FROM` database name is canonicalized through the selected catalog's resolution semantics. ## Concurrency and configuration Catalog identity is captured under the CatalogMgr lock before the remote snapshot read. After the read, admission rechecks catalog existence, identity properties, and a local identity-change generation under the same lock used by catalog DDL, then allocates the ID and creates the job. This closes both the guard-check/create race and A → B → A identity changes. IF no-ops also revalidate the target. Identity changes invalidate in-flight reads even when a tentative ALTER is rolled back; same-value rewrites with unchanged key spelling and credential-only changes remain allowed. Metadata I/O runs outside the DDL lock. Final admission uses the lock order CatalogMgr → job manager → journal. DROP and changes to target-identity properties — `lance.catalog.type`, `warehouse`, namespace parent/delimiter/root_database, and the storage-routing endpoint/region keys the Lance property chain actually consumes — reject unresolved jobs; access keys and tokens stay rotatable. Renames and replay are not blocked by the unresolved-job guard. The gate and unresolved-job quotas (table/catalog/global defaults 8/64/256) are mutable and masterOnly. ADMIN SET changes runtime values on the master; it does not persist them to the custom configuration file or automatically synchronize other FEs. Restart loads configured file values/defaults, while promotion uses the promoted FE's own configuration. The two static bounds (`num_partitions` 4096, `num_sub_vectors` 256) are masterOnly — validation runs on the master — so a single-FE ADMIN SET takes effect only there; use ADMIN SET ALL FRONTENDS CONFIG to update the cluster. ADMIN SET callbacks validate positive quotas/bounds. File-loaded quotas are checked again before ID allocation; the job manager independently rejects non-positive limits. Gate-off admission uses error 5102. Job SHOW commands use FORWARD_NO_SYNC: follower requests execute on the master, without waiting for follower journal replay. This does not make SHOW a follower-local stale read. ## Validation - FE reactor compilation passed (JDK 17, Maven 3.9.9). - 262 tests passed across 16 focused FE test classes, with zero failures/errors/skips: configuration, catalog guard, admission, snapshots, index families, static validation, schema contracts, job queries, loader defenses, parser, command responses, and job authorization. Beyond the admission/concurrency cases (concurrent DROP and identity ALTER versus admission, A → B → A, failed ALTER rollback, replay with null old properties, target changes during IF no-ops, strict numeric parsing, real command-to-CatalogMgr integration), coverage includes WHERE predicate shapes through the real parser (LIKE, reversed literals, `VarcharLiteral` values, which the parser produces for regular string literals), reserved-name rejection without a snapshot read, half-orphan row hiding for non-ADMIN, gate-off persistence assertions, the exactly-one-operation precondition, multi-column stored indexes in the IF preflight, unparsable snapshot properties, the loader's physical-entry defenses (bounds, duplicate UUIDs recorded before the system-entry filter, system-entry filtering, ordering), malformed provider schema facts (negative field ids, non-positive dimensions), the catalog's identity epoch, failure sanitization and REST early rejection, storage-routing key guarding, SHOW locator revalidation and provider-failure non-disclosure, and a real on-disk Lance dataset fixture pinning the fixed-size-list contract semantics. - JaCoCo new-code line coverage: 93% on the new classes (each ≥ 86%), 81% on the new lines of modified files, 90% overall. The remainder is the JNI dataset-open path exercised by the external regression suite. - The full FE test suite (fe-common + fe-core + reactor dependencies, ~9.6k tests) ran to completion on both the pre-review and the review-fix heads: the only failures are this machine's documented timezone-, network- and native-library-related environmental tests — 38 failing methods across 15 classes, verified identical (same classes, same counts) by running the same classes on the merge base. - Maven validate/Checkstyle passed with zero violations; `git diff --check` passed. - Groovy compilation and a harness exercising the actual regression suite's original-value restoration and cleanup-failure propagation passed. - The external MinIO/REST regression was not run locally; the corresponding Docker environment is not running. BE compilation was not run. The external regression suite runs as `nonConcurrent`, captures the original master gate/quota values, and restores them even on failure. Cleanup failures are reported. Admitted PENDING jobs/catalogs remain durable in this slice; per-run names avoid same-name collisions, but repeated runs still consume the global unresolved-job quota. ## Deferred Dispatch, BE selection, worker execution, possible-live slot configuration, and the local/file operator assertion belong to subsequent dispatch/worker slices. FORCE_RELEASE/RESOLVE and retention GC belong to PR3E (#67754). The gate default remains unchanged. REST catalog mutations, ALTER TABLE ADD/DROP INDEX, and BUILD INDEX remain unsupported. ## Release note Experimental, default-disabled FE admission and job inspection for Lance index mutations; actual index execution remains deferred. --------- Co-authored-by: u70b3 <u70b3@users.noreply.github.com>

u70b3 · 2026-09-18T02:57:55Z

run p0

u70b3 · 2026-09-18T02:58:10Z

/review

u70b3 · 2026-09-18T04:39:31Z

check_coverage_fe keeps failing before compilation: the step cannot download the maven tarball from the OSS mirror (wget exit 4, builds 1049625 / 1050496, both dead under 30s on two different agents). The coverage merge itself passes whenever the download succeeds (1047935, 1048686). Retrying.

u70b3 · 2026-09-18T04:39:44Z

run p0

u70b3 · 2026-09-18T05:53:02Z

run p0

u70b3 · 2026-09-18T08:58:24Z

run p0

u70b3 · 2026-09-20T01:24:37Z

run p0

u70b3 · 2026-09-20T01:27:30Z

@Gabriel39 @zhangstar333 @yiguolei This is ready for review when you have time. The only red gate is check_coverage_fe — a CI download flake before compilation (details: #67754 (comment)), a rerun is in flight; everything else is green.

u70b3 · 2026-09-20T03:07:56Z

run buildall

…or codes

…ce job manager

…VE command

… its locator RESOLVE authorized through SHOW's targetResolves, which slice 3C's review round made exception-safe for listing: there a provider outage folds into the orphan verdict. That is safe for a display, not for a durable release: the half-orphan branch would skip the authoritative read and release the fence while "table gone" cannot be told apart from "network down" — and nothing in the suite could catch it, because the existing blip test only fails the second resolution (inside the read). Resolve the target three ways up front instead: - RESOLVED — names resolve and the catalog's current durable locator still matches the job's: table-level ALTER authorization and the full protocol (capture, authoritative read, refresh, admission transfer). The locator leg mirrors SHOW LANCE INDEX JOBS, so a repointed dataset reusing the same names never upgrades a stale name into table-level authorization. - MISSING — catalog, database or table verifiably absent, or the locator positively points elsewhere: the orphan family under global ADMIN; a positive repoint takes the half-orphan path (no read, best-effort refresh, release). - FAILED — a resolution that errors out, or a locator that cannot be resolved right now (resolveCurrentIndexJobLocator folds provider outages into null): never an orphan verdict. After ADMIN authorization the statement fails with the typed 5105 so the fence is kept and the operator fixes the cause and retries; SHOW fails the same uncertainty closed by hiding the row, RESOLVE fails it closed by not releasing. Like the 5104 state rejection, the 5105 is only visible to an already authorized caller. Tests: a db or table resolution failure at authorization time keeps the job (5105 for ADMIN, fixed 5103 otherwise, zero manager calls); an unresolvable locator fails closed the same way; a positively repointed locator takes the half-orphan branch (ADMIN-only, no authoritative read, best-effort refresh, one release); the resolved-target fixtures stub the matching locator.

…gate, and log its cause Two review findings on the typed 5105 (ERR_LANCE_INDEX_JOB_RESOLUTION_INCOMPLETE): 1. It was thrown right after authorization, ahead of the idempotent short-circuit and the UNKNOWN state gate, so it could misstate durable facts: a retry of a successful FORCE during a provider outage got a "was not released ... remains UNKNOWN" error instead of the idempotent OK, and a terminal job with an unresolvable target got the same misstatement instead of the accurate 5104. Both checks run after authorization, so moving the rejection behind them changes no disclosure surface; an UNKNOWN job with a failed resolution still gets the 5105. 2. The message told the operator to "see fe.log for the cause", but resolution failures were swallowed without a log line anywhere (the target resolution caught silently, and the locator helper folds provider outages into a null). The command now logs the cause server-side at each failure point; client-facing text still passes only through the sanitized root-cause chain. Also in this round: - Guard the external-relation casts in the authoritative read with instanceof (no raw ClassCastException at a command boundary), and catch Exception during target resolution, mirroring SHOW's targetResolves. - The manager's idempotent short-circuit now honors only a released record in a null/UNKNOWN state: a force-released flag on a non-UNKNOWN state can only come from a corrupt journal or image, so it falls through to the revision CAS and the state gate (fail-closed). Tests: a released job survives a resolution failure with the idempotent OK; a PENDING job with a failed resolution gets 5104, not 5105; the corrupt force-released/PENDING record does not short-circuit; half-orphan authorization gains the same privilege-verification assertions as the full-orphan case; opcode 501 gets the unique-assignment wiring pin; the retention cleaner gets its first unit tests (seconds-to-millis config conversion, per-round cap, a failed round swallowed); the regression suite drops an unused REST port variable.

u70b3 mentioned this pull request Sep 10, 2026

[Feature](lance) Add vector and scalar index lifecycle management #66497

Open

12 tasks

u70b3 force-pushed the pr3e-lance-index-force-release branch from 13b8616 to c457bf3 Compare September 10, 2026 03:57

u70b3 mentioned this pull request Sep 10, 2026

branch-4.1: [feature](lance) Add Lance index admission, job inspection SQL and catalog DDL guard #67630

Merged

u70b3 force-pushed the pr3e-lance-index-force-release branch from c457bf3 to acffc41 Compare September 14, 2026 15:17

u70b3 changed the title ~~branch-4.1: [feature](lance) RESOLVE LANCE INDEX JOB (FORCE_RELEASE) and job retention GC~~ branch-4.1: [feature](lance) Add RESOLVE LANCE INDEX JOB (FORCE_RELEASE) and job retention GC Sep 15, 2026

u70b3 mentioned this pull request Sep 15, 2026

branch-4.1: [feature](lance) Add Lance index job dispatcher with thrift dispatch boundary #67978

Open

12 tasks

u70b3 force-pushed the pr3e-lance-index-force-release branch from 441d711 to bfcc182 Compare September 16, 2026 01:47

u70b3 force-pushed the pr3e-lance-index-force-release branch from bfcc182 to 3f94597 Compare September 16, 2026 15:24

u70b3 marked this pull request as ready for review September 16, 2026 15:25

u70b3 requested a review from yiguolei as a code owner September 16, 2026 15:25

u70b3 force-pushed the pr3e-lance-index-force-release branch from 3f94597 to bff53fa Compare September 20, 2026 02:54

u70b3 added 7 commits September 20, 2026 16:59

[feature](lance) Add Lance index job retention config and resolve err…

847f05b

…or codes

[feature](lance) Add force release transition and retention GC to Lan…

d3c8c56

…ce job manager

[feature](lance) Add RESOLVE LANCE INDEX JOB SQL

c510248

[test](lance) Cover Lance index force release, retention GC and RESOL…

773728a

…VE command

[test](lance) Cover RESOLVE regression negatives

b8c3bde

u70b3 force-pushed the pr3e-lance-index-force-release branch from bff53fa to c1e63ff Compare September 20, 2026 08:59

Conversation

u70b3 commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What lands

Two design readings that need reviewer confirmation

Operational notes

Tests

Deliberately not done

Release note

Uh oh!

hello-stephen commented Sep 10, 2026

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Sep 16, 2026

FE Regression Coverage Report

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

FE Regression Coverage Report

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

u70b3 commented Sep 18, 2026

Uh oh!

u70b3 commented Sep 18, 2026

Uh oh!

u70b3 commented Sep 18, 2026

Uh oh!

u70b3 commented Sep 18, 2026

Uh oh!

u70b3 commented Sep 18, 2026

Uh oh!

u70b3 commented Sep 18, 2026

Uh oh!

u70b3 commented Sep 20, 2026

Uh oh!

u70b3 commented Sep 20, 2026

Uh oh!

u70b3 commented Sep 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

u70b3 commented Sep 10, 2026 •

edited

Loading