branch-4.1: [feature](lance) Durable Lance index job infrastructure with fence, quota and replay - #67235
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
u70b3
marked this pull request as ready for review
August 29, 2026 07:55
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
u70b3
force-pushed
the
pr3b-lance-index-jobs
branch
from
August 29, 2026 07:56
36ed4ef to
bed897a
Compare
Gabriel39
reviewed
Sep 1, 2026
| owners.add(stored.getJobId()); | ||
| quota.charge(stored); | ||
| } else { | ||
| // Corrupt record tolerance: keep it queryable but out of the fence/quota books. |
Contributor
There was a problem hiding this comment.
Could we keep admission fail-closed when an unresolved replayed record lacks fence identity? This branch stores a PENDING/RUNNING/UNKNOWN job but omits it from both the fence/quota books and getUnresolvedJobs(). After replay, a new job for the same real target can therefore pass admission and be dispatched even though the old mutation may still be live or may already have committed. That conflicts with the core invariant that ambiguous outcomes retain the fence. Since the key cannot be reconstructed, please introduce a global corrupt-unresolved admission blocker (or fail replay/startup) instead of silently excluding this record from the books.
Contributor
Author
There was a problem hiding this comment.
Good catch — fixed in fb70bf2.
createJob now rejects every admission while any replayed unresolved record lacks fence identity: a derived corruptUnresolvedJobIds set is maintained in applyToMemory, rebuilt on image load, and checked under the write lock before the fence CAS. The rejection is bounded (count + smallest job id) and discloses no target identity; in-flight transitions of already-admitted jobs are untouched.
I didn't make replay/startup fail: one corrupt record would then crash-loop every FE, which is what the replay-never-throws convention is for.
The blockade lifts when a later durable record resolves the job, and the force-release follow-up will release such jobs by id, without needing the fence key. Covered by new replay/persist tests; 126 lance.job + 524 scoped cases green.
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
FE UT Coverage ReportIncrement line coverage |
u70b3
force-pushed
the
pr3b-lance-index-jobs
branch
from
September 7, 2026 08:57
fb70bf2 to
ac29b4c
Compare
Contributor
Author
|
run buildall |
Second sub-PR (PR3B) of slice 3 of the Lance index lifecycle design (apache#66497, v5.1 contract): the durable job model behind the one-shot mutation lifecycle. No user-visible entry point; admission, dispatch, and FORCE land in follow-up PRs. Model pieces (design sections in parentheses): - LanceIndexJob: the minimal durable job record (7.2) - identity, creator, revision, bounded timestamps, persisted target identity and same-name fence key material, mutation intent, admitted dataset version with the ordered schema-contract-v1 representation (4.2), independent mutation/refresh states, typed result with bounded sanitized message, dispatch identity (backend id, BE process epoch, immutable invocation id, deadline), possible-live ownership with termination proof, and the FORCE audit fields (populated by PR3E). No credentials, no unbounded values (4.3/8). - Dual state machines: PENDING -> RUNNING -> COMMITTED|NOT_COMMITTED| UNKNOWN (6.1, UNKNOWN terminal with no outgoing transition) and the independent NOT_REQUIRED|REQUIRED|RUNNING|DONE|FAILED refresh state (6.2). - LanceIndexJobResultCode: the provider-result classification table (6.3) as typed codes plus one pure classify(); IF_CONDITION_NOOP only for DROP IF EXISTS + LANCE_ERR_NOT_FOUND. - Normalization v1 (4.1): index names via toLowerCase(Locale.ROOT); dataset locators via trim, lowercased scheme, trailing-slash strip, and rejection of credential-bearing or identity-less forms. - LanceIndexFenceKey: (catalog id, DIRECTORY provider, normalized locator, normalized index name); display name is persisted on the job, never in the key. toString hides the locator.
…and quota PR3B part 2: the master-owned job/fence manager (Appendix B seam) plus edit-log and image wiring. It deliberately reuses neither the generic scheduling JobManager nor internal IndexChangeJob: the external one-shot CAS, no-redispatch rule, same-name fence, and possible-live ownership required by the design are not provided by either. - LanceIndexJobManager: every durable transition shares one write-path shape - validate under the write lock (state legality, revision CAS, callback identity), append one upsert record, then apply the same record locally - so master and followers run identical apply logic. Fence and unresolved quota (table-locator/catalog/global, 5.4) live and die together per 6.4: held by PENDING/RUNNING, by terminal jobs until their required refresh is DONE, and by UNKNOWN until a durable FORCE_RELEASE; rejection precedes any durable write, leaving no job, no fence, and no record. - Replay per 7.3: replayUpsertJob is a verbatim replace with a monotonic-revision guard and performs no state transformation, so a follower tailing a live master keeps a fresh RUNNING record RUNNING. RUNNING without a complete terminal result becomes UNKNOWN only in the master-election sweep (Env.transferToMaster, after metadata replay and before master daemons start, mirroring the insertOverwriteManager.allTaskFail precedent) through the same identity-checked channel; refresh RUNNING is downgraded to REQUIRED so the idempotent external-table refresh can resume. Replay never redispatches and never calls lance-c again. - Wiring: OP_LANCE_INDEX_JOB_UPSERT = 500 (verified unique), JournalEntity/EditLog dispatch, a lanceIndexJobManager image module appended to PersistMetaModules (no FeMetaVersion bump; old images never invoke the load method and Env pre-initializes an empty manager).
PR3B unit tests (105 cases, pure UT, no FE service): - Normalization v1 incl. the Turkish dotted-I corner; locator forms and rejections (userinfo, empty scheme, relative path, no identity). - Section 6.3 classification matrix independently restated per cell; IF_CONDITION_NOOP confined to DROP IF EXISTS + NOT_FOUND. - State machine legality: UNKNOWN has no outgoing transitions, refresh transitions stay independent, revision CAS, blank invocation ids rejected at the dispatch boundary. - Fence/quota co-release timing (immediate on NOT_REQUIRED, on refresh DONE, never for FAILED/UNKNOWN), three-level quota boundaries, and rebuild equivalence after image load. - Section 7.3 replay matrix: PENDING re-dispatchable once; RUNNING swept to UNKNOWN at master transfer and never redispatchable; terminal jobs resume only refresh (REQUIRED and FAILED stay visible to the refresh driver); UNKNOWN rebuilds fence/quota/possible-live; force-released UNKNOWN frees the name; stale callbacks rejected on revision/invocation/epoch mismatch; replay idempotent with a monotonic revision guard; identity-less corrupt records tolerated without throwing, including follow-up upserts for the same job id. - Manager image write/read round-trip rebuilds derived fence/quota; JournalEntity round-trip covers the new op-500 dispatch; over-bounds text fields rejected at construction.
…ved records A replayed unresolved job record that lacks fence identity (corrupt journal/image metadata: provider, locator, or index name missing) was stored and queryable but excluded from the fence and quota books, so a new job for the same real target could pass admission and be dispatched while the old mutation's outcome is still ambiguous — breaking the retain-the-fence-on-ambiguity invariant. Keep admission fail-closed instead: a derived corruptUnresolvedJobIds set tracks such records (maintained in applyToMemory, rebuilt on image load), and createJob rejects every admission under the write lock while the set is non-empty, before the fence CAS, with a bounded message that discloses no target identity. Only admission is blocked; in-flight lifecycle steps of healthy jobs keep proceeding. The blockade lifts when a later durable record settles the job, or when the force-release transition of a follow-up PR releases it by id without needing its fence key. Also rewrite the isUnresolved() javadoc: holding the fence requires the fence identity to survive, which the old wording got wrong for identity-less records.
u70b3
force-pushed
the
pr3b-lance-index-jobs
branch
from
September 8, 2026 01:29
ac29b4c to
2a82a34
Compare
Contributor
Author
|
run buildall |
Contributor
Author
|
@Gabriel39 The unresolved-record fence blocker you raised is fixed in fb70bf2 (details in the reply above). FE UT and coverage are green; COMPILE is still running. Could you take another look / approve when you have a moment? cc @zhangstar333 for a second approval. Thanks! |
Contributor
Author
|
run nonConcurrent |
Contributor
Author
|
run check_coverage |
u70b3
changed the title
Gabriel39
approved these changes
Sep 9, 2026
12 tasks
Contributor
Author
|
run check_coverage |
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
Ping @yiguolei @zhangstar333 — the sibling PR #67201 has been merged (thanks!). All checks here are green now; one more approval and this can go in, which unblocks the follow-up PRs that depend on both (admission / dispatch slices). PTAL when you have a moment. Thanks! |
yiguolei
approved these changes
Sep 10, 2026
Contributor
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Open
12 tasks
yiguolei
pushed a commit
that referenced
this pull request
Sep 16, 2026
…n SQL and catalog DDL guard (#67630) ## What problem does this PR solve? Issue: #66497 (design v5.1, slice 3). Target: `branch-4.1`. This is PR3C, building on #67201 (PR3A, DDL surface) and #67235 (PR3B, durable jobs), both already merged into branch-4.1. It adds the FE admission path for top-level Lance CREATE, CREATE OR REPLACE, and DROP INDEX, returns durable job IDs, and exposes job inspection SQL. **`enable_lance_index_mutation` remains false by default.** This slice does not execute index mutations: admitted jobs remain PENDING until dispatch/worker support and FORCE_RELEASE arrive in later slices. Enabling it now leaves unresolved jobs that prevent target-changing ALTER CATALOG and DROP CATALOG. ## Behavior - Admission reads dataset version, schema fields, logical indexes, and physical index families from one opened latest Dataset. It does not call `countRows()` or `getIndexStatistics()`. Schema contract v1 preserves Arrow field identity, nullability, vector dimensions/element types, and relevant type parameters for later worker revalidation. For fixed-size lists the pinned SDK reconstructs no child field — the element type lives only in the manifest logical type and element nullability is synthesized as nullable (a manifest-format fact that also binds existing datasets) — so the contract sources element facts from the reconstructed Arrow view and is pinned by a real on-disk dataset fixture. - Reserved names and ambiguous case-only collisions are rejected (reserved names fail before any metadata read). REPLACE and DROP persist the stored display name of a unique case-insensitive match. Column identities that are ambiguous under the table's case-insensitive lookup relation are rejected before anything is journaled. CREATE IF NOT EXISTS is a no-op only when algorithm, physical family, column (compared in the same path-segment representation on both sides), and exposed whitelist properties match — an omitted `num_bits` compares as the persisted 8; DROP IF EXISTS is a no-op when the authoritative name is absent. - Metric comparison ignores case. Numeric properties use strict parsed-long equality for numeric primitives or integer strings; fractions, exponent notation, overflow, and malformed values fail closed. Unexposed properties are skipped, and `num_partitions` is not compared. - An admitted statement returns one `JobId` row after the job and fence are durable; an IF no-op returns the same column with zero rows. The response works for direct and forwarded connections. - `SHOW LANCE INDEX JOBS [FROM [catalog.]db] [WHERE TableName = "t" [AND State = "PENDING"]]` and `SHOW LANCE INDEX JOB <id>` authorize each persisted target. Orphans — including targets whose provider lookup fails (credential expiry, outage) and targets whose currently resolved dataset locator no longer matches the job's persisted locator after a legal identity change — require global ADMIN; unauthorized rows are omitted, and unauthorized/missing job IDs share error 5103, so provider failures never leak through the authorization boundary. Locators, credentials, properties JSON, and schema-contract contents are not exposed. WHERE values accept what the parser actually produces for string literals, including `VarcharLiteral`; the `FROM` database name is canonicalized through the selected catalog's resolution semantics. ## Concurrency and configuration Catalog identity is captured under the CatalogMgr lock before the remote snapshot read. After the read, admission rechecks catalog existence, identity properties, and a local identity-change generation under the same lock used by catalog DDL, then allocates the ID and creates the job. This closes both the guard-check/create race and A → B → A identity changes. IF no-ops also revalidate the target. Identity changes invalidate in-flight reads even when a tentative ALTER is rolled back; same-value rewrites with unchanged key spelling and credential-only changes remain allowed. Metadata I/O runs outside the DDL lock. Final admission uses the lock order CatalogMgr → job manager → journal. DROP and changes to target-identity properties — `lance.catalog.type`, `warehouse`, namespace parent/delimiter/root_database, and the storage-routing endpoint/region keys the Lance property chain actually consumes — reject unresolved jobs; access keys and tokens stay rotatable. Renames and replay are not blocked by the unresolved-job guard. The gate and unresolved-job quotas (table/catalog/global defaults 8/64/256) are mutable and masterOnly. ADMIN SET changes runtime values on the master; it does not persist them to the custom configuration file or automatically synchronize other FEs. Restart loads configured file values/defaults, while promotion uses the promoted FE's own configuration. The two static bounds (`num_partitions` 4096, `num_sub_vectors` 256) are masterOnly — validation runs on the master — so a single-FE ADMIN SET takes effect only there; use ADMIN SET ALL FRONTENDS CONFIG to update the cluster. ADMIN SET callbacks validate positive quotas/bounds. File-loaded quotas are checked again before ID allocation; the job manager independently rejects non-positive limits. Gate-off admission uses error 5102. Job SHOW commands use FORWARD_NO_SYNC: follower requests execute on the master, without waiting for follower journal replay. This does not make SHOW a follower-local stale read. ## Validation - FE reactor compilation passed (JDK 17, Maven 3.9.9). - 262 tests passed across 16 focused FE test classes, with zero failures/errors/skips: configuration, catalog guard, admission, snapshots, index families, static validation, schema contracts, job queries, loader defenses, parser, command responses, and job authorization. Beyond the admission/concurrency cases (concurrent DROP and identity ALTER versus admission, A → B → A, failed ALTER rollback, replay with null old properties, target changes during IF no-ops, strict numeric parsing, real command-to-CatalogMgr integration), coverage includes WHERE predicate shapes through the real parser (LIKE, reversed literals, `VarcharLiteral` values, which the parser produces for regular string literals), reserved-name rejection without a snapshot read, half-orphan row hiding for non-ADMIN, gate-off persistence assertions, the exactly-one-operation precondition, multi-column stored indexes in the IF preflight, unparsable snapshot properties, the loader's physical-entry defenses (bounds, duplicate UUIDs recorded before the system-entry filter, system-entry filtering, ordering), malformed provider schema facts (negative field ids, non-positive dimensions), the catalog's identity epoch, failure sanitization and REST early rejection, storage-routing key guarding, SHOW locator revalidation and provider-failure non-disclosure, and a real on-disk Lance dataset fixture pinning the fixed-size-list contract semantics. - JaCoCo new-code line coverage: 93% on the new classes (each ≥ 86%), 81% on the new lines of modified files, 90% overall. The remainder is the JNI dataset-open path exercised by the external regression suite. - The full FE test suite (fe-common + fe-core + reactor dependencies, ~9.6k tests) ran to completion on both the pre-review and the review-fix heads: the only failures are this machine's documented timezone-, network- and native-library-related environmental tests — 38 failing methods across 15 classes, verified identical (same classes, same counts) by running the same classes on the merge base. - Maven validate/Checkstyle passed with zero violations; `git diff --check` passed. - Groovy compilation and a harness exercising the actual regression suite's original-value restoration and cleanup-failure propagation passed. - The external MinIO/REST regression was not run locally; the corresponding Docker environment is not running. BE compilation was not run. The external regression suite runs as `nonConcurrent`, captures the original master gate/quota values, and restores them even on failure. Cleanup failures are reported. Admitted PENDING jobs/catalogs remain durable in this slice; per-run names avoid same-name collisions, but repeated runs still consume the global unresolved-job quota. ## Deferred Dispatch, BE selection, worker execution, possible-live slot configuration, and the local/file operator assertion belong to subsequent dispatch/worker slices. FORCE_RELEASE/RESOLVE and retention GC belong to PR3E (#67754). The gate default remains unchanged. REST catalog mutations, ALTER TABLE ADD/DROP INDEX, and BUILD INDEX remain unsupported. ## Release note Experimental, default-disabled FE admission and job inspection for Lance index mutations; actual index execution remains deferred. --------- Co-authored-by: u70b3 <u70b3@users.noreply.github.com>
yiguolei
pushed a commit
that referenced
this pull request
Sep 17, 2026
…n SQL and catalog DDL guard (#67630) ## What problem does this PR solve? Issue: #66497 (design v5.1, slice 3). Target: `branch-4.1`. This is PR3C, building on #67201 (PR3A, DDL surface) and #67235 (PR3B, durable jobs), both already merged into branch-4.1. It adds the FE admission path for top-level Lance CREATE, CREATE OR REPLACE, and DROP INDEX, returns durable job IDs, and exposes job inspection SQL. **`enable_lance_index_mutation` remains false by default.** This slice does not execute index mutations: admitted jobs remain PENDING until dispatch/worker support and FORCE_RELEASE arrive in later slices. Enabling it now leaves unresolved jobs that prevent target-changing ALTER CATALOG and DROP CATALOG. ## Behavior - Admission reads dataset version, schema fields, logical indexes, and physical index families from one opened latest Dataset. It does not call `countRows()` or `getIndexStatistics()`. Schema contract v1 preserves Arrow field identity, nullability, vector dimensions/element types, and relevant type parameters for later worker revalidation. For fixed-size lists the pinned SDK reconstructs no child field — the element type lives only in the manifest logical type and element nullability is synthesized as nullable (a manifest-format fact that also binds existing datasets) — so the contract sources element facts from the reconstructed Arrow view and is pinned by a real on-disk dataset fixture. - Reserved names and ambiguous case-only collisions are rejected (reserved names fail before any metadata read). REPLACE and DROP persist the stored display name of a unique case-insensitive match. Column identities that are ambiguous under the table's case-insensitive lookup relation are rejected before anything is journaled. CREATE IF NOT EXISTS is a no-op only when algorithm, physical family, column (compared in the same path-segment representation on both sides), and exposed whitelist properties match — an omitted `num_bits` compares as the persisted 8; DROP IF EXISTS is a no-op when the authoritative name is absent. - Metric comparison ignores case. Numeric properties use strict parsed-long equality for numeric primitives or integer strings; fractions, exponent notation, overflow, and malformed values fail closed. Unexposed properties are skipped, and `num_partitions` is not compared. - An admitted statement returns one `JobId` row after the job and fence are durable; an IF no-op returns the same column with zero rows. The response works for direct and forwarded connections. - `SHOW LANCE INDEX JOBS [FROM [catalog.]db] [WHERE TableName = "t" [AND State = "PENDING"]]` and `SHOW LANCE INDEX JOB <id>` authorize each persisted target. Orphans — including targets whose provider lookup fails (credential expiry, outage) and targets whose currently resolved dataset locator no longer matches the job's persisted locator after a legal identity change — require global ADMIN; unauthorized rows are omitted, and unauthorized/missing job IDs share error 5103, so provider failures never leak through the authorization boundary. Locators, credentials, properties JSON, and schema-contract contents are not exposed. WHERE values accept what the parser actually produces for string literals, including `VarcharLiteral`; the `FROM` database name is canonicalized through the selected catalog's resolution semantics. ## Concurrency and configuration Catalog identity is captured under the CatalogMgr lock before the remote snapshot read. After the read, admission rechecks catalog existence, identity properties, and a local identity-change generation under the same lock used by catalog DDL, then allocates the ID and creates the job. This closes both the guard-check/create race and A → B → A identity changes. IF no-ops also revalidate the target. Identity changes invalidate in-flight reads even when a tentative ALTER is rolled back; same-value rewrites with unchanged key spelling and credential-only changes remain allowed. Metadata I/O runs outside the DDL lock. Final admission uses the lock order CatalogMgr → job manager → journal. DROP and changes to target-identity properties — `lance.catalog.type`, `warehouse`, namespace parent/delimiter/root_database, and the storage-routing endpoint/region keys the Lance property chain actually consumes — reject unresolved jobs; access keys and tokens stay rotatable. Renames and replay are not blocked by the unresolved-job guard. The gate and unresolved-job quotas (table/catalog/global defaults 8/64/256) are mutable and masterOnly. ADMIN SET changes runtime values on the master; it does not persist them to the custom configuration file or automatically synchronize other FEs. Restart loads configured file values/defaults, while promotion uses the promoted FE's own configuration. The two static bounds (`num_partitions` 4096, `num_sub_vectors` 256) are masterOnly — validation runs on the master — so a single-FE ADMIN SET takes effect only there; use ADMIN SET ALL FRONTENDS CONFIG to update the cluster. ADMIN SET callbacks validate positive quotas/bounds. File-loaded quotas are checked again before ID allocation; the job manager independently rejects non-positive limits. Gate-off admission uses error 5102. Job SHOW commands use FORWARD_NO_SYNC: follower requests execute on the master, without waiting for follower journal replay. This does not make SHOW a follower-local stale read. ## Validation - FE reactor compilation passed (JDK 17, Maven 3.9.9). - 262 tests passed across 16 focused FE test classes, with zero failures/errors/skips: configuration, catalog guard, admission, snapshots, index families, static validation, schema contracts, job queries, loader defenses, parser, command responses, and job authorization. Beyond the admission/concurrency cases (concurrent DROP and identity ALTER versus admission, A → B → A, failed ALTER rollback, replay with null old properties, target changes during IF no-ops, strict numeric parsing, real command-to-CatalogMgr integration), coverage includes WHERE predicate shapes through the real parser (LIKE, reversed literals, `VarcharLiteral` values, which the parser produces for regular string literals), reserved-name rejection without a snapshot read, half-orphan row hiding for non-ADMIN, gate-off persistence assertions, the exactly-one-operation precondition, multi-column stored indexes in the IF preflight, unparsable snapshot properties, the loader's physical-entry defenses (bounds, duplicate UUIDs recorded before the system-entry filter, system-entry filtering, ordering), malformed provider schema facts (negative field ids, non-positive dimensions), the catalog's identity epoch, failure sanitization and REST early rejection, storage-routing key guarding, SHOW locator revalidation and provider-failure non-disclosure, and a real on-disk Lance dataset fixture pinning the fixed-size-list contract semantics. - JaCoCo new-code line coverage: 93% on the new classes (each ≥ 86%), 81% on the new lines of modified files, 90% overall. The remainder is the JNI dataset-open path exercised by the external regression suite. - The full FE test suite (fe-common + fe-core + reactor dependencies, ~9.6k tests) ran to completion on both the pre-review and the review-fix heads: the only failures are this machine's documented timezone-, network- and native-library-related environmental tests — 38 failing methods across 15 classes, verified identical (same classes, same counts) by running the same classes on the merge base. - Maven validate/Checkstyle passed with zero violations; `git diff --check` passed. - Groovy compilation and a harness exercising the actual regression suite's original-value restoration and cleanup-failure propagation passed. - The external MinIO/REST regression was not run locally; the corresponding Docker environment is not running. BE compilation was not run. The external regression suite runs as `nonConcurrent`, captures the original master gate/quota values, and restores them even on failure. Cleanup failures are reported. Admitted PENDING jobs/catalogs remain durable in this slice; per-run names avoid same-name collisions, but repeated runs still consume the global unresolved-job quota. ## Deferred Dispatch, BE selection, worker execution, possible-live slot configuration, and the local/file operator assertion belong to subsequent dispatch/worker slices. FORCE_RELEASE/RESOLVE and retention GC belong to PR3E (#67754). The gate default remains unchanged. REST catalog mutations, ALTER TABLE ADD/DROP INDEX, and BUILD INDEX remain unsupported. ## Release note Experimental, default-disabled FE admission and job inspection for Lance index mutations; actual index execution remains deferred. --------- Co-authored-by: u70b3 <u70b3@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: #66497
Related PR: #66637 (merged), #67201 (open — independent; this PR shares no files with it), #66671 (open — independent)
Problem Summary:
This is the second sub-PR of delivery slice 3 of the v5.1 design (final 4.2 contract, scope confirmed in this review): the durable Lance index job infrastructure — job records, the compact dual-state lifecycle, the same-name fence, three-level unresolved quotas, typed results, and replay semantics — behind a master-owned manager. This PR has no user-visible entry point: no SQL, no admission, no dispatcher, no worker, no enablement gate. It is the persistence and state-machine foundation that the admission and dispatch follow-ups build on.
What this PR adds:
LanceIndexJob): job identity/creator/revision/bounded timestamps, persisted target identity and same-name fence key material (catalog id,DIRECTORYprovider tag, normalized dataset locator, display + normalized index name), mutation intent, admitted dataset version and the ordered schema-contract-v1 representation (Section 4.2), the independent mutation/refresh states, the typed result with bounded sanitized message, dispatch identity (backend id, BE process epoch, immutable invocation id, deadline), possible-live ownership with termination proof, and the FORCE audit fields (populated by a follow-up PR; only replay semantics land here). The record carries no credentials and no unbounded values (Section 4.3/8).classifyfunction (13 typed result codes × CREATE/REPLACE/DROP × IF-flags × external-advancement), includingIF_CONDITION_NOOPonly forDROP IF EXISTS+LANCE_ERR_NOT_FOUND. Normalization v1 (Section 4.1) for index names (toLowerCase(Locale.ROOT)) and dataset locators (scheme case, trailing slashes, credential-bearing URL rejection); the fence key and the three-level (table-locator/catalog/global) unresolved-quota counters of Section 5.4.LanceIndexJobManager(Appendix B seam: a master-owned minimal job/fence manager; it deliberately reuses neither the generic schedulingJobManagernor internalIndexChangeJob, neither of which provides external one-shot CAS, no-redispatch, same-name fence, or possible-live semantics). All durable transitions share one write-path shape — validate under the write lock (state legality, revision CAS, callback identity), append one upsert record to the edit log, then apply the same record locally — so master and followers run identical apply logic. Fence and unresolved quota live and die together exactly as Sections 5.4/6.4 require: held by PENDING/RUNNING, by terminal jobs until their required refresh is DONE, and by UNKNOWN until a durable FORCE_RELEASE; a quota or fence rejection precedes any durable write, leaving no job, no fence, and no record. Admission is fail-closed on corrupt metadata: a replayed unresolved record that lacks fence identity (whose fence key cannot be reconstructed) stays queryable but out of the fence/quota books, and instead blocks every new admission — with a bounded message that discloses no target identity — until a later durable record resolves it or the force-release transition of a follow-up PR releases it by job id.replayUpsertJobis a verbatim replace with a monotonic-revision guard and performs no state transformation — a follower tailing a live master must keep a fresh RUNNING record RUNNING. The RUNNING→UNKNOWN transition happens only in the master-election sweep (Env.transferToMaster, after metadata replay and before any master daemon starts, mirroring theinsertOverwriteManager.allTaskFail()precedent), which writes UNKNOWN upserts through the same identity-checked channel so followers converge and stale callbacks (revision/invocation/epoch mismatch) are rejected. Refresh RUNNING is downgraded to REQUIRED at the sweep so the idempotent external-table refresh can resume; replay never calls lance-c again.OP_LANCE_INDEX_JOB_UPSERT = 500),JournalEntity/EditLogdispatch, a newlanceIndexJobManagerimage module appended toPersistMetaModules.MODULE_NAMES(noFeMetaVersionbump; old images simply never invoke the load method, andEnvpre-initializes an empty manager).Explicitly not in this PR: SQL/admission/IF preflight and job SQL (the admission follow-up, including the Section 9.7 gating configuration — quota limits here are parameters, not Config); BE selection, dispatch, possible-live slot reservation on BEs, and worker invocation (the dispatch follow-up); the
FORCE_RELEASEtransition and its auth protocol (a follow-up PR — only the durable fields and the replay row exist); Arrow-schema→contract construction (the admission follow-up); job-record retention/GC (follows the bounded-retention policy with the force-release follow-up). PR1'sSHOW INDEXand PR2's inspection surface are untouched; this branch shares no files with #67201.Release note
None (internal infrastructure only; no user-visible behavior change).
Check List (For Author)
Test
org.apache.doris.datasource.lance.job(126 cases, all green with checkstyle enabled): normalization v1 (incl. the Turkish-İ corner), locator normalization and rejection forms, the full Section 6.3 classification matrix (independently restated per cell), state-machine legality (UNKNOWN has no outgoing transitions; refresh independence; revision CAS), fence/quota co-release timing, quota three-level boundaries, the Section 7.3 replay matrix (PENDING re-dispatchable once; RUNNING swept to UNKNOWN and never redispatchable; terminal jobs resume only refresh; UNKNOWN rebuilds fence/quota/possible-live; force-released UNKNOWN frees the name), stale-callback rejection, replay idempotence and monotonic revision, corrupt-record tolerance, manager image write/read round-trip with derived-state rebuild, andJournalEntityop-500 round-trip.org.apache.doris.persist.**,org.apache.doris.journal.**,org.apache.doris.dictionary.**,org.apache.doris.indexpolicy.**,org.apache.doris.job.**,org.apache.doris.datasource.lance.**(cd fe && mvn test -pl fe-common,fe-core -am -DfailIfNoTests=false -Dtest='...', no-Dcheckstyle.skip).mvn compile -pl fe-common,fe-core -amgreen with checkstyle (validate phase); the new image module binding resolves atPersistMetaModulesstatic init; the edit-log op code 500 verified unique repo-wide.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)