test: add LMI cloud lifecycle and concurrency regressions

[zhongkechen]

This adds a dedicated LMI cloud suite for the invocation lifecycle and executor problems reported in #726. It deploys the SDK commit under test on Java 25 / arm64 with LMI concurrency 1, 2 and 8 using CAPACITY_PROVIDER_ARN; no production SDK behavior is changed. The regression assertions intentionally fail until the corresponding fixes land.

Issue Link, if available

Closes #727. Regression coverage for #726; this PR does not fix or close #726.

Description

• Create or update the persistent java-lmi-e2e stack, five fixed-name functions, log groups, and private staging/control bucket. Functions declare native FunctionScalingConfig with minimum/maximum one environment, use 2 GiB / 1 vCPU, and invoke the mutable published $LATEST.PUBLISHED version supported by LMI. Use content-addressed SDK jars, wait for prior durable executions before updating, and verify the deployed artifact and runtime run/commit identity.
• Require overlapping request IDs in one JVM, trace actual wrapper/root/task boundaries and real checkpoint calls, and check suspension/replay, deadline cancellation, original-environment worker recovery, fixed-pool/nested progress, in-flight failure cleanup and repeated warm execution.
• Keep setup/placement/collection errors distinct from SDK assertions. Preserve JUnit reports, configurations, histories and diagnostic logs. The stack, functions, bucket and code artifacts remain after success or failure; automatic deletion and the janitor are removed. Control objects use per-run prefixes and one-day expiry. Test escape timers still fail the corresponding regressions.
• Run the cloud suite on every push to main, including every merged change, without path filters. Manual dispatch and same-repository PR opt-in with run-lmi-e2e remain available. There are no scheduled jobs. Local Maven builds keep cloud tests and the red regressions opt-in.

The design, prerequisites, budgets, commands and infrastructure ownership are documented in lmi-tests/README.md.

Demo/Screenshots

Not applicable. Cloud diagnostics and JUnit results are uploaded as workflow artifacts.

Checklist

• I have filled out every section of the PR template
• I have thoroughly tested this change — local validation passed; cloud deployment validation is in progress

Testing

Unit Tests

• Normal SDK reactor build/tests and fixture packaging passed: mvn -B -q -pl lmi-tests -am package.
• 40 Python evidence, persistence, deployment, and CLI contract tests passed, including negative controls for different-JVM placement, early PENDING, missing evidence, late checkpoints, replayed step bodies and executor escape false positives.
• Spotless and actionlint passed. The deployment tests verify stable names across create/update runs, content-addressed code updates, ownership checks, preservation of failed stacks, waiting for prior executions, isolated control objects, creation-time scaling limits, and complete-suite coverage. Invocation API errors are preserved in redacted artifacts and reported as collection errors instead of SDK lifecycle assertion failures. A real AWS CLI contract test against an unsigned localhost endpoint verifies synchronous and asynchronous requests, payload bytes, and durable execution ARN/result parsing. Invocation records include request state and elapsed time.
• Explicitly enabled local regressions completed with 4 tests, 4 assertion failures, 0 errors, 0 skips, reproducing all expected failures: root-finally/PENDING ordering, deadline task cancellation, two concurrent fixed-pool roots and nested child orchestration. Command: mvn -pl sdk test -Dtest=LmiLifecycleRegressionTest -Dtest.lmi.regressions.enabled=true.

Integration Tests

Latest cloud run: https://github.com/aws/aws-durable-execution-sdk-java/actions/runs/35800330231 (commit 83a38c0b), pending behind the preceding run. Persistent create/update behavior is awaiting cloud validation. The suite keeps $LATEST.PUBLISHED: LMI requires a published version, even though generic Durable Functions documentation permits $LATEST. The real-CLI contract tests and fixture packaging/format checks passed locally. The core SDK behavior in #726 remains unchanged.

Examples

These are fault-injection fixtures in a separate test module, not customer examples.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[zhongkechen]

Per-test analysis of LMI run 35794439406

Scope: workflow run, attempt 2, PR head 340433bcaa2a10cd2363ff01c60120c819face1e. The artifact manifest records the checked-out PR merge commit as 9f57aeb1919fc75835c1e9abd3d754c560ffc664. This describes that run, not subsequent revisions of this PR.

Evidence reviewed: job results and the lmi-e2e-35794439406-2 artifact, including junit.xml, diagnostics.json, cloudwatch.json, invocation responses, execution histories, and the timeout report. The fixture and assertion code was also checked.

Overall: 13 tests — 3 PASS, 6 assertion FAILURES, 4 collection ERRORS. Build, deployment/readback, final evidence collection, resource deletion, and artifact publication succeeded. The workflow failed in the cloud assertions step.

What the configurations mean

• default1, default2, and default8 use the SDK's default cached executor. Their numbers are the maximum concurrent Lambda invocations per execution environment, not executor thread counts or vCPU counts.
• fixed2 and nested2 each allow two concurrent Lambda invocations and share a fixed two-thread SDK user executor.
• Timeout fixtures use a 60-second invocation timeout and a separate 240-second durable execution timeout. The deliberately blocked tasks have an 80-second test escape. The cleanup assertion allows the invocation deadline plus 5 seconds; recovery admission allows plus 8 seconds.
• A test-only escape is a failure signal. Eventual completion after expanding the executor or releasing an escaped task does not make the regression pass.

Links to the corresponding problems in #726

• Problem 1 — invocation cleanup waits indefinitely: missing invocation deadlines, task cancellation/draining, and the resulting occupied runtime capacity. Its separate interrupted-drain/checkpoint-shutdown subproblem is not demonstrated by this run.
• Problem 2 — PENDING can precede root-handler cleanup: the root future is not included in the required drain before returning.
• Problem 3 — shared fixed-pool starvation: blocking orchestration occupies the same workers needed by its children/steps.

The links below distinguish direct reproduction, downstream effects of earlier failures, and the contract a test intended to check but did not reach. A collection error is not automatically another SDK defect.

Passing cases

Test	Result and demonstrated scope
baseline-concurrency1	PASS. Successful/failed checkpoints, real wait/resume, stable identities and stored failure meaning passed. This case does not contain delayed root cleanup.
default2-isolation	PASS. Two distinct invocations overlapped in one JVM, returned their own markers, and satisfied the tested normal lifecycle boundaries.
default8-isolation	PASS. Eight distinct invocations overlapped in one JVM and returned their own markers.

Non-passing cases

Severity rates potential customer impact in the affected workload; it is not a production-incident classification or a measured customer incidence rate. The tests establish failure mechanisms, not how many customers use the triggering patterns. A High rating for a related SDK issue does not turn a collection error or coverage gap into another confirmed product defect.

default2-suspend-cleanup-replay — assertion failure

• Scenario: On the function configured for two concurrent invocations per JVM and the default cached executor, run a victim through a successful step, a deliberately failed-and-caught step, a durable wait and replay. Keep one healthy companion active in that JVM. The victim delays its root-handler finally by two seconds.
• Expected behavior: Root cleanup and root exit must precede a normal PENDING response. The healthy companion must continue progressing during cleanup. A real resumed invocation must reproduce checkpointed successes/failures without rerunning their bodies or changing operation identity.
• Actual behavior: The SDK wrapper returned normal PENDING while the root's invocation-local finally was still running. For 4d566d3f8b4c-victim-0, wrapper return was sequence 126, root exit was 132, 1.999 seconds later. Replay and the healthy companion's progress checks passed before the ordering assertion failed.
• Severity: Medium generally; potentially High when cleanup protects critical resources or state. Customer impact is conditional on what the delayed cleanup touches and how long it lasts. It can race with later work, but trivial, invocation-private cleanup may have little visible effect. This run confirms the ordering defect; it does not demonstrate resource corruption, repeated checkpointed side effects, or lost checkpoints.
• Cause and corresponding issue: Direct reproduction of #726 problem 2. Suspension can win CompletableFuture.anyOf before the root finishes unwinding, while the root future is absent from the operations drained by close().

default2-timeout-recovery — collection error

• Scenario: Start an interruptible asynchronous step with an 80-second test escape, then return from the root while the step is still blocked. The Lambda invocation timeout is 60 seconds. Stagger one healthy companion into the same JVM, hold its slot, and submit probes for the victim's slot.
• Expected behavior: Establish the healthy companion first, then observe real deadline handling: request cancellation, actual task exit and outer SDK return by the invocation deadline plus five seconds, continued healthy progress, and probe admission in the original JVM by deadline plus eight seconds while the other slot remains held. No test escape should be needed. Log delivery has a separate observation budget and must not be confused with handler entry or task-exit timing.
• Actual behavior: The test aborted inside healthy_peers(), before its cancellation/recovery assertions. Peer 6be2865fcffd-peer-0-0 entered 0.842 seconds after submission and emitted heartbeats, but its entry event took 9.011 seconds to reach CloudWatch, longer than the driver's 8-second observation window. The peer ultimately succeeded. Separately, final victim traces show no interruption, an 80-second escape, and the first wrapper returning about 20.06 seconds after its invocation deadline; the execution later failed with Sandbox.Timedout.
• Severity: Validation failure; no additional customer-defect severity is established by this collection error. Its immediate cause is the short log-observation window, not a slow or absent healthy invocation. The separately observed timeout/cleanup defect is High severity for affected workloads because it can retain capacity after timeout, including with the default executor. This aborted case does not establish the severity or correctness of the full recovery path.
• Cause and corresponding issue: The immediate error is a test-driver observation-window defect. The intended SDK contract is #726 problem 1, but the complete cancellation/healthy-peer/recovery sequence was not assessed. That supplementary evidence supports problem 1 without making this aborted case complete recovery coverage.

non-cooperative-child — assertion failure

• Scenario: On default2, start an asynchronous child that catches and ignores InterruptedException for a bounded 80-second period. Return from the root immediately. At the end of that period the child attempts child.step("late-work", ...). Maintain a healthy companion and recovery probes around the 60-second invocation deadline; the overall durable execution timeout is 240 seconds.
• Expected behavior: The SDK must attempt cancellation and finish its own cleanup within the invocation deadline plus five seconds. If the child has not exited, the outer invocation must take a bounded invocation-level failure path, not claim clean success or suspension. The child may continue its deliberate bounded loop, but further durable work must be rejected before its body/checkpoint is admitted. Healthy progress and timely recovery of the original runtime slot must still be demonstrated; Java is not required to forcibly terminate the uncooperative thread.
• Actual behavior: The child recorded no received interruption. The first SDK wrapper returned 20.164 seconds after the invocation deadline, beyond the +5-second cleanup budget, and still emitted a local SUCCEEDED response marker. A late-work body ran about 20.05 seconds after the deadline; its call subsequently raised UnrecoverableDurableExecutionException. The body ran before wrapper return, so the demonstrated boundary violation is work after the invocation deadline, not proof of work starting after wrapper return. The overall durable execution eventually became TIMED_OUT at 240 seconds. No probe entered the original environment within deadline +8 seconds while the healthy slot was still occupied. The healthy peer's task exited at about +12.799 seconds; the earliest probe entered at +13.382 seconds, before the victim returned at +20.164.
• Severity: High severity for affected workloads. The missing deadline/cancellation handling can retain runtime capacity after timeout and admit additional step bodies, creating availability risk and a risk of unexpected or duplicate downstream actions. Exposure is not limited to custom executors; slow or failing dependencies can make deadline overrun relevant to default-executor users as well. The fixture records marker bodies, so external-data corruption was not demonstrated. Later probes may use the released healthy slot and do not prove victim-slot recovery. This rating concerns the missing cancellation attempt, bounded SDK waiting and late-work guard; it does not require Java to forcibly terminate an interruption-ignoring thread.
• Cause and corresponding issue: Direct evidence for #726 problem 1, plus the missing deadline/cancellation gate addressed by proposal A and proposal B's prohibition on late SDK work. The SDK keeps waiting instead of ending its own cleanup within budget, and a late step is admitted far enough to run its body before the operation fails.

return-with-inflight-step — collection error

• Scenario: First hold a healthy invocation in the default2 JVM. Then launch a victim that starts an asynchronous step doing 1.5 seconds of work, waits until it has started, and returns its marker from the root before that step finishes.
• Expected behavior: The required in-flight step must drain or be cancelled before a clean outer response, within the cleanup budget. The execution must return its own marker, the healthy companion must remain usable, and no new SDK checkpoint/poll activity may continue after the closing boundary.
• Actual behavior: The intended return-with-an-active-1.5-second-step victim was never launched. Its prerequisite healthy anchor, 8fe12f1c76a5-healthy, entered 94.167 seconds after submission, beyond the 25-second admission observation budget, then succeeded. At submission, both runtime slots in that JVM were occupied by earlier timeout and stubborn invocations.
• Severity: High severity for the observed downstream availability problem, not an independently confirmed return-path defect. Earlier stuck invocations delayed a healthy request by 94.167 seconds before handler entry. Similar delays could violate customer latency objectives and cause retries in clients with shorter timeouts. The intended return-with-in-flight-work handler was never launched, so that specific cleanup path remains unassessed.
• Cause and corresponding issue: This is downstream capacity interference from the earlier #726 problem 1 cleanup/deadline failures, combined with lack of quiescence between fault cases. It is not a direct reproduction of the target return/drain path because that root handler never started.

failure-with-inflight-step — collection error

• Scenario: First establish a healthy companion in the default2 JVM. Then launch a victim that starts a 1.5-second asynchronous step and deliberately throws from its root while that step is still active.
• Expected behavior: The intended root failure must produce the expected failed durable outcome. Invocation-owned work must drain or cancel within budget, required checkpoint/poll cleanup must still be attempted, and the healthy invocation must remain usable. The failure path must not admit later SDK work after the invocation closes.
• Actual behavior: The deliberately failing root and its asynchronous step were never launched. Healthy anchor e287198ab7b1-healthy entered 73.234 seconds after submission and ultimately succeeded. The same earlier timeout/stubborn invocations occupied both runtime slots when it was submitted.
• Severity: High severity for the observed downstream availability problem, not an independently confirmed failure-path defect. A healthy request was delayed by 73.234 seconds before entry while earlier fault invocations occupied the environment. The intended failing root was never launched; cleanup after a root throws, preservation of its error, and interrupted-drain checkpoint shutdown cannot be rated as separately demonstrated defects from this case.
• Cause and corresponding issue: Again, an indirect consequence of #726 problem 1, with cross-case interference preventing the intended failure-cleanup test from being established.

warm-repeated-batches — assertion failure

• Scenario: Use the warm default2 environment for three batches of a successful execution, an intentionally failed execution, and a wait/replay execution. Capture invocation-owned task/root counts, executor queue depth and JVM thread counts after the executions.
• Expected behavior: After earlier fault work has reached bounded quiescence or has been accounted for separately, each completed warm execution must leave its invocation-owned tasks and roots at baseline, with no accumulated queued work. The final JVM thread count must remain within the test's +16-thread tolerance, allowing cached-pool idle retention. Replay must preserve checkpointed results and failures. The baseline must not mistakenly attribute another case's still-running task to the warm loop.
• Actual behavior: Failing snapshots contain liveTasks=1, liveRoots=0, queued=0. Matching task enter/exit events by JVM sequence attributes the outstanding task to stubborn-child from earlier execution marker 4ec47272176b-victim. It was not created by the warm batch. The thread-growth assertion occurs after this failing check and was not executed.
• Severity: No independently established warm-loop leak severity. The related cleanup/deadline problem is High severity, but this snapshot attributes work from an earlier stubborn child to the warm case. It does not show that the warm loop leaks its own tasks, accumulates queued work, or grows the thread pool. A deliberately non-cooperative child may remain alive for its bounded escape period even with correct bounded SDK cleanup. Establish quiescence or an invocation-specific baseline before assigning a severity to a separate warm-loop resource leak.
• Cause and corresponding issue: This is residual activity associated with the earlier #726 problem 1 scenario, exposed by a global counter without an isolated warm-test baseline. Proposal A's invocation-owned task accounting is relevant to attributing the work correctly.

default8-suspend-cleanup-replay — assertion failure

• Scenario: Repeat the successful-step/failed-step/wait/replay victim with a two-second root finally delay on the function configured for up to eight concurrent invocations per JVM. This cleanup scenario uses one healthy companion in the same JVM.
• Expected behavior: The root and its cleanup must exit before normal PENDING return, while the companion continues progressing. Replay must use stored successes/failures and stable operation identities. Full eight-invocation overlap is established separately by default8-isolation; this scenario tests the cleanup contract on that configuration.
• Actual behavior: For def38ad21146-victim-0, normal PENDING was returned at sequence 267, while root exit was sequence 273, 2.000 seconds later. The successful/failed-step replay sequence and healthy companion progress passed before the ordering failure.
• Severity: Medium generally; potentially High when cleanup protects critical resources or state. The same response/cleanup ordering defect is reproduced on the concurrency-8 configuration. Customer-visible harm depends on cleanup duration and resource ownership. This case uses one healthy companion and does not demonstrate eight simultaneous cleanup races or actual cross-execution data corruption.
• Cause and corresponding issue: A second direct reproduction of #726 problem 2, with the function configured for up to eight concurrent invocations per environment.

default8-timeout-recovery — collection error

• Scenario: Run the 80-second interruptible asynchronous-step victim on the concurrency-8 function with a 60-second invocation timeout. Admit seven healthy companions into the same JVM and keep their slots occupied while probing recovery of the affected slot.
• Expected behavior: All seven companions must be established before interpreting the timeout assertions. The SDK must cancel the victim, bound actual task exit and its outer return by deadline plus five seconds, preserve healthy progress, and restore admission in the original environment by deadline plus eight seconds. Ultimately all configured slots must be usable again. Delayed log ingestion must not itself be interpreted as an invocation that never entered.
• Actual behavior: All seven healthy peers entered approximately 1.809–2.087 seconds after submission and produced heartbeats, but their entry events reached CloudWatch 8.744–9.022 seconds later. The driver failed its 8-second peer-observation window and never reached the formal cancellation/recovery assertions. The peers later succeeded; the victim was still RUNNING in the collected durable-execution snapshot.
• Severity: Validation failure; no additional customer-defect severity is established by this collection error. The peers ran and succeeded; the driver failed to observe their logs in time. The targeted timeout/cleanup problem is High severity in affected workloads, but this case did not reach its cancellation/recovery assertions. Neither the log-ingestion delay nor the victim's RUNNING durable status establishes a separate concurrency-8 availability defect.
• Cause and corresponding issue: Immediate cause: the same log-observation defect as the concurrency-2 case. The test targets #726 problem 1, but this run does not complete that contract's validation at concurrency 8.

fixed-two-roots — assertion failure

• Scenario: Use two concurrent Lambda invocations sharing a fixed two-thread SDK executor. Synchronize both root handlers at a barrier, then have each synchronously wait for its own step. An eight-second test-only escape can expand the pool so the fixture cannot hang indefinitely.
• Expected behavior: Under the executor progress contract asserted by this test, both steps and executions must complete within the eight-second progress budget using the original two-thread executor. Completion must not depend on expanding the pool or replacing the environment; using the escape is a regression failure.
• Actual behavior: Two synchronized roots occupy both workers of the shared fixed executor and synchronously wait for their steps. The pre-escape snapshot is 2 active workers, 2 queued tasks, pool size 2. Neither queued step can run. After eight seconds, the test-only escape increases executor capacity and the executions finish; the use of that escape is the failing assertion.
• Severity: High severity for affected custom bounded-executor configurations. Once blocking roots occupy the available pool, requests cannot make business progress and subsequent work sharing that executor can also stall. The exposure is narrower than the default-executor lifecycle issues: the default cached executor avoids this particular fixed-capacity deadlock. Success after the test expands the pool does not reduce the severity under the original configuration; customer adoption of affected executors is not yet measured.
• Cause and corresponding issue: Direct reproduction of #726 problem 3. Blocking roots and the work they await consume the same finite executor capacity. Proposal C addresses that dependency cycle.

fixed-nested-map-parallel — assertion failure

• Scenario: Use the same two-invocation/shared-two-thread setup. After each root's first ordinary success step, run child-context orchestration containing map and parallel branches, whose handlers also await step work. Retain the eight-second escape for test termination.
• Expected behavior: The roots, first steps, child contexts, map/parallel coordinators and branch steps must progress under the original executor contract within the budget, without an escape. To independently demonstrate nested starvation, the nested contention point must be reached while that original capacity is still in effect.
• Actual behavior: The first ordinary success steps already stall behind the two root handlers: 2 active workers, 2 queued tasks, pool size 2. Escape events occur at sequences 10/12. The nested child bodies only start at 25/26, after executor capacity has been increased.
• Severity: High severity for the demonstrated root-stage deadlock; independent nested-starvation severity remains unestablished. Requests cannot reach their nested workflow under the original bounded executor. Child/map/parallel work starts only after the escape changes capacity, so this run cannot establish that nested orchestration is safe or independently defective after a root-only fix. The confirmed High rating maps to the same bounded-executor availability defect as fixed-two-roots, not an additional demonstrated nested defect.
• Cause and corresponding issue: The observed failure directly maps to the root-starvation part of #726 problem 3. The further nested-orchestration concern in proposal C is relevant but not independently established by this run.

Evidence limits that apply across these cases

• The concrete task/body delays are bounded observations from the fixtures, not proof that a particular production thread ran forever.
• No case here establishes the separate problem-1 subclaim that interruption during task draining skips checkpoint/poll shutdown.
• late-work ran after the invocation deadline; its later rejection does not prove an SDK admission guard worked. These records do not show a new operation starting after the SDK wrapper's recorded return, nor do they prove whether an already-issued remote checkpoint committed after that boundary.
• The passing replay/isolation checks provide no observed completed-step duplication or cross-execution result corruption. Potential external side effects and user-facing latency consequences above are distinguished from the marker-based effects and timings actually measured.

Implications for the PR

1. The red result is meaningful, but not every red case is a validated SDK regression. The two delayed-cleanup cases establish early PENDING; the fixed-pool cases establish root/step starvation; the non-cooperative case supplies deadline, cancellation-attempt, and late-body evidence.
2. Separate log-delivery deadlines from invocation-lifecycle deadlines. The two peer-admission errors are explained by measured CloudWatch ingestion latency. Extend or otherwise improve evidence observation while keeping the actual in-JVM cleanup/recovery deadlines unchanged.
3. Isolate cases after timeout faults. Earlier executions continued/reinvoked and occupied both slots when the later in-flight cases were submitted. The intended victims were never launched. Establish bounded quiescence of previous executions/tasks before interpreting those cases or the warm-loop resource checks.
4. Give nested starvation an independent oracle. Nested child/map/parallel work must be exercised under the original executor contract before the escape can alter capacity; the current root step fails first.

These results support several findings in #726, but the four collection errors, cross-case-contaminated warm check, and nested-coverage limitation still need to be resolved before treating this run as complete acceptance evidence for #727.