02435_rollback_cancelled_queries fails 1 run in 4 on the ASan + CAS-S3 lane — INSERTs outlive their clients by more than 15 s

[CarlosFelipeOR]

Describe the situation

02435_rollback_cancelled_queries runs three threads for 20 seconds — one inserting, one selecting, one killing clients mid-query. When they stop, the test waits for the server to go quiet. Some INSERTs keep running for more than 15 seconds after that, so the test dumps system.processes and the output no longer matches the reference.

It needs two things at once. ASan alone never fails it. CAS alone almost never does. Together they fail it a quarter of the time:

Lane	Failures	Passes	Rate	Query
ASan + CAS-S3	20	25	25.6%	list the 20
CAS without ASan	2	142	1.3%	list the 2
ASan without CAS	0	359	0%	returns nothing
everything else	1	2,026	0%	list the 1

Upstream CI runs the same test 122,907 times in 60 days and fails it once — 0.001%. Our rate on that lane is about 25,000 times higher.

This issue:

• Has run for six weeks: first failure 2026-08-04, latest 2026-09-14.
• Is not the memory ceiling of CAS-S3 ASan stateless hits MEMORY_LIMIT_EXCEEDED with cas_gc_interval_sec 60 #2299. Several failing jobs carry zero Code: 241 — 7515aa5ae had 21 failures and none, cc9034ea1 had 16 and none.
• Reproduces within a run: the harness reports Failed 3 out of 3 reruns.

---

Actual behavior

The leftover rows in the diff come from system.processes, not from the query log. tests/queries/shell_config.sh:189 polls for 15 seconds, then dumps whatever is still running:

@@ -1,2 +1,48 @@
 1000000
 0
+Row 1:
+──────
+is_initial_query:      1
+query_id:              02435_insert_default_14483-25065-2
+is_cancelled:          0
+memory_usage:          61360603
...

is_cancelled: 0 on every leftover row. These INSERTs were never cancelled — they're simply still going.

All 23 failures, with job and report links: query. Most recent in a release run: 34481990917.

---

Root cause analysis

⚠️ This section is AI analysis (Claude, via Claude Code), from the CI database and the test source. The rates and the log excerpt are checkable from the links. The mechanism is not established.

Both ingredients slow writes down. CAS writes go to object storage, and ASan slows everything further. An INSERT that was mid-flight when its client died then needs more than 15 seconds to wind down, and the test's wait expires first.

What that guess doesn't explain is Failed 3 out of 3 reruns. Plain slowness should vary between attempts. Failing every rerun points at a run reaching a state and staying in it, which is a different shape.

Two questions would settle it:

• Are those INSERTs making progress, or are they stuck? system.processes gives elapsed and written_rows over successive polls.
• Does raising the wait past 15 seconds turn the failures green? If it does, this is lane slowness and belongs in the test. If it doesn't, an INSERT isn't finishing.

---

Additional context

• Job: Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2)
• Branch: antalya-26.6
• First seen: 2026-08-04 · Latest: 2026-09-14

[filimonov]

Root cause

The slow INSERTs are not stuck and not cancelled. They are running, but every small function on a "polluted" thread is about 100x slower, because ASan's fake stack on that thread is full. Two things combine:

1. The fork parses every listed S3 key as JSON. In src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h the string constructor of RelativePathWithMetadata always builds CommandInTaskResponse, which runs Poco::JSON::Parser::parse on the string inside try/catch. This exists so that the swarm task distributor can answer a replica with {"retry_after_us": N} in the path field, and the receiver tells a command from a path by trying to parse it. But this is the base listing type at the disk layer, so every key returned by listObjects / iterate of any object storage (S3, Azure, HDFS, local, ObjectStorageQueue, Iceberg, DeltaLake) throws and catches one JSONException. Upstream's constructor only assigns the path; CommandInTaskResponse does not exist upstream. The code came from the Iceberg metadata work in Antalya 25.6 (#1019), was meant to be removed by #1360 (not merged), and reached 26.6 through #1414 and #1687.

2. ASan never frees the fake frame of a function that exits by exception and is called again at the same stack depth. A fake frame is released in the function epilogue or by FakeStack::GC, which runs at the next fake-frame allocation and frees only frames strictly below that point. An exception skips the epilogue, and parseImpl is called again for the next key at exactly the same depth, so its previous frame is never collected. One leaked 128-byte frame (size class 1) per listed key, per thread, for the life of the process. When the class holds 8192 live frames, every __asan_stack_malloc_1 call scans all 8192 flags and fails. That is the 100x slowdown of every function with a 65..128-byte frame: sort comparators, PODArray::operator[], IColumn::assertTypeEquality, and so on.

Evidence

• CI, PR CAS: gc scheduler stop join race #2326, shard 2/2, trace_log symbolized with the job's binary: 16579 of 59746 CPU samples (27.7%) of the whole server run are in __asan_stack_malloc_1, on 197 threads and 990 queries. Only size class 1 is affected.
• The three slow 02435 inserts (sort 205 s instead of 2 s) ran their sort on threads 278130, 278308, 278286; the fast ones on neighbouring pool threads. The very first insert of the test was already slow, so nothing inside the test causes it.
• Local ASan build of the lane config, server started as a child of gdb, fake stacks read from memory per thread: 109 s after start, without load, 1215 live class-1 frames, all with pc = Poco::JSON::ParserImpl::parseImpl. After 8 minutes of clickhouse-test --cas-s3-storage -j 8: 67557 live frames, all parseImpl, one pool thread at 8192/8192. A breakpoint on parseImpl fired 224 times in 15 seconds, every time from RelativePathWithMetadata::CommandInTaskResponse called by the RelativePathWithMetadata constructor for keys returned by S3IteratorAsync::getBatchAndCheckNext.
• Standalone reproduction, 40 lines, system unwinder, clang++-21 -O1 -fsanitize=address: a function with an escaping 96-byte local throws, the immediate caller catches, repeat. Live frames in its size class: 3001 after 3000 throws, 4096/4096 after 6000. (The local must escape, otherwise clang's stack-safety analysis emits no fake frame and the toy shows nothing.)

Why only this lane, and why since August

Three things must coincide: ASan, steady object-storage listing, and enough keys per pool thread to fill 8192 slots. A plain MergeTree on an s3 disk almost never lists S3 (metadata is local). The fork's other ASan stateless lanes run on local disk and do not list at all; the s3-storage stateless lanes run under tsan, debug and coverage, not ASan; upstream has no amd_asan_ubsan + s3 storage stateless lane either, so its 1-in-122907 rate says nothing about this combination. CAS lists as part of its protocol (GC fold every 20 s, namespace janitor, orphan-manifest sweep, ref-table recovery, part folder listing), about 140 keys per second on the local rig, always on the same long-lived pool threads. The lane amd_asan_ubsan, cas s3 storage was added on 2026-08-05; the first failure here is dated 2026-08-04, the first run of the lane.

This also explains the other observations: Failed 3 out of 3 reruns (the state lives in pool threads for the life of the process), is_cancelled: 0 with inserts alive 200 s after the clients died (they are in ipnsort under fake-stack scans), and first slowdowns spread over the whole run with no link to any query type.

Fix

Temporary workaround in #2393: CommandInTaskResponse only tries to parse strings that start with { after whitespace. Object keys never do, the distributor answer always does, so the swarm protocol is unchanged. This removes the leak and the thrown-and-caught exception per listed key on every object storage, ASan or not. No detection loss: detect_stack_use_after_return stays on. Expected effect on the amd_asan_ubsan, cas s3 storage lane: __asan_stack_malloc_1 disappears from the profile and 02435 sorts 1M rows in about 2 s again.

Proper fix, to be done by the swarm owners: take the command out of the path field, as a separate ObjectInfo type or field that only the task distributor creates and only StorageObjectStorageSource::next / ClusterFunctionReadTask read, or send retry_after_us through the versioned cluster-function protocol like the Iceberg metadata already is (the direction of #1360). Then generic listing never sees commands, a key that happens to be valid JSON can no longer be silently rewritten into its file_path, and the probe in #2393 can be removed.

Two follow-ups outside this issue:

• Check the fork for the same pattern: any try { parse(x) } catch used as a format probe in a per-item loop leaks one fake frame per item under ASan.
• Optional upstream change for compiler-rt: FakeStack::GC could treat frames at the same depth as the new allocation as dead (<= instead of <); a frame that was live at the position now being allocated cannot still be live. The standalone reproduction above is enough to file it.

ASAN_OPTIONS=detect_stack_use_after_return=0 or max_uar_stack_size_log would only hide this by turning off or shrinking the detector; they are not needed.

See also: https://gist.github.com/filimonov/62eb579bac67be399ee403408a37d1ee

ASAN report: llvm/llvm-project#224084

[filimonov]

Workaround applied in #2393

Final solution - tech debt for @ianton-ru