Skip to content

Stable 25.3.14 - CI fixes: job timeouts, TSan test timeouts, and stale report after rerun - #2410

Merged
MyroTk merged 4 commits into
releases/25.3.14from
ci/25.3.14-ci-fixes
Sep 20, 2026
Merged

MyroTk merged 4 commits into
releases/25.3.14from
ci/25.3.14-ci-fixes

Conversation

@CarlosFelipeOR

@CarlosFelipeOR CarlosFelipeOR commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator
  • 00988_expansion_aliases_limit — cherry-pick of ClickHouse#84067: SET max_expanded_ast_elements = 10000. The test was hitting the 600s limit under TSan.
  • 02354_vector_search_*no-tsan tag on both tests, ported by hand from ClickHouse#84510. Same problem: 11 timeouts in the last 45 runs of that shard.
  • Integration job timeouts (tests/ci/ci_config.py) — asan 2h→4h, asan old analyzer 3h→4h, tsan 3h→4h, release 2h→3h. asan was the only sanitizer job still on the default and hit the ceiling in 10.2% of runs.
  • Report showing failures that already passed on rerun (create_workflow_report.py) — anchor CTE per check_name from Antalya 26.1 - Fix report old results #1572 and Antalya-26.6 - fix integration test fails being hidden in report #2142, with STARTED/COMPLETED in the guard instead of Pre Hooks/Post Hooks.

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

@CarlosFelipeOR CarlosFelipeOR added cicd Improvements and fixes to the CICD process 25.3 25.3.14 25.3.14.10001 labels Sep 20, 2026
alexey-milovidov and others added 4 commits September 20, 2026 16:03
(cherry picked from commit a8860b0)
Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
Ported by hand from upstream ClickHouse#84510
(0cba6f5), not cherry-picked: the upstream
commit also drops `SET max_execution_time = 600;`, which does not exist on this
branch, and the neighbouring context differs
(`allow_experimental_vector_similarity_index` here vs
`enable_vector_similarity_index` upstream). Only the tag change applies.

Both tests sit at the 600s ceiling on `Stateless tests (tsan) [6/6]`:
02354_vector_search_expansion_search timed out in 11 of the last 45 runs of
that shard and averages 524s on release runs.

Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
…or release

Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
Ports the anchor CTE from #1572 and #2142, keeping only rows at or after the
latest result batch per check_name. The guard excludes STARTED/COMPLETED instead
of praktika's Pre Hooks/Post Hooks, since COMPLETED lands after the tests on the
old CI and would otherwise hide every failure.

Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
@MyroTk
MyroTk merged commit 5dc76dc into releases/25.3.14 Sep 20, 2026
200 of 214 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

25.3 25.3.14 25.3.14.10001 cicd Improvements and fixes to the CICD process

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants