Fix broadcast polling query plans

[cardmagic]

Summary

Split broadcast claiming into separate pending-delivery and stale-processing
recovery probes. Each probe can follow the existing
(status, available_at, id) polling index. Both candidates are locked in the
same short transaction with FOR UPDATE SKIP LOCKED, and the executor chooses
the earlier (available_at, id) candidate, preserving the previous global
ordering.

No schema migration or new index is included. The existing polling indexes are
the correct indexes for effects, reminders, and both broadcast probes. Another
index would add write cost without improving the measured plans. Existing
installations receive the query change by upgrading the gem, and schema coverage
now protects those canonical indexes.

Production evidence

LEADx Web App on MySQL reported these Solid Objects polling spans on 2026-09-03
around 06:00 UTC:

Query	Duration
SolidObjects::Effect Load	1187.12 ms
SolidObjects::Broadcast Load	1229.03 ms
SolidObjects::Reminder Load	1251.63 ms

The broadcast query combined pending delivery and stale recovery with an OR:

WHERE (status = 'pending' AND available_at <= ?)
   OR (status = 'processing' AND claimed_at <= ?)
ORDER BY available_at, id
LIMIT 1
FOR UPDATE SKIP LOCKED

The effect and reminder queries already led with their matching ordered
indexes. The similar production durations across three different tables are
not explained by those two query plans. They are consistent with a shared
database wait or contention event, but the supplied spans do not identify the
specific production wait. This change therefore fixes the independently
reproduced broadcast planner defect and does not pretend an extra effect or
reminder index would cure shared database contention.

Query plans

I loaded 50,000 production-shaped rows per table and ran EXPLAIN ANALYZE on
MySQL 8.4 and PostgreSQL 18. SQLite 3.53.2 was checked with
EXPLAIN QUERY PLAN.

Before

Effects and reminders used their intended ordered range scans:

Database	Effect	Reminder
MySQL 8.4	idx_so_effects_poll, 0.639 ms	idx_so_reminders_due, 0.012 ms
PostgreSQL 18	polling index scan, 0.030 ms	due index scan, 0.015 ms
SQLite 3.53.2	covering polling-index search	due-index search

The broadcast OR gathered every eligible row before applying the limit:

• MySQL: range scan returned 2,000 rows, then Sort ... limit input to 1;
  4.1 ms.
• PostgreSQL: BitmapOr and bitmap heap scan returned 2,000 rows, followed by
  quicksort; 1.101 ms and 1,047 shared-buffer hits.
• SQLite: MULTI-INDEX OR followed by USE TEMP B-TREE FOR ORDER BY.

After

The executor issues two ordered probes:

WHERE status = 'pending' AND available_at <= ?
ORDER BY available_at, id
LIMIT 1
FOR UPDATE SKIP LOCKED

WHERE status = 'processing' AND claimed_at <= ?
ORDER BY available_at, id
LIMIT 1
FOR UPDATE SKIP LOCKED

• MySQL: both use idx_so_broadcasts_poll, return one row directly, and take
  0.091 ms plus 0.037 ms.
• PostgreSQL: both use ordered polling-index scans, return one row directly,
  and take 0.069 ms plus 0.020 ms.
• SQLite: both use idx_so_broadcasts_poll; neither uses a temporary sort.

Correctness and compatibility

• Preserves pending eligibility and stale-processing recovery thresholds.
• Preserves global (available_at, id) ordering across the two states.
• Preserves FOR UPDATE SKIP LOCKED on PostgreSQL, mysql2, and Trilogy.
• Keeps locks and the state transition in one transaction.
• Keeps retries and at-least-once broadcast delivery unchanged.
• Adds a synchronized two-executor test proving different rows are claimed
  while the first delivery is blocked.
• Adds recovery coverage proving an older stale broadcast wins over newer
  pending work.
• Adds SQL-shape coverage rejecting a combined OR poll and schema coverage
  for all three canonical polling indexes.
• Makes no public API, authorization, storage-format, or schema change.

TDD evidence

Red, before the implementation:

BroadcastsTest#test_polls_pending_and_stale_broadcasts_separately
Expected: 2
  Actual: 1
1 runs, 2 assertions, 1 failures, 0 errors, 0 skips

Green, after the implementation:

1 runs, 8 assertions, 0 failures, 0 errors, 0 skips

Validation

mise exec ruby@4.0.5 -- bundle exec rake
611 runs, 1983 assertions, 0 failures, 0 errors, 15 skips
Standard: pass
RuboCop: 229 files, no offenses
RBS/Steep: no type errors
Brakeman: 0 security warnings

SOLID_OBJECTS_DATABASE_URL=postgresql://solid_objects:solid_objects@127.0.0.1:5433/solid_objects_polling \
  mise exec ruby@4.0.5 -- bundle exec rake test
611 runs, 1967 assertions, 0 failures, 0 errors, 15 skips

SOLID_OBJECTS_DATABASE_URL=mysql2://solid_objects:solid_objects@127.0.0.1:3307/solid_objects_polling \
  mise exec ruby@4.0.5 -- bundle exec rake test
611 runs, 1947 assertions, 0 failures, 0 errors, 24 skips

SOLID_OBJECTS_DATABASE_URL=trilogy://solid_objects:solid_objects@127.0.0.1:3307/solid_objects_polling \
  mise exec ruby@4.0.5 -- bundle exec rake test
611 runs, 1947 assertions, 0 failures, 0 errors, 24 skips

[greptile-apps]

Greptile Summary

The PR replaces the combined broadcast polling query with separate pending and stale-processing probes while preserving eligibility and global delivery ordering.

• Locks the first candidate from each status-specific probe in one transaction and claims the earlier (available_at, id) record.
• Adds integration coverage for stale recovery ordering, concurrent claims, and generated SQL shape.
• Adds schema coverage for the canonical effect, broadcast, and reminder polling indexes.
• Updates the generated RBS signature for the extracted candidate-claim helper.

Confidence Score: 5/5

The PR appears safe to merge with no concrete blocking or independently actionable non-blocking issue identified.

The split probes preserve the existing eligibility predicates, compare non-null ordering fields consistently, retain transactional claim locking, and are covered by ordering and concurrency tests.

Important Files Changed

Filename	Overview
lib/solid_objects/broadcast_executor.rb	Splits broadcast polling into two ordered locked probes and selects the globally earlier candidate without changing eligibility or claim ownership semantics.
sig/generated/lib/solid_objects/broadcast_executor.rbs	Adds the generated signature for the new relation-based candidate helper.
test/integration/broadcasts_test.rb	Covers cross-status ordering, concurrent executor claims, and the absence of a combined OR polling query.
test/models/schema_constraints_test.rb	Verifies that all three durable polling tables retain their expected delivery-order indexes.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Begin claim transaction] --> B[Probe earliest pending broadcast]
  B --> C[Probe earliest stale processing broadcast]
  C --> D{Any candidate?}
  D -->|No| E[Return without work]
  D -->|Yes| F[Choose minimum available_at and id]
  F --> G[Update selected row to processing]
  G --> H[Commit transaction]
  H --> I[Deliver outside claim transaction]

Loading

_{Reviews (1): Last reviewed commit: "fix: split broadcast polling queries" | Re-trigger Greptile}