Bug report
Since #6491 (0.43.0), Chain::block_pointer_from_number checks chain_store.block_ptrs_by_numbers before calling the RPC. If the block isn't in recent_blocks_cache, that lookup goes to blocks_from_store_by_numbers, which calls pool.get_permitted(). So it waits for the indexing semaphore (pool_size + GRAPH_EXTRA_QUERY_PERMITS), with no timeout.
The index-node resolver wraps the lookup in BLOCK_HASH_FROM_NUMBER_TIMEOUT (10s). When indexing holds every permit, blockHashFromNumber returns null and the RPC is never tried. 0.41.1 went straight to the RPC.
indexer-agent reports this as Failed to query graph node for blockHashFromNumber: no data returned (IE070), which blocks epoch resolution and POI closes. Epoch start blocks are hours old, so they always take the store path.
Repro: v0.45.0 against 0.41.1, pool_size = 2, mainnet and gnosis ingestors. To exhaust the permits I held LOCK TABLE public.ethereum_networks IN EXCLUSIVE MODE. Both ingestors then wait in the chain head UPDATE while holding their permit, which I confirmed in pg_stat_activity. It's a stand-in for write load.
| build |
block 25961537 (not in recent cache) |
recent block |
| 0.41.1 |
hash, 0.08s |
hash, 0.12s |
| 0.45.0 |
null, 10.04s |
hash, 0.02s |
| 0.45.0 with the cache check removed |
hash, 0.08s |
hash, 0.08s |
store_semaphore_wait_ms doesn't show it. The wait is recorded only once the permit is acquired, and this lookup is cancelled before that (the metric read 52 during the null).
GRAPH_EXTRA_QUERY_PERMITS=4 does return the hash, in 5.11s, because the lookup times out on the pool connection and falls through to the RPC. That timeout calls state_tracker.mark_unavailable though, so it isn't a good workaround.
Suggested fix: drop the cache check from block_pointer_from_number, as #6537 did for is_on_main_chain, because the block cache isn't finality-aware here either. The callers are the index-node resolver, the registrar on deploy/graft and the public POI fallback, none of them hot. Still present on master (6838f4e).
Can open a PR with that if the direction sounds right?
Relevant log output
WARN Failed to fetch block hash from number, error: deadline has elapsed, block_number: 25961537, chain: ethereum, network: mainnet, component: IndexNodeServer > IndexNodeResolver
INFO Query timing (GraphQL), block: 0, query_time_ms: 10002, variables: null, query: { version { version } blockHashFromNumber(network: "mainnet", blockNumber: 25961537) } , subgraph_id: indexnode, component: IndexNodeServer
IPFS hash
No response
Subgraph name or link to explorer
No response
Some information to help us out
OS information
macOS
Bug report
Since #6491 (0.43.0),
Chain::block_pointer_from_numbercheckschain_store.block_ptrs_by_numbersbefore calling the RPC. If the block isn't inrecent_blocks_cache, that lookup goes toblocks_from_store_by_numbers, which callspool.get_permitted(). So it waits for the indexing semaphore (pool_size + GRAPH_EXTRA_QUERY_PERMITS), with no timeout.The index-node resolver wraps the lookup in
BLOCK_HASH_FROM_NUMBER_TIMEOUT(10s). When indexing holds every permit,blockHashFromNumberreturnsnulland the RPC is never tried. 0.41.1 went straight to the RPC.indexer-agent reports this as
Failed to query graph node for blockHashFromNumber: no data returned(IE070), which blocks epoch resolution and POI closes. Epoch start blocks are hours old, so they always take the store path.Repro: v0.45.0 against 0.41.1,
pool_size = 2, mainnet and gnosis ingestors. To exhaust the permits I heldLOCK TABLE public.ethereum_networks IN EXCLUSIVE MODE. Both ingestors then wait in the chain headUPDATEwhile holding their permit, which I confirmed inpg_stat_activity. It's a stand-in for write load.null, 10.04sstore_semaphore_wait_msdoesn't show it. The wait is recorded only once the permit is acquired, and this lookup is cancelled before that (the metric read 52 during the null).GRAPH_EXTRA_QUERY_PERMITS=4does return the hash, in 5.11s, because the lookup times out on the pool connection and falls through to the RPC. That timeout callsstate_tracker.mark_unavailablethough, so it isn't a good workaround.Suggested fix: drop the cache check from
block_pointer_from_number, as #6537 did foris_on_main_chain, because the block cache isn't finality-aware here either. The callers are the index-node resolver, the registrar on deploy/graft and the public POI fallback, none of them hot. Still present on master (6838f4e).Can open a PR with that if the direction sounds right?
Relevant log output
IPFS hash
No response
Subgraph name or link to explorer
No response
Some information to help us out
OS information
macOS