Perf: Optimize performance of the get rows endpoints

[Koc]

This PR built on top of #2238.
Closes #1490.

It fixes loading of the large tables. Right now it is not possible to load a table with 30k rows and 8 columns due to an error:

"Previous":{"Exception":"Doctrine\\DBAL\\Exception\\DriverException","Message":
"An exception occurred while executing a query: SQLSTATE[HY000]: 
General error: 7 number of parameters must be between 0 and 65535",

PR contains 2 commits: fix itself and performance improvement.

I've measured latency for a various scenarios for /row/table/{tableId} endpoint:

Scenario	master	#2238	1st commit from this PR	2nd commit from this PR
8 columns x 2k rows	0.68s	0.31s	0.31s	0.17s
8 columns x 10k rows	2.86s	1.21s	1.21s	0.44s
8 columns x 60k rows	🐞 sql error	🐞 sql error	6.97s	2.27s

📹 Video for performance comparison

Next step: move filtering/sorting/pagination to BE side and speedup FE. But this is completely separate topic which will be implemented in upcoming PRs.

[enjeck]

if all comments are addressed, can be merged after conflict is fixed?

[Koc]

@enjeck I don't see any blockers. I did the rebase, fixed all conflicts, tested manually few CRUD operations and upgrade - everything works for me

[blizzz]

@Koc loading views gives me a 500, likely related to them containing a meta column.

[blizzz]

since we have two updates that are related, they should be bundled in a transaction, as you pointed out. OCP\AppFramework\Db\TTransactional->atomic() can be used for convenience.

Also valid for insert logic, not just update.

[Koc]

added atomic() + SELECT FOR UPDATE for databases that support it

[blizzz]

Other yet unverified findings; unfold to read.

Potentially Blocking

B1 — Lost update on concurrent edits

Row2Mapper::update() is a read-modify-write of the whole blob: read at line 748, cell writes in
between, write back at 760-761. No lock, no version check (QBMapper::update() is a plain
UPDATE … WHERE id = ?).

Nextcloud pins the primary connection to READ COMMITTED
(OC\DB\SetTransactionIsolationLevel::postConnect()), so a transaction alone does not fix this:

T1  BEGIN; SELECT cached_cells → {5:"a", 6:"x"}
T2  BEGIN; SELECT cached_cells → {5:"a", 6:"x"}
T1  write cell col 5; UPDATE … SET cached_cells = {5:"b", 6:"x"}; COMMIT
T2  write cell col 6; UPDATE … SET cached_cells = {5:"a", 6:"y"}; COMMIT

Final cache {5:"a", 6:"y"} — T1's change to column 5 is gone from the cache while
tables_row_cells_text correctly holds "b". Both committed, neither errored. MySQL REPEATABLE
READ does not help either (non-locking snapshot reads); only SERIALIZABLE would, and NC does not
run it.

Fix: re-read the sleeve with IQueryBuilder::forUpdate() inside the transaction before line 748
(@since 33.0.0, matches the app's min-version). Needs a RowSleeveMapper::findForUpdate().
Alternative: optimistic version column with retry on 0 affected rows.

Collaborative editing is the point of this app, so this is reachable in normal use.

B2 — No reconciliation path

Once the cache diverges there is no way back. CacheSleeveCells only picks up rows
WHERE cached_cells IS NULL, which a stale row is not. There is no occ command to rebuild.
Given B1 and M4/M5 below, ship a rebuild command with this.

---

Architecture — worth settling before merging

The join in the old query is dead weight, and removing it may be most of the win.
NormalizedRowLoader::getRowsChunk() selects created_by, created_at, t1.last_edit_by, t1.last_edit_at, table_id and joins tables_row_sleeves — but parseResult() never reads any of
them. All row metadata comes from the separate findMultiple($rowIds) call, and every
formatRowData() implementation only touches value / value_type. That join forces the UNION to
be materialized into a temp table and joined per cell row. There is already an index on
(column_id, row_id) (Version000700Date20230916000000.php:82).

Dropping the join — or going further and issuing five plain indexed
SELECT row_id, column_id, value FROM tables_row_cells_X WHERE column_id IN (…) AND row_id IN (…)
queries merged in PHP, with no CAST and no UNION — would be a fraction of this diff with no
schema change, no dual-write, no migration and no consistency risk. Worth benchmarking before
accepting a denormalized copy of every cell in the database.

If the cache is kept: it roughly doubles row-data storage, and CachedRowLoader transfers the
entire row from the DB even for a view showing 2 of 50 columns (filtered in PHP at
CachedRowLoader.php:59-73) — a bandwidth/memory regression exactly for the wide tables this
targets. Per AGENTS.md this warrants a ticket for maintainer alignment first.

---

Medium

M1 — Fresh installs stay on the slow path until the next app update

live-migration steps are only queued by AppManager::upgradeApp() (AppManager.php:1156).
Installer::installAppLastSteps() runs only install / pre-migration / post-migration
(Installer.php:545,551,573) and then sets installed_version to the current version.

A fresh install therefore never flips cachingSleeveCellsComplete and getRowLoader()
(Row2Mapper.php:203) keeps returning LOADER_NORMALIZED. Writes are unaffected —
insert()/update() always populate cached_cells — so nothing is wrong, the install just runs
slow while it accumulates data. It self-heals at the next release the admin installs; indefinite if
the version is pinned.

Fix: also list the step under <install> in info.xml, or flip the flag when nothing is pending.

Separately, confirm which server version introduced live-migration support. The app declares
min-version="33"; the server commit is from Aug 2026 and only master/36 was available to check,
so it is clearly not in 33/34. Either raise min-version or handle its absence.

M2 — The backfill is an N+1 in a loop (AGENTS.md forbids this)

Three call sites, all through ColumnsHelper::getCachedCellsForRow()
(lib/Helper/ColumnsHelper.php:141-155), which runs one query per column and builds the query
builder inside the loop (RowCellMapperSuper::findManyByRowAndColumn(),
lib/Db/RowCellMapperSuper.php:129-136):

• Backfill — CacheSleeveCells.php:59-63, with find() + update() per row at lines 74-85.
  Cost is rows × (columns + 2): a 100k-row × 20-column table is ~2.2M queries in one
  BackgroundRepair run. It is at least resumable (the IS NULL filter survives a kill).
• Every edit of a row whose cache reads empty — Row2Mapper.php:748-756 (see M3).
• User migration import — TablesMigrator.php:351, per row, inside the import transaction.

Fix for all three: one query per cell table over a batch of row IDs (WHERE row_id IN (…)), which
is what the (column_id, row_id) index is for.

M3 — The rebuild also fires on ordinary empty rows

Row2Mapper.php:749 treats $cachedCells === [] as "not yet migrated", but
getCachedCellsArray() returns [] for both NULL and "[]" — i.e. any genuinely empty row.
Filling the first cell of a new row triggers findAllByTable() plus one query per column, every
time, forever. Distinguish null from [].

M4 — Rows the backfill cannot reach become silently empty

CacheSleeveCells::getTableIds() snapshots tables_tables at the start and only backfills sleeves
whose table_id matches, then sets the flag unconditionally (line 68). Any sleeve with a
NULL/orphaned table_id is skipped but still read through the cached loader, where
json_decode($sleeve['cached_cells'] ?? '{}') (CachedRowLoader.php:72) yields nothing and the row
renders with no cells.

Fix: CachedRowLoader should fall back to the normalized loader when cached_cells IS NULL rather
than returning an empty row.

M5 — importRow() writes '[]' before the cells exist

RowService.php:1026 inserts cached_cells = json_encode([]), which makes those rows invisible to
the backfill's IS NULL filter. TablesMigrator::rebuildCachedCells() covers it, but it catches
\Throwable and only logs (TablesMigrator.php:354), so a failure leaves a permanently-wrong row
with no way back. Insert NULL instead.

M6 — find() silently uses the slow loader

Row2Mapper::find() (line 89) calls getRows() without a loader argument, so it takes the
LOADER_NORMALIZED default. If deliberate, say so in a comment; as written the default parameter
reads like an oversight. It also means post-write API responses come from the normalized tables
while the grid comes from the cache — which is exactly what hides B1 from testing.

M7 — Normalized chunk size drops from 1000 to ≤200 rows

NormalizedRowLoader::getRows() computes floor(1000/5) - count($columns). Correct fix for the
placeholder explosion on main (the named :rowsIds is expanded once per UNION branch), but it is
5× more round trips in the fallback path — the only path until the background job runs, and the
permanent one in M1. Also, with ≥200 columns $chunkSize clamps to 1 and 5 × (200+1) still
exceeds the 1000 limit.

M8 — Test fixture does not match the production format

DatabaseTestCase::updateCachedCells() writes ['value' => …] for every column type, but usergroup
columns are hasMultipleValues() and production stores a list of ['value', 'value_type'] pairs.
CachedRowLoader's array_map branch (lines 65-67) is never exercised against realistic data.

---

Minor

• RowCellMapperSuper::findAllForRow() (line 142) is dead code — never called. AGENTS.md forbids it.
• CacheSleeveCells declares strict_types=1 and passes fetchAll(\PDO::FETCH_COLUMN) results
  straight into cacheCellsForRow(int $rowId). Whether PDO returns int or string is
  driver/emulation dependent; Row2Mapper gets away with the same pattern only because it has no
  strict_types. Add an explicit (int).
• findManyByRowAndColumn() documents @throws MultipleObjectsReturnedException|DoesNotExistException;
  findEntities() throws neither.
• get_class($this) replaces static::class in several new/edited error messages, inconsistent with
  the rest of Row2Mapper.
• RowSleeveMapper::findMultiple() no longer calls closeCursor().
• setCachedCellsArray(array $cachedCells):void — missing space before the return type.
• Migration named Version2230Date20260906000000 while info.xml goes to 2.3.2; the AGENTS.md
  convention gives Version2320…. Sorts fine after Version2210, so cosmetic.
• Commit subjects (Perf: Optimize performance…) are not Conventional Commits as AGENTS.md
  requires — should be perf(rows): ….
• cached_cells is added to RowSleeve::jsonSerialize(). RowSleeve is not in
  ResponseDefinitions.php or any controller, so no OpenAPI regeneration is needed — but the full
  row payload now serializes anywhere a sleeve is dumped (e.g. logs).

[Koc]

@blizzz views loading works fine for me for both cached/non-cached scenarios. Can you provide more details (error/stacktrace) or just push fix?

Also I've fixed extra code review comments except B2, M2, M6, M7 and some minior

[blizzz]

Looking more closely, the stack trace is strange. No such issues on main though. Would not rule out that there could be an artifiact on my dev setup.

Stacktrace

{
  "reqId": "BHMc5EZhqoX5MrtyYIpW",
  "level": 3,
  "time": "2026-09-16T21:08:39+00:00",
  "remoteAddr": "127.0.0.1",
  "user": "master",
  "app": "tables",
  "method": "GET",
  "url": "/index.php/apps/tables/row/view/872",
  "scriptName": "/index.php",
  "message": "An internal error or exception occurred: OCA\\Tables\\Db\\Row2Mapper - findAll: Did expect one result but found none when executing: query \"SELECT * FROM `*PREFIX*tables_columns` WHERE `id` = :dcValue1\"; ",
  "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:155.0) Gecko/20100101 Firefox/155.0",
  "version": "36.0.0.0",
  "exception": {
    "Exception": "OCA\\Tables\\Errors\\InternalError",
    "Message": "OCA\\Tables\\Db\\Row2Mapper - findAll: Did expect one result but found none when executing: query \"SELECT * FROM `*PREFIX*tables_columns` WHERE `id` = :dcValue1\"; ",
    "Code": 0,
    "Trace": [
      {
        "file": "/srv/http/nextcloud/apps-repos/tables/lib/Service/RowService.php",
        "line": 134,
        "function": "findAll",
        "class": "OCA\\Tables\\Db\\Row2Mapper",
        "type": "->",
        "args": [
          [
            -1,
            -5,
            5356,
            5360
          ],
          1465,
          null,
          null,
          [],
          [
            {
              "columnId": -5,
              "mode": "DESC"
            }
          ],
          "master"
        ]
      },
      {
        "file": "/srv/http/nextcloud/apps-repos/tables/lib/Controller/RowController.php",
        "line": 57,
        "function": "findAllByView",
        "class": "OCA\\Tables\\Service\\RowService",
        "type": "->",
        "args": [
          872,
          "master"
        ]
      },
      {
        "file": "/srv/http/nextcloud/apps-repos/tables/lib/Controller/Errors.php",
        "line": 23,
        "function": "{closure:OCA\\Tables\\Controller\\RowController::indexView():52}",
        "class": "OCA\\Tables\\Controller\\RowController",
        "type": "->",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/srv/http/nextcloud/apps-repos/tables/lib/Controller/RowController.php",
        "line": 52,
        "function": "handleError",
        "class": "OCA\\Tables\\Controller\\RowController",
        "type": "->",
        "args": [
          {
            "__class__": "Closure"
          }
        ]
      },
      {
        "file": "/srv/http/nextcloud/lib/private/AppFramework/Http/Dispatcher.php",
        "line": 198,
        "function": "indexView",
        "class": "OCA\\Tables\\Controller\\RowController",
        "type": "->",
        "args": [
          872
        ]
      },
      {
        "file": "/srv/http/nextcloud/lib/private/AppFramework/Http/Dispatcher.php",
        "line": 86,
        "function": "executeController",
        "class": "OC\\AppFramework\\Http\\Dispatcher",
        "type": "->",
        "args": [
          {
            "__class__": "OCA\\Tables\\Controller\\RowController"
          },
          "indexView"
        ]
      },
      {
        "file": "/srv/http/nextcloud/lib/private/AppFramework/App.php",
        "line": 138,
        "function": "dispatch",
        "class": "OC\\AppFramework\\Http\\Dispatcher",
        "type": "->",
        "args": [
          {
            "__class__": "OCA\\Tables\\Controller\\RowController"
          },
          "indexView"
        ]
      },
      {
        "file": "/srv/http/nextcloud/lib/private/Route/Router.php",
        "line": 324,
        "function": "main",
        "class": "OC\\AppFramework\\App",
        "type": "::",
        "args": [
          "OCA\\Tables\\Controller\\RowController",
          "indexView",
          {
            "__class__": "OC\\AppFramework\\DependencyInjection\\DIContainer"
          },
          {
            "_route": "tables.row.indexview",
            "viewId": "872"
          }
        ]
      },
      {
        "file": "/srv/http/nextcloud/lib/OC.php",
        "line": 1232,
        "function": "match",
        "class": "OC\\Route\\Router",
        "type": "->",
        "args": [
          "/apps/tables/row/view/872"
        ]
      },
      {
        "file": "/srv/http/nextcloud/index.php",
        "line": 28,
        "function": "handleRequest",
        "class": "OC",
        "type": "::",
        "args": []
      },
      {
        "file": "/srv/http/nextcloud/lib/OC.php",
        "line": 1397,
        "function": "{closure:/srv/http/nextcloud/index.php:25}",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/srv/http/nextcloud/index.php",
        "line": 25,
        "function": "handleRequests",
        "class": "OC",
        "type": "::",
        "args": [
          {
            "__class__": "Closure"
          }
        ]
      }
    ],
    "File": "/srv/http/nextcloud/apps-repos/tables/lib/Db/Row2Mapper.php",
    "Line": 195,
    "message": "An internal error or exception occurred: OCA\\Tables\\Db\\Row2Mapper - findAll: Did expect one result but found none when executing: query \"SELECT * FROM `*PREFIX*tables_columns` WHERE `id` = :dcValue1\"; ",
    "exception": "{\"class\":\"OCA\\Tables\\Errors\\InternalError\",\"message\":\"OCA\\Tables\\Db\\Row2Mapper - findAll: Did expect one result but found none when executing: query \\\"SELECT * FROM `*PREFIX*tables_columns` WHERE `id` = :dcValue1\\\"; \",\"code\":0,\"file\":\"/srv/http/nextcloud/apps-repos/tables/lib/Db/Row2Mapper.php:195\",\"trace\":\"#0 /srv/http/nextcloud/apps-repos/tables/lib/Service/RowService.php(134): OCA\\Tables\\Db\\Row2Mapper->findAll(Array, 1465, NULL, NULL, Array, Array, '...')\\n#1 /srv/http/nextcloud/apps-repos/tables/lib/Controller/RowController.php(57): OCA\\Tables\\Service\\RowService->findAllByView(872, '...')\\n#2 /srv/http/nextcloud/apps-repos/tables/lib/Controller/Errors.php(23): OCA\\Tables\\Controller\\RowController->{closure:OCA\\Tables\\Controller\\RowController::indexView():52}()\\n#3 /srv/http/nextcloud/apps-repos/tables/lib/Controller/RowController.php(52): OCA\\Tables\\Controller\\RowController->handleError(Object(Closure))\\n#4 /srv/http/nextcloud/lib/private/AppFramework/Http/Dispatcher.php(198): OCA\\Tables\\Controller\\RowController->indexView(872)\\n#5 /srv/http/nextcloud/lib/private/AppFramework/Http/Dispatcher.php(86): OC\\AppFramework\\Http\\Dispatcher->executeController(Object(OCA\\Tables\\Controller\\RowController), '...')\\n#6 /srv/http/nextcloud/lib/private/AppFramework/App.php(138): OC\\AppFramework\\Http\\Dispatcher->dispatch(Object(OCA\\Tables\\Controller\\RowController), '...')\\n#7 /srv/http/nextcloud/lib/private/Route/Router.php(324): OC\\AppFramework\\App::main('...', '...', Object(OC\\AppFramework\\DependencyInjection\\DIContainer), Array)\\n#8 /srv/http/nextcloud/lib/OC.php(1232): OC\\Route\\Router->match('...')\\n#9 /srv/http/nextcloud/index.php(28): OC::handleRequest()\\n#10 /srv/http/nextcloud/lib/OC.php(1397): {closure:/srv/http/nextcloud/index.php:25}()\\n#11 /srv/http/nextcloud/index.php(25): OC::handleRequests(Object(Closure))\\n#12 {main}\"}",
    "CustomMessage": "An internal error or exception occurred: OCA\\Tables\\Db\\Row2Mapper - findAll: Did expect one result but found none when executing: query \"SELECT * FROM `*PREFIX*tables_columns` WHERE `id` = :dcValue1\"; "
  }
}

[blizzz]

PPS.:

select * from oc_tables_columns where id = 5356;

results in the column data, but

select * from oc_tables_columns where id = -5;

results in zero results, hence the error.

[Koc]

@blizzz reproduced and fixed 🎉