Resolve InvokeAI board media through their recorded subfolder

[lstein]

The bug

Indexing an InvokeAI board containing videos found none of them on disk.

InvokeAI does not store board media flat under outputs/images / outputs/videos. The subfolder strategy is a server-side setting (flat, type, date, hash), and the subfolder each file was actually written into is recorded per row at save time. On a type-organized backend every non-intermediate video lives at outputs/videos/general/<name>.mp4.

PhotoMap asked the /boards/{id}/image_names and /videos/names endpoints, which return only names, and joined them straight onto <invokeai_root>/outputs/videos/<name> — so is_file() was false for every video on the board. Depending on the mix, the album either indexed as empty or failed with "None of the N board video(s) listed by InvokeAI were found under …/outputs/videos — check the InvokeAI root directory", which pointed at the wrong culprit.

Images had the same bug and it was simply less visible: a long-lived outputs/images still holds a pile of legacy flat files, so old images resolved while newer ones (image_subfolder: "general") silently vanished.

The fix

The client now pages the DTO listings GET /api/v1/{images,videos}/ instead of the two name endpoints (deprecated upstream anyway, in favour of the polymorphic gallery listing) and reads image_subfolder / video_subfolder off each row, returning paths relative to that media type's outputs directory. routers/index.py joins them.

A backend predating the setting reports no subfolder, which yields exactly the old flat path — verified against real legacy rows, which report ''. Existing indexes see no churn, and nothing is re-keyed or re-embedded.

Cost: roughly one request per 1000 rows and a fatter payload, since a whole DTO is fetched to read one field. Measured on the largest local board — 14,103 images in 2.7 s.

Hardening (second commit, from an adversarial review)

• The subfolder sanitizer could be walked out of the outputs directory. It normalized backslashes to / before validating, but tested is-absolute on the raw string, so \etc passed the check and then became /etc — and joining an absolute path discards the left-hand side entirely. The normalization was never needed: InvokeAI's own _validate_subfolder rejects backslashes outright and every strategy emits /. Names were checked with PurePosixPath, which treats ..\..\secret as an ordinary filename (true on Linux, false on a Windows PhotoMap host). Both now go through one rule matching InvokeAI's own grammar: relative, /-separated, no empty or dot segments, no backslashes anywhere.
• Offset pagination is not a snapshot. InvokeAI orders by (starred, created_at) and serves each page from its own query, so deleting or unstarring one row between pages shifts every later row down an offset and skips exactly one. Since an index update prunes every indexed row the listing does not mention, that silently deleted a file still on the board. Each board is now checked against the total its own listing reported, re-walked once if short, and only then failed — an aborted update leaves the previous index intact, which is recoverable; pruning live rows is not. Rows added mid-walk only produce duplicates, which the dedupe absorbs.
• The same check catches an intermediary that truncates pages, and the walk stops once it has paged past the declared total, so a proxy that ignores offset costs a handful of requests rather than the whole page budget.

Verification

• Live against InvokeAI 6.14.0-alpha: 14103/14103 images on the largest board (15 pages, 2.7 s, no spurious re-read), 30/30 and 20/20 videos on the video boards, 250/250 on the uncategorized bucket across a mix of flat and subfoldered rows. Paging re-checked live at a forced page size of 100.
• New test_board_media_in_subfolders_are_found covers both depths (type's one segment, date's three) end-to-end through indexing and asserts the stored paths, not basenames. It fails against the old flat join, so it is a genuine regression test.
• Escape cases are parameterized over every backslash, absolute, traversal, and empty-segment spelling, asserted through the joined path.
• New tests for the short-listing re-read, the fail-rather-than-prune path, the offset-ignoring server, a payload with no total, and rows added mid-walk.
• 773 passed, ruff clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JLPwCcYrkpzwaTbk4FwjuB