Add low-memory JPEG decode path - #1
Draft
mariusandra wants to merge 29 commits into
Draft
Conversation
- pixie/decodebudget: runtime per-decode memory budget (replaces the compile-time frameosEmbedded consts); decoders raise catchable PixieErrors instead of exhausting memory - jpeg: decode plan checked against the budget before any image-sized allocation; progressive JPEGs now use target-sized channel masks; sampling resolution clamps itself to the budget, trading sharpness for a successful decode - jpeg: streaming decode (decodeJpegStreamScaled/Into) pulls the compressed input through a 32K sliding window so files never need to be fully buffered; bit-identical output across the test suite - jpeg: decodeJpegInfo probe + jpegDecodeIntermediateBytes for pre-decode budget planning - jpeg/png: ScaledDecodeFit (stretch/cover/contain) on all scaled decode paths, enabling aspect-correct decode-into-canvas; cover inflates mask sampling density for the cropped region within budget - png: budget check at IHDR; inflated scanlines released before the pixel seq allocation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
seekEntropyMarker could set state.pos behind the sliding window start on a long 0xFF run in damaged entropy data; clamp to windowStart so recovery matches the buffered decoder instead of failing the whole decode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unfiltering allocated a second scanline-sized buffer, putting the decode plan for a canvas-sized RGBA PNG at pixels + 2x scanlines (4.5MB for 480x800). Compacting the [filter byte][row] stride in place drops the plan to pixels + scanlines (3.0MB), which fits the decode budget on ESP32-class devices. Interlaced images keep the per-pass copy and the old plan formula. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Non-interlaced PNGs now decode row by row through zippy's streaming inflate (pinned to the FrameOS fork): scanlines are unfiltered against a single previous-row buffer and written straight into the pixel data, so the whole-image inflate buffer disappears. Full decodes plan pixels + a fixed ~64KB; decodePngScaled/Into sample rows on the fly and never allocate the full-size pixel buffer at all, so a huge PNG can scale into display bounds like a streamed JPEG. Interlaced and 16-bit-scaled decodes keep the buffered path. Verified pixel-identical to the previous decoder across the whole pngsuite corpus, plus differential streamed-vs-buffered scaled decode tests and tightened budget assertions (480x800 RGBA: full decode within 2MB, scaled-into within 256KB). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A transitive URL requirement breaks nimble's CI resolution (two zippy sources for one package name); frameos.nimble pins the FrameOS zippy fork at the root instead, the same proven pattern used for pixie. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI's nimble cannot reliably resolve a second forked package (transitive or root URL requirements both produced incomplete nimble.paths), so the streaming inflate now lives inside pixie as a self-contained module (huffman machinery vendored from zippy 0.10.16, MIT) raising PixieError directly. The dependency graph returns to the CI-proven shape: one forked package (pixie), stock guzba zippy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The streaming inflate only took one contiguous buffer, so PNGs that split their compressed data across many IDAT chunks (libpng emits 8K chunks; a 2.6MB gallery PNG has 326) were concatenated into a second multi-MB allocation before decoding — the allocation that OOMed streamed decodes on fragmented ESP32 PSRAM. The bit reader now consumes a list of InflateSegments, crossing chunk boundaries byte-wise and keeping the fast word-load path within a segment. Verified: a real 2.6MB multi-IDAT PNG stream-decodes into a 1200x1600 target under a 1MB decode budget, pixel-identical to the buffered path; IDATs re-chunked at 1..8192 bytes decode identically; PNG fuzzer clean over 10k iterations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
decodePngScaledInto now accepts a list of InflateSegments, so a PNG whose bytes arrived in fixed-size download chunks decodes without ever being assembled into one contiguous buffer: the segmented parser validates every chunk CRC across boundaries (incremental crc32) and hands IDAT spans to the segmented inflater as sub-slices. Non-interlaced <=8-bit images stream scanlines into the target; interlaced/16-bit fall back to coalescing plus the buffered decode. streamIdatRows now takes inflater segments directly, shared by the contiguous and segmented paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
decodePngStreamScaledInto(source, totalLen, target, fit) decodes a PNG read sequentially from a callback (e.g. a download spilled to disk on a device without the memory to buffer it). The chunk walker validates CRCs as bytes stream by and feeds IDAT payloads to the inflater through a 16KB read buffer, so peak memory is that buffer plus the fixed streaming overhead — the compressed file is never resident. The streaming inflater gains InflatePull, a pull-based input the reader falls back to when its segment list is exhausted; input consumption is strictly sequential so each pulled buffer may be reused. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The orientation SHORT occupies the first two bytes of the 4-byte IFD data field. After the full-word maybeSwap it sits in the LOW word for little-endian files, but the value was always taken from the high word (`shr 16`) — so orientation from II-endian cameras (Sony, Canon) was silently read as 0 and photos rendered sideways. Big-endian (MM) files, which all the existing f1..f8 fixtures use, were unaffected. Adds II-endian variants of the f1..f8 orientation fixtures and a test asserting they decode pixel-identically to the MM originals. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PNG and JPEG each declared their own pull-source callback with an identical signature, and frameos already passed one file closure to both. Name it once as ImageSourceProc and keep PngSourceProc/JpegSourceProc as aliases so callers compile unchanged. scaledFitRects moves to common.nim alongside it: BMP and PPM need the same fit arithmetic, and a second copy would be a second place for fitContain's untouched-border rule to drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A spilled BMP had no file-backed decoder, so a download too big for RAM could only fail — the whole point of spilling to storage is that the decode never needs the full body back. BMP is the easiest format to stream: no entropy coder, fixed stride, strictly sequential rows. decodeBmpStreamScaledInto pulls rows through the same engine PNG uses — one source row in RAM, a monotonic target cursor, nearest-neighbour sampling — with the row cursor running upward for bottom-up files so the trailing flipVertical (and its whole-image buffer) disappears. Rows no target row samples are consumed without conversion, and the walk stops once the last sampled row is read, so a fitCover crop never touches the file's tail. decodeDib is rebuilt on the same header parse and row converter, and now checks the decode budget before newImage: a 20000x20000 header used to walk straight into a 1.6 GB allocation. PPM P6 gets the same treatment; ASCII P3 raises rather than buffering back. Also fixes a pre-existing crash found while fuzzing: decodeBmp checked for 14 bytes and then indexed byte 14, so an exactly-14-byte file raised IndexDefect instead of a catchable PixieError. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`newImage(svg)` allocates the image it draws into, so a caller that already owns a correctly sized buffer — a render canvas, a cell of a larger image — has to take a second full-size allocation and then blend it away. On a memory-tight device that second image is the difference between rendering and not. `renderInto(svg, target)` is the same rasterization with the destination supplied. The body moves to a shared `renderSvg` and nothing else changes; `newImage` keeps its overwrite-first start, which is only an optimisation for the fresh transparent image it just allocated, where overwrite and normal agree. `renderInto` composites from the first path instead, because its target may already have content and there the two are not the same: a semi-transparent first path would replace what is underneath rather than blend with it. Tested both ways round. On a fresh target it is bit-identical to `newImage`. On a target with content it matches rendering on transparency and then drawing the result, to within a few units of 255 — the Tiger overlaps hundreds of semi-transparent paths, and compositing onto an opaque background quantizes slightly differently at each one than compositing onto transparency does before the final blend.
Not for merging. This is the exploration behind the design note in frameos docs/value-pipeline.md, parked so the findings are not lost. What works: `Image` gains `stride`, `origin`, a shared `pixels` pointer and a `root` that keeps the owner alive; `view(image, x, y, w, h)` returns a window that writes through and flattens views-of-views onto the original owner. `data` became a template over the pointer, so every per-pixel site and everything already routed through `dataIndex` is view-correct with no edit and no extra indirection. What that buys, and it is the important part: `data.len` no longer compiles, so the compiler enumerates exactly the code that assumed the whole image is one contiguous run. About 160 sites, of which the great majority are decoders and encoders working on an image they just allocated — those are always owners and were mechanically renamed to `dataLen` (scripted, driven off the compiler). What remains is the real work: roughly fifteen whole-image operations in images.nim — fill, applyOpacity, invert, ceil, flipHorizontal, magnifyBy2 and friends — plus their per-backend SIMD variants, each of which walks the buffer flat. Every one needs to either iterate spans (one span when contiguous, so owners keep today's exact performance, one per row otherwise) or fall back to an owned copy. The optimisation predicates (isOneColor, isTransparent, isOpaque) turn out to have no callers inside pixie at all, so those can simply answer conservatively for a view. Also here: tests/bench_view_cost.nim, a dependency-free bench of the raster hot paths, so the indirection cost can be measured as a before/after rather than argued about. Baseline on an M-series mac, before any of this: fill 0.042 ms draw scaled 2x 0.215 ms draw opaque over canvas 0.010 ms per-pixel read+write 0.440 ms draw alpha over canvas 0.026 ms fillPath rounded rect 0.023 ms subImage copy (quarter) 0.034 ms newImage quarter 0.027 ms
`subImage` allocates a buffer and copies a region into it, so handing a caller a sub-region costs a full copy out and, if the caller wrote to it, a copy back. For a tiled render that draws each cell into its own region, that is one buffer per cell — and nested tiles stack them, holding every level's copy live while the innermost one renders. `view(image, x, y, w, h)` returns a window onto the original instead. It writes through, so there is nothing to copy back, and views of views flatten onto the first owner, so nesting costs one object and no extra indirection however deep it goes. `subImage` keeps copying; callers that want a snapshot still get one. ## How it fits `Image` gains `stride` (elements between rows in the shared buffer) and `origin` (index of its top-left), and `data` becomes a pointer to the buffer it addresses — its own, or its owner's. Everything already routed through `dataIndex` became view-correct with no edit and no extra work per pixel. Making `data` a pointer also removed `data.len`, which is what turned "find every place that assumes the whole image is one contiguous run" from an audit into a compile error. Most were decoders and encoders working on an image they had just allocated — always owners — and became `dataLen`. The rest are the whole-image operations, and they now go through `forEachSpan`: one span for an owner, so the SIMD path gets the same single large range it always did, one span per row for a view. `isOneColor` and `isTransparent` decline for a view rather than answer wrongly; they had no callers inside pixie, and a caller that gets `false` simply takes the general path. `isOpaque` is exact for views instead, since nothing could bypass it. `rotate90` refuses a view outright — it swaps the dimensions, and a window cannot change shape inside its parent. ## Bugs this turned up, all pre-existing * `isOpaqueSse2` seeded its aligned pointer from `data[0]` while its index started at `start`, so it answered for the wrong range whenever `start != 0`. * `applyOpacityNeon`'s scalar tail multiplied two `uint8`s and wrapped: 240 at half opacity came out 0 instead of 120. * `isOpaqueNeon` had the same misaligned probe, without the miscomputation. * `encodePng` did `var copy = image.data` and then converted `copy` to straight alpha. That was a seq copy before and is a pointer copy now, so it would have rewritten the caller's image; it takes a packed copy via `toContiguousSeq`. * `rotate90` ended with `image.data = move rotated.data`, which after the type change hands the image a pointer into a temporary that dies on return. ## Also `==` on images compares pixels. It has to exist now: `a.data == b.data` compares addresses, which is never what anyone means and is quietly false for identical images — the test suite was full of it. `newImageFrom`/`newImageFromUnchecked` adopt a decoder's buffer without copying, keeping the move that `convertToImage` exists for. Whole suite green. tests/bench_view_cost.nim measures the cost: on the shared operations there is none (fill, draws, per-pixel access and fillPath all within noise of the buffer-owning build), and taking a quarter-canvas region goes from 0.032 ms to nothing at all.
`Image` is a ref, so `==` already means 'the same object', and callers rely on that: 'did the producer draw straight into my canvas?' is an identity question, and answering it by comparing every pixel is both wrong and O(n). Overloading it silently changed two such checks in frameos (render/image and zoomPan) and one in a test that asserts two images are distinct objects. The proc is still worth having — `a.data == b.data` compares addresses now — it just needs a name that says what it does.
`for c in image.data` worked because `data` was a seq. A pointer has no length to iterate, so `items`/`pairs` on the image replace it — and they walk row by row, which is also the only thing that is correct for a view.
The SOF plan check existed so oversized decodes fail with a catchable budget error before any image-sized allocation. Two allocations escaped it: the reconstruction path magnifies subsampled channels to full stride resolution (with the half-size source still alive during the last doubling), and buildImage allocates the output image next to everything the plan approved. On a fragmented embedded heap that is a plan that "fits the budget" followed by an allocation that aborts the render. Masks in the non-scaled path are now accounted at their upsample peak, and both buildImage variants check plan-plus-output before newImage — the *Into variants stay unchecked on the output, their target is the caller's to have afforded. tests/test_jpeg.nim pins all three edges: the accounted total covers the chroma peak, buffers-fit-output-doesn't refuses catchably naming the output, and honest headroom decodes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
decodeWebpScaledInto samples straight from the decoder's own buffers into the fitted target rect — YUV planes for lossy, the ARGB buffer VP8L cannot avoid for lossless — so the full-size RGBA intermediate never exists. The chroma upsampling is factored into per-row/per-pixel helpers shared with the buffered path; a fingerprint sweep over all 129 suite files pins the refactor byte-identical, and the new tests pin native-size scaled decode pixel-identical to decodeWebp. decodeWebpStreamScaledInto decodes a spilled body from a pull source. Unlike PNG or baseline JPEG, a WebP bitstream cannot be windowed — VP8 interleaves macroblock rows across coefficient partitions and VP8L's LZ77 window is the whole image — so the honest tier is "compressed body resident, full-size RGBA never", with the body budget-checked before it is pulled back. Every entry point now checks its decode plan (padded YUV planes, per-macroblock state, alpha buffers, RGBA outputs) against the memory budget first, so an oversized WebP refuses catchably instead of dying in an allocation. decodeImageScaled/Into dispatch WebP alongside PNG, JPEG and BMP. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Non-integer downscales through the scaled JPEG decode were nearest decimation, and worse: fillImage walked target -> image -> sample, and the double floor division duplicated some sample columns and skipped others. Fine texture (a 960px Wikimedia thumb covered onto an 800px panel) came out visibly rough — the trigger the quality-scaler gate was waiting for. Two fixes. idctBlockScaled now routes blocks through a full-width band that drains row by row into accumulators: every target pixel becomes the rounded average of its exact source footprint. Footprints tile the source, partial sums survive band boundaries, upscale axes keep nearest (boxes would leave holes), and 1:1 reduces to the identity — pinned byte-identical across the whole masters corpus, upscales included. And fillImage now decides geometry in image space but walks the sample grid directly, one floor mapping, no round trip. The WebP samplers get the same box treatment (premultiplied-domain averaging, verified exact against an independent reference), and the band and accumulator bytes are counted in the JPEG decode plan. Peak memory stays a band plus two target-width rows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The box filter left one gap: subsampled chroma channels are decoded at half the sample grid and were nearest-replicated up to it, so chroma edges on downscaled 4:2:0 photos stayed blocky. channelAt now interpolates between a downsampled channel's two nearest samples per axis, center-aligned — and only for channels boxed below their native resolution, so Y planes, 1:1 decodes and upscales keep their exact former bytes (pinned by a fingerprint sweep: 288 unchanged cases, changes confined to subsampled-chroma downscales). The 4:2:0 tolerance against the ideal RGB-domain box tightens from 24 to 18. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The box filter reached JPEG and WebP but never the row-streamed decoders, and PNG is what an XKCD comic is: nearest decimation through a contain fit swallowed letter stems whole (the D in DWARFS lost its vertical). A shared RowBoxSampler in common.nim now folds full scanlines — arriving top-down or bottom-up — into the fitted rect: downscale axes are area filtered, 1:1 axes are the identity and upscale axes keep the exact former nearest replication, byte for byte across the whole fixture corpus (fingerprint-swept). PNG's streaming and whole-buffer paths, BMP's streaming engine and PPM's P6 stream all feed it; peak memory grows by two target-width accumulator rows, counted in each decoder's budget plan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… hosts sharing it across a .so The root field, added with image views, turns Image into a cyclic type for ORC. Non-final decrefs then go through the cycle collector's root list, and FrameOS passes Images into driver shared libraries that carry their own ORC runtime: the .so registers the object in its list, the host unregisters it from a different one and segfaults in unregisterCycle after every render. A view's root is always an owner, so the type cannot actually cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`<text>` used to fail the whole document — one tag, and a drawing that was otherwise entirely supported turned into "Unsupported SVG tag". Callers worked around it by layering a separate text renderer over the rasterized SVG, which means the SVG can no longer describe its own labels. The parser now typesets `<text>` (and `<tspan>`) and appends the glyph outlines as ordinary paths, so fill, stroke, gradients, opacity, transforms and band-wise rendering all apply to text exactly as they do to a `<path>` — no second code path, nothing new in the renderer. Pixie ships no fonts, so font-family resolution is a hook the application installs: `setSvgTypefaceResolver` is asked for one candidate at a time, most-preferred first, and finally for the empty family (the default face). Declining every candidate skips the text and keeps the rest of the drawing, which is what a missing font should cost. Supported: x/y/dx/dy, font-family lists, font-size in CSS units, font-weight, font-style, text-anchor, dominant-baseline, per-tspan positions and paint. Not in this pass: textPath, textLength, letter-spacing, per-glyph x/y lists, wrapping, complex shaping, and color (bitmap) emoji, which have no outline. Whitespace needed one more thing: Nim's XML parser drops whitespace after a closing tag unless asked for it, which is invisible everywhere except inside `<text>`, where the space in `…</tspan> tail` is content. `parseSvgXml` asks for it, and only for documents that contain text — a node per whitespace run is real memory on a small device.
Anyone landing here — including us, six months from now — sees upstream's README and no sign that this is a fork, let alone which of the API is ours. The additions are all in service of one thing (drawing on hardware that has no room for the picture), so the section leads with that and then lists what followed: the decode budget, scaled and streaming decoding, the vendored streaming inflate, image views, SVG text and renderInto, color emoji, and the EXIF fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Needed this to be able to load JPEGs on an ESP32...