Streaming encoders leave Frame_Content_Size out unless told the size up front,
and their readers often can't know it either: every zstd frame in an Anki
package (its collection, its media list, each media file) is written that way.
Such a frame needed the exact size passed as expectedSize, and anything else
was refused.
Now it gets the most its blocks can hold, Block_Maximum_Size (128 KiB) each,
found by walking the block headers alone, and the output is trimmed to what
the blocks held. The output is still allocated once. expectedSize keeps its
exact check when given, and maxOutputSize bounds the allocation as before.
Checked against libzstd (python-zstandard) on Anki 26.09.3's exports: a small
package's four frames and a 30 MB collection decode byte for byte the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eighteen commits of history, grafted onto the scaffold rather than over
it. It leaves behind everything that only meant something inside that
project: paths under one machine's mount, references to a transcoder
that is not here, and a benchmark defaulting to a tile nobody else has.
One fixture is real terrain and stays, because it is the only small
input that exercises the direct-nibble Huffman weight form. Its licence
requires attribution to travel with it, so the README's
acknowledgements and the corpus manifest both carry the Copernicus
notice now. Everything else committed here is synthetic or was written
by the zstd CLI, including the .hgtz container, whose name looks like a
tile and whose samples are a function of their position.
MIT rather than a dual with BSD-3. Dual licensing solves a
compatibility problem this does not have, since both are permissive and
compatible with everything, so the choice would cost a downstream reader
a decision and buy them nothing.
The debt to zstd is real and a licence is the wrong instrument for it.
No code came from libzstd; the format came from RFC 8878 and every
answer is checked against libzstd through pyzstd. That belongs in prose
where people read it.
Also drops the line calling this a feasibility spike. It has 103 tests,
an audit against the RFC, and five defects found and closed.
Only the unverified checksum is left, and it is narrower here than it
looks: the downloader verifies a SHA-256 over the whole container before
it renames the .part, so a tile damaged in transit is already caught.
What remains is corruption at rest on the phone.
The table said several frames in one buffer are refused. They are not:
the block loop breaks on the last block and never checks the cursor
against the input length, so a second frame, a trailing skippable frame
and appended junk are all accepted and silently ignored.
Four more, each with a reproducer and none fixed yet. The content size
sizes an allocation with no ceiling, and one flipped bit in a real
tile's descriptor kills the isolate with an error no caller can catch.
The interleaved FSE decode writes three symbols in a pass that checks
room for two. RLE blocks skip the block size ceiling. And the checksum
is never verified, which is the one that makes the others systemic:
34% of 600,000 mutations decoded into wrong bytes in silence.
0.414 ms a block in profile AOT on the Pixel, so a 113-block sweep is 47 ms.
The phone is about twice as quick at this as the laptop the first figures
came from, which read 0.917 ms and 104 ms for the same sixteen blocks.
Debug on device is 0.540 ms, so the shipped configuration is the fast one
rather than the slow one.
The blocks are real, out of N39W105 in srtm1z. The committed .hgtz fixture
is no use for timing: its samples are a synthetic pattern that compresses
to almost no sequences, and decode cost tracks sequence count, so it reads
about six times too fast. It was measured that way once before this was
noticed.
A feasibility spike, standalone so it can move to its own repository. Not
wired into the app and not a dependency of it.
Decompression only, and only what our containers need: no dictionaries, no
multi-frame, no streaming, and the content checksum is parsed but never
verified. Native only, since the bit reader carries 56 bits in an int.
It is correct on real data. The committed fixture decodes to the samples
the generator wrote, and tool/oracle.py drives a byte-for-byte comparison
against pyzstd over 32 containers and 2,615 blocks from both tiers, every
thinned column count from 3601 to 361, and 88 S to 80 N. All identical.
The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median
against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles
the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The
cost is the sequence loop and nothing else: a terrain block at level 19
carries around 1,700 literals and 14,400 sequences, so optimising the
literal path bought nothing.
