Streaming encoders leave Frame_Content_Size out unless told the size up front,
and their readers often can't know it either: every zstd frame in an Anki
package (its collection, its media list, each media file) is written that way.
Such a frame needed the exact size passed as expectedSize, and anything else
was refused.
Now it gets the most its blocks can hold, Block_Maximum_Size (128 KiB) each,
found by walking the block headers alone, and the output is trimmed to what
the blocks held. The output is still allocated once. expectedSize keeps its
exact check when given, and maxOutputSize bounds the allocation as before.
Checked against libzstd (python-zstandard) on Anki 26.09.3's exports: a small
package's four frames and a 30 MB collection decode byte for byte the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eighteen commits of history, grafted onto the scaffold rather than over
it. It leaves behind everything that only meant something inside that
project: paths under one machine's mount, references to a transcoder
that is not here, and a benchmark defaulting to a tile nobody else has.
One fixture is real terrain and stays, because it is the only small
input that exercises the direct-nibble Huffman weight form. Its licence
requires attribution to travel with it, so the README's
acknowledgements and the corpus manifest both carry the Copernicus
notice now. Everything else committed here is synthetic or was written
by the zstd CLI, including the .hgtz container, whose name looks like a
tile and whose samples are a function of their position.
A sweep found no defect in the decoder and several in its tests. Three
of the five fixes could be reverted with all 103 tests still green.
The content size against expectedSize check is the worst of those,
because the .hgtz reader was simplified to rest on it: deleting
it left everything passing. Eleven cases cover it now. The RLE block
ceiling and the requirement that a frame be the whole input had none
either, so there are fixtures for a 13-byte frame that returns 2 MB, for
the same shape at exactly the maximum, for four bytes of junk after a
frame, and for a trailing skippable frame.
Two of the new fixtures close coverage gaps rather than guards. No legal
frame in the suite decoded a one-stream Huffman literals section, since
the only one present had to be refused. And the only test of hashing
each block as it lands rode on the zstd CLI being installed, so a
multi-block frame carrying a checksum is committed now.
Also replaces an assertion that constrained nothing. "Fills the buffer
exactly" checked that the count was over 256 and under 1024, which
passes with the guard at two, at three, or absent. The stream encodes
257 symbols, a buffer of 257 takes it and one of 256 does not.
And two comments that were wrong. The weight buffer is one larger than
the maximum on purpose, and shrinking it to 255 breaks 124 tests. A
caller passing expectedSize gets the exact check in addition to the
ceiling, not instead of it.
Three of the shapes worth covering produce frames of a few hundred kilobytes,
which is too much to commit for what they prove. The input is built from a
seeded generator and handed to the zstd CLI instead, so the frame is libzstd's
own output and the expectation is the exact bytes that went in.
The pool size decides which sequence tables libzstd picks. Two hundred tokens
gets RLE and repeat tables for the literal lengths, two thousand gets an RLE
table for the match lengths, and both put more than 32,512 sequences in one
block, which is the only thing that writes the three-byte sequence count.
It has no symbol count to stop on and runs until the bitstream is spent, so
its own room check is the only thing keeping it inside the caller's buffer.
Both exhausted branches write a third symbol in the same pass. This is the
53-byte reproducer with the frame stripped away, so a regression points at
the function rather than at a block somewhere.
A ZstdDecoder keeps its Huffman table, its three FSE tables, the repeat flags
and the three recent offsets between calls, and every one of those belongs to
a single frame. A leak is invisible while every frame in the list looks alike,
which is what the .hgtz corpus is. Reversing the order gives each frame a
different predecessor, so a leak shows up as a difference between the passes.
A test that asserts what the decoder currently does proves nothing. Every
legal frame here is compared byte for byte with what pyzstd decompressed it
to, and every malformed frame is one pyzstd also refuses.
The refusal has to be a ZstdException. RangeError, OutOfMemoryError and
IndexError are all Error rather than Exception, so a caller writing
"on ZstdException" does not catch them and the isolate dies.
The only fixture was one .hgtz container written by one producer at level 19.
An audit ran 16,490 real blocks through an instrumented decoder and found
thirteen branches of RFC 8878 that corpus never reaches at all: raw blocks,
RLE blocks, RLE literals, treeless literals at the three-byte size format,
frames with no content size, frames with a checksum, the one and eight byte
content sizes, the three-byte sequence count, a block with no sequences, and
RLE sequence tables for literal and match length.
make_corpus.py builds frames for them, either through libzstd or by hand and
then past pyzstd for a verdict, so legal and malformed are libzstd's opinion
rather than ours. The expected plain bytes of a legal frame are whatever
pyzstd decompressed it to, gzipped to keep the fixture small, and the
manifest records the exact command behind every frame. 46 KB in all.
The decoder parses Content_Checksum and steps over it without reading it.
An audit mutated real block data 600,000 times behind a valid frame header
and 34 percent decoded into wrong bytes with no error raised. A frame can
also set the checksum flag and supply no bytes at all. This is the only
mechanism that would catch a wrong decode from anything else, and the
package decodes elevation tiles on a phone.
RFC 8878 section 3.1.1 stores the low 32 bits of the XXH64 of the whole
content, seed zero, little-endian. Xxh64.digest32 is that value. The
incremental form is for the decoder, which writes the output block by
block and should not walk it a second time.
Every operation relies on int being a wrapping 64-bit two's complement
integer, which holds on the VM and on AOT and not under dart2js. The
package already targets native only.
Verified against the xxhash Python package, version 3.0.0, over both byte
patterns at every length class around the 32, 8, 4 and 1 byte boundaries,
over 400 random inputs one shot and in random chunkings, and end to end
against the trailing four bytes of a real zstd -19 --check frame.
A feasibility spike, standalone so it can move to its own repository. Not
wired into the app and not a dependency of it.
Decompression only, and only what our containers need: no dictionaries, no
multi-frame, no streaming, and the content checksum is parsed but never
verified. Native only, since the bit reader carries 56 bits in an int.
It is correct on real data. The committed fixture decodes to the samples
the generator wrote, and tool/oracle.py drives a byte-for-byte comparison
against pyzstd over 32 containers and 2,615 blocks from both tiers, every
thinned column count from 3601 to 361, and 88 S to 80 N. All identical.
The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median
against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles
the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The
cost is the sequence loop and nothing else: a terrain block at level 19
carries around 1,700 literals and 14,400 sequences, so optimising the
literal path bought nothing.
