A sweep found no defect in the decoder and several in its tests. Three
of the five fixes could be reverted with all 103 tests still green.
The content size against expectedSize check is the worst of those,
because the .hgtz reader was simplified to rest on it: deleting
it left everything passing. Eleven cases cover it now. The RLE block
ceiling and the requirement that a frame be the whole input had none
either, so there are fixtures for a 13-byte frame that returns 2 MB, for
the same shape at exactly the maximum, for four bytes of junk after a
frame, and for a trailing skippable frame.
Two of the new fixtures close coverage gaps rather than guards. No legal
frame in the suite decoded a one-stream Huffman literals section, since
the only one present had to be refused. And the only test of hashing
each block as it lands rode on the zstd CLI being installed, so a
multi-block frame carrying a checksum is committed now.
Also replaces an assertion that constrained nothing. "Fills the buffer
exactly" checked that the count was over 256 and under 1024, which
passes with the guard at two, at three, or absent. The stream encodes
257 symbols, a buffer of 257 takes it and one of 256 does not.
And two comments that were wrong. The weight buffer is one larger than
the maximum on purpose, and shrinking it to 255 breaks 124 tests. A
caller passing expectedSize gets the exact check in addition to the
ceiling, not instead of it.
Skipping it left the decoder with no integrity check at all. Over
600,000 mutations of real block data with the frame header held valid, a
third decoded into wrong bytes and raised nothing, because a damaged
match or literal length usually still produces plausible output of the
right length. Every other guard in here catches a frame that cannot be
parsed; this is the only one that catches a frame that parses and is
wrong.
Hashed only when the frame carries a checksum. The .hgtz reader would
pay for it on every block and has no use for it: pyzstd writes none, and
the container's own SHA-256 is verified before the download is kept.
The decoder parses Content_Checksum and steps over it without reading it.
An audit mutated real block data 600,000 times behind a valid frame header
and 34 percent decoded into wrong bytes with no error raised. A frame can
also set the checksum flag and supply no bytes at all. This is the only
mechanism that would catch a wrong decode from anything else, and the
package decodes elevation tiles on a phone.
RFC 8878 section 3.1.1 stores the low 32 bits of the XXH64 of the whole
content, seed zero, little-endian. Xxh64.digest32 is that value. The
incremental form is for the decoder, which writes the output block by
block and should not walk it a second time.
Every operation relies on int being a wrapping 64-bit two's complement
integer, which holds on the VM and on AOT and not under dart2js. The
package already targets native only.
Verified against the xxhash Python package, version 3.0.0, over both byte
patterns at every length class around the 32, 8, 4 and 1 byte boundaries,
over 400 random inputs one shot and in random chunkings, and end to end
against the trailing four bytes of a real zstd -19 --check frame.
Three ways a header was taken on trust.
Frame_Content_Size is up to a 64-bit number straight from the input and
it sized the output buffer unchecked, so a sixteen-byte frame could ask
for four exbibytes. It is reachable by accident: one bit flipped in a
real tile's frame descriptor moves the content-size flag from 2 to 3,
eight bytes of compressed data are read as the size, and the eighth
lands on the sign bit. That is a RangeError or an OutOfMemoryError, both
Errors rather than Exceptions, so a caller writing `on ZstdException`
catches neither and the isolate dies. There is a ceiling now, and a
caller that passes expectedSize gets an exact check instead, which is
what the .hgtz reader wants since a block is height * width * 2.
Reading one frame is the whole API, and nothing compared the cursor with
the input length, so a second frame, a trailing skippable frame and four
bytes of junk were accepted and silently ignored. Two frames
concatenated returned the first one's bytes and raised nothing. The
checksum's four bytes were stepped over rather than read, so a frame
could claim one and supply none.
And an RLE block was exempt from the block size ceiling, which let 97
bytes return 44 MB that a caller cannot tell from a good tile.
Both exhausted branches write a third symbol in the same pass that the
guard admitted for two. The Huffman weight decoder calls this with a
256-byte buffer and a limit of 256, so `at` reached 256 exactly, which
is one past the end, and a crafted weight stream turned that into an
IndexError rather than a refusal. A 53-byte input reproduces it, and
random mutation of a real tile found it independently.
A feasibility spike, standalone so it can move to its own repository. Not
wired into the app and not a dependency of it.
Decompression only, and only what our containers need: no dictionaries, no
multi-frame, no streaming, and the content checksum is parsed but never
verified. Native only, since the bit reader carries 56 bits in an int.
It is correct on real data. The committed fixture decodes to the samples
the generator wrote, and tool/oracle.py drives a byte-for-byte comparison
against pyzstd over 32 containers and 2,615 blocks from both tiers, every
thinned column count from 3601 to 361, and 88 S to 80 N. All identical.
The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median
against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles
the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The
cost is the sequence loop and nothing else: a terrain block at level 19
carries around 1,700 literals and 14,400 sequences, so optimising the
literal path bought nothing.
