Three ways a header was taken on trust.
Frame_Content_Size is up to a 64-bit number straight from the input and
it sized the output buffer unchecked, so a sixteen-byte frame could ask
for four exbibytes. It is reachable by accident: one bit flipped in a
real tile's frame descriptor moves the content-size flag from 2 to 3,
eight bytes of compressed data are read as the size, and the eighth
lands on the sign bit. That is a RangeError or an OutOfMemoryError, both
Errors rather than Exceptions, so a caller writing `on ZstdException`
catches neither and the isolate dies. There is a ceiling now, and a
caller that passes expectedSize gets an exact check instead, which is
what the .hgtz reader wants since a block is height * width * 2.
Reading one frame is the whole API, and nothing compared the cursor with
the input length, so a second frame, a trailing skippable frame and four
bytes of junk were accepted and silently ignored. Two frames
concatenated returned the first one's bytes and raised nothing. The
checksum's four bytes were stepped over rather than read, so a frame
could claim one and supply none.
And an RLE block was exempt from the block size ceiling, which let 97
bytes return 44 MB that a caller cannot tell from a good tile.
Both exhausted branches write a third symbol in the same pass that the
guard admitted for two. The Huffman weight decoder calls this with a
256-byte buffer and a limit of 256, so `at` reached 256 exactly, which
is one past the end, and a crafted weight stream turned that into an
IndexError rather than a refusal. A 53-byte input reproduces it, and
random mutation of a real tile found it independently.
The table said several frames in one buffer are refused. They are not:
the block loop breaks on the last block and never checks the cursor
against the input length, so a second frame, a trailing skippable frame
and appended junk are all accepted and silently ignored.
Four more, each with a reproducer and none fixed yet. The content size
sizes an allocation with no ceiling, and one flipped bit in a real
tile's descriptor kills the isolate with an error no caller can catch.
The interleaved FSE decode writes three symbols in a pass that checks
room for two. RLE blocks skip the block size ceiling. And the checksum
is never verified, which is the one that makes the others systemic:
34% of 600,000 mutations decoded into wrong bytes in silence.
Sequential runs of same_blocks.dart disagreed by 70% between the first
and third invocation on this box: schedutil plus boost ramp is a larger
effect than any difference between levels. Interleaving one block from
each file in turn puts that drift on every level equally.
0.414 ms a block in profile AOT on the Pixel, so a 113-block sweep is 47 ms.
The phone is about twice as quick at this as the laptop the first figures
came from, which read 0.917 ms and 104 ms for the same sixteen blocks.
Debug on device is 0.540 ms, so the shipped configuration is the fast one
rather than the slow one.
The blocks are real, out of N39W105 in srtm1z. The committed .hgtz fixture
is no use for timing: its samples are a synthetic pattern that compresses
to almost no sequences, and decode cost tracks sequence count, so it reads
about six times too fast. It was measured that way once before this was
noticed.
A feasibility spike, standalone so it can move to its own repository. Not
wired into the app and not a dependency of it.
Decompression only, and only what our containers need: no dictionaries, no
multi-frame, no streaming, and the content checksum is parsed but never
verified. Native only, since the bit reader carries 56 bits in an int.
It is correct on real data. The committed fixture decodes to the samples
the generator wrote, and tool/oracle.py drives a byte-for-byte comparison
against pyzstd over 32 containers and 2,615 blocks from both tiers, every
thinned column count from 3601 to 361, and 88 S to 80 N. All identical.
The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median
against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles
the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The
cost is the sequence loop and nothing else: a terrain block at level 19
carries around 1,700 literals and 14,400 sequences, so optimising the
literal path bought nothing.
