From d970a0cca853f5f0d13f80bccc196cb3bbc1276b Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 21:37:56 -0600 Subject: [PATCH 01/18] Add a pure-Dart zstd decoder, as a benchmark A feasibility spike, standalone so it can move to its own repository. Not wired into the app and not a dependency of it. Decompression only, and only what our containers need: no dictionaries, no multi-frame, no streaming, and the content checksum is parsed but never verified. Native only, since the bit reader carries 56 bits in an int. It is correct on real data. The committed fixture decodes to the samples the generator wrote, and tool/oracle.py drives a byte-for-byte comparison against pyzstd over 32 containers and 2,615 blocks from both tiers, every thinned column count from 3601 to 361, and 88 S to 80 N. All identical. The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The cost is the sequence loop and nothing else: a terrain block at level 19 carries around 1,700 literals and 14,400 sequences, so optimising the literal path bought nothing. --- .gitignore | 5 + README.md | 101 +++++++ analysis_options.yaml | 1 + benchmark/bench.dart | 122 ++++++++ lib/pure_zstd.dart | 8 + lib/src/bit_reader.dart | 140 +++++++++ lib/src/decoder.dart | 604 +++++++++++++++++++++++++++++++++++++ lib/src/exception.dart | 9 + lib/src/fse.dart | 230 ++++++++++++++ lib/src/huffman.dart | 201 ++++++++++++ lib/src/tables.dart | 67 ++++ pubspec.yaml | 11 + test/fixtures/N51E000.hgtz | Bin 0 -> 33633 bytes test/hgtz_block_test.dart | 63 ++++ tool/oracle.py | 103 +++++++ tool/verify.dart | 52 ++++ 16 files changed, 1717 insertions(+) create mode 100644 .gitignore create mode 100644 README.md create mode 100644 analysis_options.yaml create mode 100644 benchmark/bench.dart create mode 100644 lib/pure_zstd.dart create mode 100644 lib/src/bit_reader.dart create mode 100644 lib/src/decoder.dart create mode 100644 lib/src/exception.dart create mode 100644 lib/src/fse.dart create mode 100644 lib/src/huffman.dart create mode 100644 lib/src/tables.dart create mode 100644 pubspec.yaml create mode 100644 test/fixtures/N51E000.hgtz create mode 100644 test/hgtz_block_test.dart create mode 100755 tool/oracle.py create mode 100644 tool/verify.dart diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..337f9af --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +# Destined for its own repository, so it carries its own rules rather than +# relying on the enclosing one. +.dart_tool/ +# A library, not an application: the lockfile is the consumer's business. +pubspec.lock diff --git a/README.md b/README.md new file mode 100644 index 0000000..9e6f2e3 --- /dev/null +++ b/README.md @@ -0,0 +1,101 @@ +# pure_zstd + +A zstd decompressor in plain Dart, with no native dependency. + +It exists because every zstd binding on pub.dev is a wrapper over libzstd, and +an app that reads block-compressed elevation tiles on a phone would rather not +carry a native library per platform to do it. Decompression only. There is no +compressor and there is no plan for one. + +This is a feasibility spike, not a finished library. It decodes the real data +it was written for, correctly and completely, and it is roughly four times +slower than native zstd. See **Speed** below before depending on it. + +## Using it + +```dart +import 'package:pure_zstd/pure_zstd.dart'; + +final plain = zstdDecode(frame); + +// Decoding many frames: hold the decoder, which keeps its entropy tables and +// scratch buffers between calls and then allocates only the output. +final decoder = ZstdDecoder(); +for (final frame in frames) { + use(decoder.decode(frame)); +} +``` + +A frame that declares no content size needs `expectedSize`, because the +output is allocated whole rather than grown. + +## What it implements + +RFC 8878 is the specification. Of it: + +| Part of the format | State | +| --- | --- | +| Frame header, single segment and windowed | Decoded. The window descriptor is skipped, since the output is allocated from the content size. | +| Raw, RLE and compressed blocks | All three. | +| Multi-block frames | Yes, including a later block reusing an earlier one's tables and recent offsets. | +| Raw, RLE, compressed and treeless literals | All four, in one-stream and four-stream form. | +| Huffman weights | Both the direct nibble form and the FSE-coded form. | +| Sequence tables | Predefined, RLE, FSE-compressed and repeat, for all three of literal length, match length and offset. | +| Repeated offsets | Yes, including the no-literals shift. | +| Content checksum | Parsed and **skipped, never verified**. | +| Dictionaries | Refused. | +| Skippable frames, several frames in one buffer | Refused. Reading one frame is the whole API. | +| Streaming, or decoding part of a frame | Not offered. | + +It targets native Dart and not the web: the literals header reads a five-byte +field with a 32-bit shift, and the bit reader carries 56 bits in an `int`. +Both are exact on the VM and on AOT, and neither is on JavaScript. + +## Correctness + +The reference is pyzstd, which wraps libzstd. `tool/oracle.py` extracts every +block of a `.hgtz` container as its own frame, decompresses it with pyzstd, +hands it to `tool/verify.dart`, and compares byte for byte. + + ./tool/oracle.py path/to/srtm1z/N39W105.hgtz + ./tool/oracle.py --every 1700 path/to/srtm1z path/to/srtm3z + +As of 2026-09-21 that is 32 real containers, 2,615 blocks and 0.3 GB of +output, all identical, spanning both the one and three arc-second sets, every +thinned column count from 3601 down to 361, and latitudes from 88 S to 80 N. + +`dart test` needs no Python. It decodes the five blocks of the container in +`test/fixtures/`, whose every sample is a known function of its position, so a +wrong decode cannot pass. + +## Speed + +Measured 2026-09-21 on this eight-core Linux box, `dart compile exe`, Dart +3.12.2. One 256 by 256 block is 131,072 bytes out. The frames are read into +memory first, so only decoding is timed, and the figure includes allocating +the output buffer because the API allocates it. + +| Block set | pure_zstd, AOT | pyzstd (libzstd) | +| --- | --- | --- | +| Colorado, `N39W105`, 196 blocks, median | **0.66 ms** | 0.18 ms | +| Alaska, `N60W160`, 98 blocks, median | 0.94 ms | not measured | +| One block, repeated 1000 times | 0.85 ms | 0.22 ms | +| 113 blocks, a 360 degree sweep | 67 ms | 19 ms | + +That is 140 to 200 MB/s of output. The JIT is about 25% slower than AOT, so +measure with `dart compile exe`. + +The cost is almost all in the sequence loop rather than in Huffman literals. +A block of real terrain carries around 1,700 literals and 14,400 sequences, +so the hot path is three FSE state updates, six bit reads and a nine-byte +match copy, about 46 ns per sequence against native's dozen or so. The +headroom left is bounds checks and the byte-at-a-time match copy; widening +that copy to 64-bit words, the way libzstd does, is the next thing to try. + +Run it yourself: + + dart compile exe benchmark/bench.dart -o /tmp/bench && /tmp/bench + +## Licence + +Not chosen yet. diff --git a/analysis_options.yaml b/analysis_options.yaml new file mode 100644 index 0000000..572dd23 --- /dev/null +++ b/analysis_options.yaml @@ -0,0 +1 @@ +include: package:lints/recommended.yaml diff --git a/benchmark/bench.dart b/benchmark/bench.dart new file mode 100644 index 0000000..a7c2e05 --- /dev/null +++ b/benchmark/bench.dart @@ -0,0 +1,122 @@ +import 'dart:io'; +import 'dart:typed_data'; + +import 'package:pure_zstd/pure_zstd.dart'; + +/// Times decoding real `.hgtz` blocks. +/// +/// dart run benchmark/bench.dart [path to a .hgtz] +/// dart compile exe benchmark/bench.dart -o /tmp/bench && /tmp/bench +/// +/// Reports the cost of one 256 by 256 block, which is 131,072 bytes out, and +/// the cost of the 113 blocks a 360 degree skyline sweep touches. The frames +/// are read into memory first, so nothing but decoding is timed. The output +/// buffer is allocated per call, as the API allocates it, so the figure +/// includes that allocation and the garbage it makes. +const int sweepBlocks = 113; + +void main(List arguments) { + final path = arguments.isEmpty + ? 'path/to/srtm1z/N39W105.hgtz' + : arguments.single; + final bytes = File(path).readAsBytesSync(); + final header = ByteData.sublistView(bytes); + final rows = header.getUint16(6); + final columns = header.getUint16(8); + final block = header.getUint16(10); + final blockColumns = (columns + block - 1) ~/ block; + final blockRows = (rows + block - 1) ~/ block; + + // Only the square blocks, so every timing is over the same 131,072 bytes. + final full = []; + for (var index = 0; index < blockRows * blockColumns; index++) { + final entry = 12 + 8 * index; + final top = (index ~/ blockColumns) * block; + final left = (index % blockColumns) * block; + if (rows - top < block || columns - left < block) { + continue; + } + final offset = header.getUint32(entry); + full.add( + Uint8List.sublistView( + bytes, + offset, + offset + header.getUint32(entry + 4), + ), + ); + } + if (full.isEmpty) { + stderr.writeln('$path has no full $block by $block block'); + exit(2); + } + + final decoder = ZstdDecoder(); + final size = decoder.decode(full.first).length; + stdout.writeln(path); + stdout.writeln( + '${full.length} full blocks of $size bytes, ' + '${(full.map((f) => f.length).reduce((a, b) => a + b) / full.length).round()}' + ' compressed bytes on average', + ); + + // One block, over and over. The same frame every time, so the caches are as + // warm as they will ever be. This is the optimistic figure. + final one = full[full.length ~/ 2]; + for (var i = 0; i < 300; i++) { + decoder.decode(one); + } + final repeated = []; + for (var i = 0; i < 1000; i++) { + final watch = Stopwatch()..start(); + decoder.decode(one); + watch.stop(); + repeated.add(watch.elapsedMicroseconds); + } + _report('one block repeated, warm', repeated, size); + + // Every full block once, which is the mix a real sweep meets: different + // entropy tables, different literal counts, different sequence counts. + final each = []; + for (var pass = 0; pass < 8; pass++) { + for (final frame in full) { + final watch = Stopwatch()..start(); + decoder.decode(frame); + watch.stop(); + if (pass > 0) { + each.add(watch.elapsedMicroseconds); + } + } + } + _report('every full block, warm', each, size); + + // A sweep as one unit, since that is the latency the user feels. + final sweeps = []; + for (var pass = 0; pass < 30; pass++) { + final watch = Stopwatch()..start(); + for (var i = 0; i < sweepBlocks; i++) { + decoder.decode(full[i % full.length]); + } + watch.stop(); + sweeps.add(watch.elapsedMicroseconds); + } + sweeps.sort(); + final sweep = sweeps[sweeps.length ~/ 2] / 1000; + stdout.writeln( + '$sweepBlocks block sweep, median ${sweep.toStringAsFixed(2)} ms ' + '(${(sweep / sweepBlocks * 1000).toStringAsFixed(0)} us per block)', + ); +} + +void _report(String label, List microseconds, int size) { + microseconds.sort(); + final median = microseconds[microseconds.length ~/ 2]; + final low = microseconds[microseconds.length ~/ 10]; + final high = microseconds[microseconds.length * 9 ~/ 10]; + final mbPerSecond = size / median; // bytes per microsecond is MB/s. + stdout.writeln( + '${label.padRight(28)} ${(median / 1000).toStringAsFixed(3)} ms ' + 'median (p10 ${(low / 1000).toStringAsFixed(3)}, ' + 'p90 ${(high / 1000).toStringAsFixed(3)}, ' + 'n=${microseconds.length}) ${mbPerSecond.toStringAsFixed(0)} MB/s', + ); +} diff --git a/lib/pure_zstd.dart b/lib/pure_zstd.dart new file mode 100644 index 0000000..833338e --- /dev/null +++ b/lib/pure_zstd.dart @@ -0,0 +1,8 @@ +/// A pure Dart zstd decompressor. +/// +/// Decompression only, and only the subset of RFC 8878 that the elevation +/// containers this was written for use. See the README for what is left out. +library; + +export 'src/decoder.dart' show ZstdDecoder, zstdDecode; +export 'src/exception.dart' show ZstdException; diff --git a/lib/src/bit_reader.dart b/lib/src/bit_reader.dart new file mode 100644 index 0000000..bae6766 --- /dev/null +++ b/lib/src/bit_reader.dart @@ -0,0 +1,140 @@ +import 'dart:typed_data'; + +import 'exception.dart'; + +/// Reads a zstd entropy bitstream, which runs backwards. +/// +/// Every FSE and Huffman stream in a zstd frame is written from its end: the +/// last byte holds the first bits, and within a byte the most significant bit +/// comes first. Read as a little-endian integer the whole stream is one big +/// number consumed from the top down, which is what this does. +/// +/// The final byte carries a set bit marking where the data stops; the zeroes +/// above it are padding and the marker itself is not data. +class ReverseBitReader { + ReverseBitReader(); + + Uint8List _bytes = Uint8List(0); + + /// A view over [_bytes], so a refill can take four bytes at a time. Kept + /// between resets, since every stream in one block shares a buffer. + ByteData _view = ByteData(0); + + /// Index of the first byte of the stream. Reading stops when it is reached. + int _start = 0; + + /// Index of the next byte to pull into [_acc], counting down to [_start]. + int _next = 0; + + /// The unread bits, in the low [_available] bits, next bit at the top. + int _acc = 0; + int _available = 0; + + /// Zero bits shifted in past the front of the stream. They are only ever + /// at the bottom of [_acc], so any unread count below this means the + /// stream was over-read. + int _padding = 0; + + void reset(Uint8List bytes, int start, int end) { + if (end <= start) { + throw const ZstdException('empty entropy stream'); + } + final last = bytes[end - 1]; + if (last == 0) { + throw const ZstdException('entropy stream has no end marker'); + } + if (!identical(bytes, _bytes)) { + _bytes = bytes; + _view = ByteData.sublistView(bytes); + } + _start = start; + _next = end - 1; + _acc = last; + // bitLength - 1 is the marker's position; the data is everything below. + _available = last.bitLength - 1; + _padding = 0; + } + + /// True once more bits have been taken than the stream held. + bool get exhausted => _available < _padding; + + /// Tops [_acc] up to at least 49 bits, so a [peek] of up to 32 is safe and + /// several small reads can follow one call. + void fill() { + if (_available > 48) { + return; + } + var acc = _acc & ((1 << _available) - 1); + var available = _available; + var next = _next; + // Four bytes at a time while there is room for them. The stream is one + // little-endian number read from the top down, so a little-endian word + // lands the four bytes in the order the byte-at-a-time loop would. + while (available <= 24 && next - 4 >= _start) { + next -= 4; + acc = (acc << 32) | _view.getUint32(next, Endian.little); + available += 32; + } + while (available <= 48) { + if (next > _start) { + next--; + acc = (acc << 8) | _bytes[next]; + } else { + acc <<= 8; + _padding += 8; + } + available += 8; + } + _acc = acc; + _available = available; + _next = next; + } + + /// The next [count] bits without consuming them. Caller has called [fill]. + int peek(int count) => (_acc >> (_available - count)) & ((1 << count) - 1); + + void skip(int count) { + _available -= count; + } + + /// Reads [count] bits, refilling as needed. `count == 0` yields 0. + int read(int count) { + if (_available < count) { + fill(); + } + _available -= count; + return (_acc >> _available) & ((1 << count) - 1); + } +} + +/// Reads the forward, least-significant-bit-first bitstream that an FSE table +/// description uses. Table descriptions are tiny and read once per block, so +/// this favours being obviously correct over being quick. +class ForwardBitReader { + ForwardBitReader(this._bytes, this._end, int startByte) + : _position = startByte * 8; + + final Uint8List _bytes; + final int _end; + int _position; + + /// The next 32 bits, unconsumed. Reads past the end come back as zeroes. + int peek32() { + final index = _position >> 3; + var value = 0; + for (var k = 0; k < 5; k++) { + final at = index + k; + if (at < _end) { + value |= _bytes[at] << (8 * k); + } + } + return value >> (_position & 7); + } + + void skip(int count) { + _position += count; + } + + /// The number of bytes consumed, counting a partial byte as whole. + int bytesUsed(int startByte) => ((_position + 7) >> 3) - startByte; +} diff --git a/lib/src/decoder.dart b/lib/src/decoder.dart new file mode 100644 index 0000000..b38663a --- /dev/null +++ b/lib/src/decoder.dart @@ -0,0 +1,604 @@ +import 'dart:typed_data'; + +import 'bit_reader.dart'; +import 'exception.dart'; +import 'fse.dart'; +import 'huffman.dart'; +import 'tables.dart'; + +/// Decodes zstd frames, holding its entropy tables and scratch between calls +/// so that decoding a second frame allocates nothing but the output. +/// +/// One instance is not safe to share between isolates, and a frame must be +/// decoded to completion before the next one starts. +class ZstdDecoder { + static const int magic = 0xFD2FB528; + + /// The largest block a frame may contain, and so the most literals one + /// block can hold. + static const int maxBlockSize = 128 * 1024; + + final ReverseBitReader _bits = ReverseBitReader(); + final HuffmanTable _huffman = HuffmanTable(); + final FseTable _literalLengths = FseTable(literalLengthMaxLog); + final FseTable _matchLengths = FseTable(matchLengthMaxLog); + final FseTable _offsets = FseTable(offsetCodeMaxLog); + + bool _literalLengthsReady = false; + bool _matchLengthsReady = false; + bool _offsetsReady = false; + + /// The three recent offsets. They live for the whole frame, so a later + /// block can refer back to one a previous block established. + int _repeat1 = 1; + int _repeat2 = 4; + int _repeat3 = 8; + + Uint8List _literals = Uint8List(maxBlockSize); + + /// Where the current block's literals live: either [_literals] or, for raw + /// literals, the frame itself. + Uint8List _literalSource = Uint8List(0); + int _literalAt = 0; + int _literalEnd = 0; + + /// Decodes one frame. + /// + /// [expectedSize] is only needed when the frame header omits its content + /// size, which the containers this was written for never do. + Uint8List decode(Uint8List src, {int? expectedSize}) { + if (src.length < 6) { + throw const ZstdException('too short to be a zstd frame'); + } + if (src[0] | (src[1] << 8) | (src[2] << 16) | (src[3] << 24) != magic) { + throw const ZstdException('not a zstd frame: bad magic'); + } + + final descriptor = src[4]; + var at = 5; + final contentSizeFlag = descriptor >> 6; + final singleSegment = (descriptor >> 5) & 1; + if ((descriptor >> 3) & 1 != 0) { + throw const ZstdException('frame header reserved bit set'); + } + final checksum = (descriptor >> 2) & 1; + final dictionaryFlag = descriptor & 3; + + if (singleSegment == 0) { + at++; // Window_Descriptor. The output is allocated whole, so unused. + } + if (dictionaryFlag != 0) { + throw const ZstdException('dictionaries are not supported'); + } + + final contentSizeBytes = switch (contentSizeFlag) { + 0 => singleSegment == 1 ? 1 : 0, + 1 => 2, + 2 => 4, + _ => 8, + }; + if (at + contentSizeBytes > src.length) { + throw const ZstdException('truncated frame header'); + } + int? contentSize; + if (contentSizeBytes > 0) { + var value = 0; + for (var k = 0; k < contentSizeBytes; k++) { + value |= src[at + k] << (8 * k); + } + // The two-byte form is biased, since a frame that small would use the + // one-byte form. + contentSize = contentSizeBytes == 2 ? value + 256 : value; + at += contentSizeBytes; + } + + final size = contentSize ?? expectedSize; + if (size == null) { + throw const ZstdException( + 'frame declares no content size and none was given', + ); + } + final out = Uint8List(size); + + _huffman.log = 0; + _literalLengthsReady = false; + _matchLengthsReady = false; + _offsetsReady = false; + _repeat1 = 1; + _repeat2 = 4; + _repeat3 = 8; + + var written = 0; + while (true) { + if (at + 3 > src.length) { + throw const ZstdException('truncated block header'); + } + final header = src[at] | (src[at + 1] << 8) | (src[at + 2] << 16); + at += 3; + final last = header & 1; + final type = (header >> 1) & 3; + final blockSize = header >> 3; + if (blockSize > maxBlockSize && type != 1) { + throw ZstdException('block of $blockSize bytes is over the maximum'); + } + + switch (type) { + case 0: + if (at + blockSize > src.length || written + blockSize > out.length) { + throw const ZstdException('raw block does not fit'); + } + out.setRange(written, written + blockSize, src, at); + written += blockSize; + at += blockSize; + case 1: + if (at >= src.length || written + blockSize > out.length) { + throw const ZstdException('RLE block does not fit'); + } + out.fillRange(written, written + blockSize, src[at]); + written += blockSize; + at += 1; + case 2: + if (at + blockSize > src.length) { + throw const ZstdException('truncated compressed block'); + } + written = _decodeBlock(src, at, at + blockSize, out, written); + at += blockSize; + default: + throw const ZstdException('reserved block type'); + } + + if (last == 1) { + break; + } + } + + if (checksum == 1) { + // Parsed so the frame is consumed whole, but not verified. Every block + // in a .hgtz is its own frame and pyzstd writes none. + at += 4; + } + if (written != out.length) { + throw ZstdException('frame decoded to $written bytes, not ${out.length}'); + } + + return out; + } + + /// Decodes one compressed block, returning the new output position. + int _decodeBlock( + Uint8List src, + int start, + int end, + Uint8List out, + int written, + ) { + var at = _decodeLiterals(src, start, end); + if (at == end) { + throw const ZstdException('block has no sequences section'); + } + + // Number_of_Sequences, one to three bytes. + final first = src[at++]; + int sequences; + if (first == 0) { + sequences = 0; + } else if (first < 128) { + sequences = first; + } else if (first < 255) { + if (at >= end) { + throw const ZstdException('truncated sequence count'); + } + sequences = ((first - 128) << 8) + src[at++]; + } else { + if (at + 2 > end) { + throw const ZstdException('truncated sequence count'); + } + sequences = src[at] + (src[at + 1] << 8) + 0x7F00; + at += 2; + } + + if (sequences == 0) { + return _copyRemainingLiterals(out, written); + } + + if (at >= end) { + throw const ZstdException('truncated compression modes'); + } + final modes = src[at++]; + if (modes & 3 != 0) { + throw const ZstdException('sequence modes reserved bits set'); + } + at = _readSequenceTable( + _literalLengths, + modes >> 6, + literalLengthDefault, + 6, + literalLengthMaxSymbol, + src, + at, + end, + _literalLengthsReady, + ); + _literalLengthsReady = true; + at = _readSequenceTable( + _offsets, + (modes >> 4) & 3, + offsetCodeDefault, + 5, + offsetCodeMaxSymbol, + src, + at, + end, + _offsetsReady, + ); + _offsetsReady = true; + at = _readSequenceTable( + _matchLengths, + (modes >> 2) & 3, + matchLengthDefault, + 6, + matchLengthMaxSymbol, + src, + at, + end, + _matchLengthsReady, + ); + _matchLengthsReady = true; + + return _executeSequences(src, at, end, out, written, sequences); + } + + int _readSequenceTable( + FseTable table, + int mode, + List predefined, + int predefinedLog, + int maxSymbol, + Uint8List src, + int at, + int end, + bool ready, + ) { + switch (mode) { + case 0: + buildPredefined(table, predefined, predefinedLog); + return at; + case 1: + if (at >= end) { + throw const ZstdException('truncated RLE sequence table'); + } + if (src[at] > maxSymbol) { + throw ZstdException('RLE sequence symbol ${src[at]} out of range'); + } + table.setRle(src[at]); + return at + 1; + case 2: + return at + readFseTable(table, src, at, end, maxSymbol); + default: + if (!ready) { + throw const ZstdException('repeat mode with no previous table'); + } + return at; + } + } + + /// Reads the literals section, leaving the literals addressable through + /// [_literalSource]. Returns the offset just past the section. + int _decodeLiterals(Uint8List src, int start, int end) { + if (start >= end) { + throw const ZstdException('empty compressed block'); + } + var at = start; + final header = src[at]; + final type = header & 3; + final sizeFormat = (header >> 2) & 3; + + if (type < 2) { + int regenerated; + if (sizeFormat & 1 == 0) { + regenerated = header >> 3; + at += 1; + } else if (sizeFormat == 1) { + if (at + 2 > end) { + throw const ZstdException('truncated literals header'); + } + regenerated = (header >> 4) | (src[at + 1] << 4); + at += 2; + } else { + if (at + 3 > end) { + throw const ZstdException('truncated literals header'); + } + regenerated = (header >> 4) | (src[at + 1] << 4) | (src[at + 2] << 12); + at += 3; + } + if (regenerated > maxBlockSize) { + throw ZstdException('$regenerated literals is over the maximum'); + } + if (type == 0) { + if (at + regenerated > end) { + throw const ZstdException('truncated raw literals'); + } + // No copy: the literals are already contiguous in the frame. + _literalSource = src; + _literalAt = at; + _literalEnd = at + regenerated; + return at + regenerated; + } + if (at >= end) { + throw const ZstdException('truncated RLE literals'); + } + _literals.fillRange(0, regenerated, src[at]); + _literalSource = _literals; + _literalAt = 0; + _literalEnd = regenerated; + return at + 1; + } + + final int regenerated; + final int compressed; + final int streams; + switch (sizeFormat) { + case 0: + case 1: + if (at + 3 > end) { + throw const ZstdException('truncated literals header'); + } + final value = src[at] | (src[at + 1] << 8) | (src[at + 2] << 16); + regenerated = (value >> 4) & 0x3FF; + compressed = (value >> 14) & 0x3FF; + streams = sizeFormat == 0 ? 1 : 4; + at += 3; + case 2: + if (at + 4 > end) { + throw const ZstdException('truncated literals header'); + } + final value = + src[at] | + (src[at + 1] << 8) | + (src[at + 2] << 16) | + (src[at + 3] << 24); + regenerated = (value >> 4) & 0x3FFF; + compressed = (value >> 18) & 0x3FFF; + streams = 4; + at += 4; + default: + if (at + 5 > end) { + throw const ZstdException('truncated literals header'); + } + final value = + src[at] | + (src[at + 1] << 8) | + (src[at + 2] << 16) | + (src[at + 3] << 24) | + (src[at + 4] << 32); + regenerated = (value >> 4) & 0x3FFFF; + compressed = (value >> 22) & 0x3FFFF; + streams = 4; + at += 5; + } + if (regenerated > maxBlockSize) { + throw ZstdException('$regenerated literals is over the maximum'); + } + if (at + compressed > end) { + throw const ZstdException('truncated compressed literals'); + } + if (_literals.length < regenerated) { + _literals = Uint8List(regenerated); + } + + var streamStart = at; + final streamEnd = at + compressed; + if (type == 2) { + streamStart += _huffman.read(src, at, streamEnd, _bits); + } + + if (streams == 1) { + _huffman.decodeStream( + _bits, + src, + streamStart, + streamEnd, + _literals, + 0, + regenerated, + ); + } else { + if (streamStart + 6 > streamEnd) { + throw const ZstdException('truncated literals jump table'); + } + final one = src[streamStart] | (src[streamStart + 1] << 8); + final two = src[streamStart + 2] | (src[streamStart + 3] << 8); + final three = src[streamStart + 4] | (src[streamStart + 5] << 8); + var from = streamStart + 6; + final four = streamEnd - from - one - two - three; + if (four < 0) { + throw const ZstdException('literals jump table does not add up'); + } + // The first three streams carry a quarter each, rounded up. + final segment = (regenerated + 3) >> 2; + final sizes = [one, two, three, four]; + var outAt = 0; + for (var i = 0; i < 4; i++) { + final count = i == 3 ? regenerated - 3 * segment : segment; + if (count < 0) { + throw const ZstdException('literals do not split into four'); + } + if (count > 0) { + _huffman.decodeStream( + _bits, + src, + from, + from + sizes[i], + _literals, + outAt, + count, + ); + } + from += sizes[i]; + outAt += count; + } + } + + _literalSource = _literals; + _literalAt = 0; + _literalEnd = regenerated; + + return streamEnd; + } + + int _copyRemainingLiterals(Uint8List out, int written) { + final count = _literalEnd - _literalAt; + if (written + count > out.length) { + throw const ZstdException('literals overflow the frame content size'); + } + out.setRange(written, written + count, _literalSource, _literalAt); + _literalAt = _literalEnd; + + return written + count; + } + + int _executeSequences( + Uint8List src, + int start, + int end, + Uint8List out, + int written, + int sequences, + ) { + final bits = _bits; + bits.reset(src, start, end); + + final llSymbol = _literalLengths.symbol; + final llBits = _literalLengths.nbBits; + final llNext = _literalLengths.newState; + final mlSymbol = _matchLengths.symbol; + final mlBits = _matchLengths.nbBits; + final mlNext = _matchLengths.newState; + final ofSymbol = _offsets.symbol; + final ofBits = _offsets.nbBits; + final ofNext = _offsets.newState; + + var llState = bits.read(_literalLengths.log); + var ofState = bits.read(_offsets.log); + var mlState = bits.read(_matchLengths.log); + + final literals = _literalSource; + var literalAt = _literalAt; + final literalEnd = _literalEnd; + var at = written; + + var repeat1 = _repeat1; + var repeat2 = _repeat2; + var repeat3 = _repeat3; + + for (var n = 0; n < sequences; n++) { + // Every table's symbols were bounded when it was built, so a code out + // of range is impossible here and is not re-checked per sequence. + final llCode = llSymbol[llState]; + final mlCode = mlSymbol[mlState]; + final ofCode = ofSymbol[ofState]; + + // Additional bits come off the stream offset first, then match + // length, then literals length. + bits.fill(); + final offsetValue = (1 << ofCode) + bits.read(ofCode); + final matchLength = + matchLengthBaseline[mlCode] + bits.read(matchLengthExtra[mlCode]); + final literalLength = + literalLengthBaseline[llCode] + bits.read(literalLengthExtra[llCode]); + + if (n + 1 < sequences) { + // The three updates together need at most 26 bits, which one fill + // covers, so they peek rather than each testing for a refill. + bits.fill(); + final llTake = llBits[llState]; + llState = llNext[llState] + bits.peek(llTake); + bits.skip(llTake); + final mlTake = mlBits[mlState]; + mlState = mlNext[mlState] + bits.peek(mlTake); + bits.skip(mlTake); + final ofTake = ofBits[ofState]; + ofState = ofNext[ofState] + bits.peek(ofTake); + bits.skip(ofTake); + } + + int offset; + if (offsetValue > 3) { + offset = offsetValue - 3; + repeat3 = repeat2; + repeat2 = repeat1; + repeat1 = offset; + } else { + // One to three is a reference to a recent offset, and a sequence + // with no literals shifts which one it means. + final which = offsetValue - 1 + (literalLength == 0 ? 1 : 0); + if (which == 0) { + offset = repeat1; + } else { + offset = which == 3 ? repeat1 - 1 : (which == 1 ? repeat2 : repeat3); + if (offset == 0) { + throw const ZstdException('repeated offset of zero'); + } + if (which != 1) { + repeat3 = repeat2; + } + repeat2 = repeat1; + repeat1 = offset; + } + } + + if (literalAt + literalLength > literalEnd) { + throw const ZstdException('sequence wants more literals than exist'); + } + if (at + literalLength + matchLength > out.length) { + throw const ZstdException('sequences overflow the content size'); + } + final from = at + literalLength - offset; + if (from < 0) { + throw const ZstdException('match reaches before the output'); + } + + if (literalLength > 0) { + if (literalLength < 16) { + for (var i = 0; i < literalLength; i++) { + out[at++] = literals[literalAt++]; + } + } else { + out.setRange(at, at + literalLength, literals, literalAt); + at += literalLength; + literalAt += literalLength; + } + } + + if (matchLength < 16 || offset < matchLength) { + // A short or overlapping match is quicker byte by byte than through + // a range copy, and the median match here is nine bytes. + var source = from; + for (var i = 0; i < matchLength; i++) { + out[at++] = out[source++]; + } + } else { + out.setRange(at, at + matchLength, out, from); + at += matchLength; + } + } + + if (bits.exhausted) { + throw const ZstdException('sequence stream ran past its end'); + } + + _literalAt = literalAt; + _repeat1 = repeat1; + _repeat2 = repeat2; + _repeat3 = repeat3; + + return _copyRemainingLiterals(out, at); + } +} + +/// Decodes one zstd frame into a new list. +/// +/// Allocates a decoder per call. Decoding many frames, which is what reading +/// a tile does, should hold a [ZstdDecoder] instead. +Uint8List zstdDecode(Uint8List src, {int? expectedSize}) => + ZstdDecoder().decode(src, expectedSize: expectedSize); diff --git a/lib/src/exception.dart b/lib/src/exception.dart new file mode 100644 index 0000000..f4496f6 --- /dev/null +++ b/lib/src/exception.dart @@ -0,0 +1,9 @@ +/// Thrown when a frame is malformed, or uses a feature this decoder omits. +class ZstdException implements Exception { + const ZstdException(this.message); + + final String message; + + @override + String toString() => 'ZstdException: $message'; +} diff --git a/lib/src/fse.dart b/lib/src/fse.dart new file mode 100644 index 0000000..6c2a067 --- /dev/null +++ b/lib/src/fse.dart @@ -0,0 +1,230 @@ +import 'dart:typed_data'; + +import 'bit_reader.dart'; +import 'exception.dart'; + +/// An FSE decoding table: one entry per state, giving the symbol that state +/// emits, how many bits to read next, and the base the read is added to. +/// +/// The three arrays are parallel and sized for the largest accuracy log the +/// format allows, so a table is built in place and never reallocated. +class FseTable { + FseTable(this.maxLog) + : symbol = Uint8List(1 << maxLog), + nbBits = Uint8List(1 << maxLog), + newState = Uint16List(1 << maxLog), + _spread = Uint8List(1 << maxLog); + + final int maxLog; + + final Uint8List symbol; + final Uint8List nbBits; + final Uint16List newState; + + /// Accuracy log. Zero for an RLE table, which has a single state. + int log = 0; + + /// The normalised distribution this table was last built from. Written by + /// [readFseTable] and [buildPredefined], read by [build]. + final Int16List distribution = Int16List(256); + + final Uint8List _spread; + final Int32List _next = Int32List(256); + + /// Makes this a table with one state emitting [value] and reading no bits. + void setRle(int value) { + log = 0; + symbol[0] = value; + nbBits[0] = 0; + newState[0] = 0; + } + + /// Builds the table from [distribution], where a count of -1 is the "less + /// than one in the table" symbol the format reserves. + void build(int maxSymbol, int accuracyLog) { + if (accuracyLog > maxLog) { + throw ZstdException('FSE accuracy log $accuracyLog above $maxLog'); + } + final size = 1 << accuracyLog; + log = accuracyLog; + + // Low-probability symbols go at the top of the table, working down, and + // then count as one when the states are numbered. + var highThreshold = size - 1; + for (var s = 0; s <= maxSymbol; s++) { + final count = distribution[s]; + if (count == -1) { + _spread[highThreshold--] = s; + _next[s] = 1; + } else { + _next[s] = count; + } + } + + // The step is coprime with the table size, so walking it visits every + // slot exactly once. Skipping the low-probability slots at the top is + // what keeps those symbols where they were just put. + final mask = size - 1; + final step = (size >> 1) + (size >> 3) + 3; + var position = 0; + for (var s = 0; s <= maxSymbol; s++) { + final count = distribution[s]; + for (var i = 0; i < count; i++) { + _spread[position] = s; + position = (position + step) & mask; + while (position > highThreshold) { + position = (position + step) & mask; + } + } + } + if (position != 0) { + throw const ZstdException('FSE distribution does not fill its table'); + } + + for (var state = 0; state < size; state++) { + final s = _spread[state]; + final nth = _next[s]++; + final bits = accuracyLog - (nth.bitLength - 1); + symbol[state] = s; + nbBits[state] = bits; + newState[state] = (nth << bits) - size; + } + } +} + +/// Reads an FSE table description into [table.distribution] and builds it. +/// +/// Returns the number of bytes the description occupied. The description is a +/// forward, least-significant-bit-first bitstream: an accuracy log, then one +/// value per symbol coded in as few bits as the remaining probability mass +/// allows, with a repeat flag after any zero. +int readFseTable( + FseTable table, + Uint8List src, + int start, + int end, + int maxSymbol, +) { + final bits = ForwardBitReader(src, end, start); + final counts = table.distribution; + for (var s = 0; s <= maxSymbol; s++) { + counts[s] = 0; + } + + final accuracyLog = (bits.peek32() & 0xF) + 5; + bits.skip(4); + if (accuracyLog > table.maxLog) { + throw ZstdException('FSE accuracy log $accuracyLog above ${table.maxLog}'); + } + + var remaining = (1 << accuracyLog) + 1; + var threshold = 1 << accuracyLog; + var nbBits = accuracyLog + 1; + var symbol = 0; + var previousZero = false; + + while (remaining > 1 && symbol <= maxSymbol) { + if (previousZero) { + var repeat = 0; + while (true) { + final flag = bits.peek32() & 3; + bits.skip(2); + repeat += flag; + if (flag != 3) { + break; + } + } + while (repeat-- > 0) { + if (symbol > maxSymbol) { + throw const ZstdException('FSE zero run past the last symbol'); + } + counts[symbol++] = 0; + } + previousZero = false; + continue; + } + + final value = bits.peek32(); + final small = (2 * threshold - 1) - remaining; + int count; + if ((value & (threshold - 1)) < small) { + count = value & (threshold - 1); + bits.skip(nbBits - 1); + } else { + count = value & (2 * threshold - 1); + if (count >= threshold) { + count -= small; + } + bits.skip(nbBits); + } + + count -= 1; + remaining -= count < 0 ? -count : count; + counts[symbol++] = count; + previousZero = count == 0; + while (remaining < threshold) { + nbBits--; + threshold >>= 1; + } + } + if (remaining != 1) { + throw const ZstdException('FSE distribution is short'); + } + + table.build(symbol - 1, accuracyLog); + + return bits.bytesUsed(start); +} + +/// Builds [table] from one of the format's fixed distributions. +void buildPredefined(FseTable table, List values, int accuracyLog) { + final counts = table.distribution; + for (var s = 0; s < values.length; s++) { + counts[s] = values[s]; + } + table.build(values.length - 1, accuracyLog); +} + +/// Decodes a whole FSE bitstream with the two interleaved states the format +/// uses, appending symbols to [out] from [outStart] and returning the count. +/// +/// The symbol count is not recorded anywhere, so decoding runs until the +/// stream is spent. This is how the Huffman weights are coded. +int fseDecodeInterleaved( + FseTable table, + ReverseBitReader bits, + Uint8List out, + int outStart, + int outLimit, +) { + final symbol = table.symbol; + final nbBits = table.nbBits; + final newState = table.newState; + final log = table.log; + + var state1 = bits.read(log); + var state2 = bits.read(log); + var at = outStart; + + while (true) { + if (at + 2 > outLimit) { + throw const ZstdException('FSE stream longer than its output allows'); + } + out[at++] = symbol[state1]; + bits.fill(); + state1 = newState[state1] + bits.read(nbBits[state1]); + if (bits.exhausted) { + out[at++] = symbol[state2]; + break; + } + out[at++] = symbol[state2]; + bits.fill(); + state2 = newState[state2] + bits.read(nbBits[state2]); + if (bits.exhausted) { + out[at++] = symbol[state1]; + break; + } + } + + return at - outStart; +} diff --git a/lib/src/huffman.dart b/lib/src/huffman.dart new file mode 100644 index 0000000..2d66b4c --- /dev/null +++ b/lib/src/huffman.dart @@ -0,0 +1,201 @@ +import 'dart:typed_data'; + +import 'bit_reader.dart'; +import 'exception.dart'; +import 'fse.dart'; + +/// The Huffman table for a block's literals, flattened so that decoding one +/// literal is a peek of [log] bits and one array index. +/// +/// Weights, not code lengths, are what the format stores: a symbol of weight +/// `w` occupies `1 << (w - 1)` of the table's slots, and the last symbol's +/// weight is left out and recovered from the slack. +class HuffmanTable { + /// The largest table log the format allows for literals. + static const int maxLog = 12; + + static const int _maxSymbols = 256; + + /// One entry per table slot: the symbol in the low byte, the code length + /// in the next. One load per literal rather than two. + final Uint16List entries = Uint16List(1 << maxLog); + + /// Bits to peek per literal. Zero until a table has been read. + int log = 0; + + final Uint8List _weights = Uint8List(_maxSymbols); + final Int32List _rankCount = Int32List(maxLog + 2); + final Int32List _rankStart = Int32List(maxLog + 2); + + /// The FSE table the weights are coded with, when they are coded at all. + final FseTable _weightTable = FseTable(6); + + /// Reads a Huffman tree description starting at [start], returning the + /// number of bytes it occupied. + int read(Uint8List src, int start, int end, ReverseBitReader bitReader) { + if (start >= end) { + throw const ZstdException('literals section has no tree'); + } + final header = src[start]; + int nbWeights; + int used; + if (header >= 128) { + // Directly encoded: one nibble per weight, high nibble first. + nbWeights = header - 127; + final packed = (nbWeights + 1) >> 1; + if (start + 1 + packed > end) { + throw const ZstdException('truncated Huffman weights'); + } + for (var i = 0; i < nbWeights; i += 2) { + final byte = src[start + 1 + (i >> 1)]; + _weights[i] = byte >> 4; + if (i + 1 < nbWeights) { + _weights[i + 1] = byte & 0xF; + } + } + used = 1 + packed; + } else { + final size = header; + if (start + 1 + size > end) { + throw const ZstdException('truncated Huffman weight stream'); + } + final read = readFseTable( + _weightTable, + src, + start + 1, + start + 1 + size, + 255, + ); + bitReader.reset(src, start + 1 + read, start + 1 + size); + nbWeights = fseDecodeInterleaved( + _weightTable, + bitReader, + _weights, + 0, + _maxSymbols, + ); + used = 1 + size; + } + + _buildFromWeights(nbWeights); + + return used; + } + + void _buildFromWeights(int nbWeights) { + if (nbWeights < 1 || nbWeights >= _maxSymbols) { + throw ZstdException('Huffman table with $nbWeights weights'); + } + var total = 0; + for (var i = 0; i < nbWeights; i++) { + final weight = _weights[i]; + if (weight > maxLog) { + throw ZstdException('Huffman weight $weight out of range'); + } + if (weight > 0) { + total += 1 << (weight - 1); + } + } + if (total == 0) { + throw const ZstdException('Huffman table with no used symbol'); + } + final tableLog = total.bitLength; + if (tableLog > maxLog) { + throw ZstdException('Huffman table log $tableLog above $maxLog'); + } + final size = 1 << tableLog; + final rest = size - total; + if (rest & (rest - 1) != 0) { + throw const ZstdException('Huffman weights do not complete a tree'); + } + // The weight left out of the description is whatever fills the slack. + _weights[nbWeights] = rest.bitLength; + final nbSymbols = nbWeights + 1; + log = tableLog; + + for (var n = 0; n < _rankCount.length; n++) { + _rankCount[n] = 0; + } + for (var s = 0; s < nbSymbols; s++) { + final weight = _weights[s]; + if (weight > 0) { + _rankCount[tableLog + 1 - weight]++; + } + } + // Codes are handed out from the lowest weight up, so the longest codes + // take the lowest table indices and the shortest take the highest. + var start = 0; + for (var n = tableLog; n >= 1; n--) { + _rankStart[n] = start; + start += _rankCount[n] << (tableLog - n); + } + if (start != size) { + throw const ZstdException('Huffman ranks do not fill the table'); + } + + // Within one code length, symbols keep their natural order. + for (var s = 0; s < nbSymbols; s++) { + final weight = _weights[s]; + if (weight == 0) { + continue; + } + final length = tableLog + 1 - weight; + final span = 1 << (weight - 1); + var at = _rankStart[length]; + _rankStart[length] = at + span; + final entry = s | (length << 8); + for (var i = 0; i < span; i++) { + entries[at++] = entry; + } + } + } + + /// Decodes [count] literals from the reverse bitstream in [src] between + /// [start] and [end] into [out] at [outStart]. + void decodeStream( + ReverseBitReader reader, + Uint8List src, + int start, + int end, + Uint8List out, + int outStart, + int count, + ) { + if (log == 0) { + throw const ZstdException('literals coded with no Huffman table'); + } + reader.reset(src, start, end); + final table = entries; + final peekBits = log; + var at = outStart; + final stop = outStart + count; + // One fill leaves at least 49 bits, which covers four literals at the + // format's twelve-bit maximum, so the inner steps are a peek and an + // index with nothing between them. + final groups = stop - 4; + while (at <= groups) { + reader.fill(); + var entry = table[reader.peek(peekBits)]; + out[at++] = entry & 0xFF; + reader.skip(entry >> 8); + entry = table[reader.peek(peekBits)]; + out[at++] = entry & 0xFF; + reader.skip(entry >> 8); + entry = table[reader.peek(peekBits)]; + out[at++] = entry & 0xFF; + reader.skip(entry >> 8); + entry = table[reader.peek(peekBits)]; + out[at++] = entry & 0xFF; + reader.skip(entry >> 8); + } + while (at < stop) { + reader.fill(); + final entry = table[reader.peek(peekBits)]; + out[at++] = entry & 0xFF; + reader.skip(entry >> 8); + } + if (reader.exhausted) { + throw const ZstdException('Huffman stream ran past its end'); + } + } +} diff --git a/lib/src/tables.dart b/lib/src/tables.dart new file mode 100644 index 0000000..50ea8cf --- /dev/null +++ b/lib/src/tables.dart @@ -0,0 +1,67 @@ +/// The fixed tables RFC 8878 defines for sequence decoding. +library; + +import 'dart:typed_data'; + +/// Literals length code to the smallest length it can mean. +final Int32List literalLengthBaseline = Int32List.fromList(const [ + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, // + 16, 18, 20, 22, 24, 28, 32, 40, // + 48, 64, 128, 256, 512, 1024, 2048, 4096, // + 8192, 16384, 32768, 65536, +]); + +/// Extra bits read after a literals length code. +final Uint8List literalLengthExtra = Uint8List.fromList(const [ + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // + 1, 1, 1, 1, 2, 2, 3, 3, // + 4, 6, 7, 8, 9, 10, 11, 12, // + 13, 14, 15, 16, +]); + +/// Match length code to the smallest length it can mean. +final Int32List matchLengthBaseline = Int32List.fromList(const [ + 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, // + 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, // + 35, 37, 39, 41, 43, 47, 51, 59, // + 67, 83, 99, 131, 259, 515, 1027, 2051, // + 4099, 8195, 16387, 32771, 65539, +]); + +/// Extra bits read after a match length code. +final Uint8List matchLengthExtra = Uint8List.fromList(const [ + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // + 1, 1, 1, 1, 2, 2, 3, 3, // + 4, 4, 5, 7, 8, 9, 10, 11, // + 12, 13, 14, 15, 16, +]); + +/// The default literals length distribution, accuracy log 6. +const List literalLengthDefault = [ + 4, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, // + 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 2, 1, 1, 1, 1, 1, // + -1, -1, -1, -1, +]; + +/// The default match length distribution, accuracy log 6. +const List matchLengthDefault = [ + 1, 4, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, // + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -1, -1, // + -1, -1, -1, -1, -1, +]; + +/// The default offset code distribution, accuracy log 5. +const List offsetCodeDefault = [ + 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, // + 1, 1, 1, 1, 1, 1, 1, 1, -1, -1, -1, -1, -1, +]; + +const int literalLengthMaxLog = 9; +const int matchLengthMaxLog = 9; +const int offsetCodeMaxLog = 8; + +const int literalLengthMaxSymbol = 35; +const int matchLengthMaxSymbol = 52; +const int offsetCodeMaxSymbol = 31; diff --git a/pubspec.yaml b/pubspec.yaml new file mode 100644 index 0000000..952cf4c --- /dev/null +++ b/pubspec.yaml @@ -0,0 +1,11 @@ +name: pure_zstd +description: A pure Dart zstd decompressor, with no native dependency. +version: 0.0.1 +publish_to: none + +environment: + sdk: ^3.8.0 + +dev_dependencies: + lints: ^6.0.0 + test: ^1.25.0 diff --git a/test/fixtures/N51E000.hgtz b/test/fixtures/N51E000.hgtz new file mode 100644 index 0000000000000000000000000000000000000000..8cd6cf8c5127c201157ab0e1e35b6c5c61835978 GIT binary patch literal 33633 zcmeF3k{Lj;a^S_@wZ9)$YXsDtYwDF;AcMh`Y4BUVJ{ZWwqAYU0k?kDhb_Da}=hP|P} z>XxDsu1v!lgcF1lg|kaHm-KS@sZ)yZ5agBa^Ih7x#O0xFM<3xf$V=U4yVP?DFNew< z^@S@SM!Gz^lymXRgU*hE!r2fl-KV?cb03!r)O6W*$ucElAWFKdyFQQ4-$7ni`dmH_ zfhdSd20^5C*Sdc&a*@nQyxS$twCJdvH{0d&%l@iu?NXm%E>)XlHq7@byHVT1r8dK? zW2u@fQ*}PJ z>{M+Hmrof+9o43$Tzur&aoQ>_r5T1&)l{Z9eCXNlwG~`SG7LJZ=1q2aQCUISGA>0K z`chR{Cd0g&S-#qmE(IBS9hIggmAo@qZrY+Qc^SG=l~g9NynR{r+Cnb589E&m^Tw{c zU$QK;`CYOzw52MtjMaHdvkbJkT{1JYI?7Frxp>pFG_*Nf>}hzvM|qTsiO}OGhp9_* z6r{sPUCKp7=;D)P)#*7B(+5Xil|L7Gf}eD+cFP``-aTqtE+|49pV+Oo!tRycI{LEw znFtMjVwhSdyLEcqs98Cm2z7jdtXe+1PI~#MQ8|wY6@J3KDwtg+y>L{ooJ)i26m#d|J29j&It}CIh^O(VBRdNOBS<1n zjw+L9hut|^QOo<@-V!1Vxyj?h&K%9CC3o*<5)=$^$^F9)98IXjeXnEjzw|-Lox?U9 z4X6couR8Hrdbi}}VM~rW)X#lSZ}Db&i{#J4rW~J9^X{I{#53qMlFNn-IjT`}`yR() zAbQE<{9#>=O4O{oN1d24J$G`(uqH=2YG&Wv8+akK$qB=%9A&6!clT#v6!f^sQNxNH zP}J1E+p*|hx~nAcuq;P0YSP`UPIQ(|W_w{xaD?qETh;D7h#o`-@*q(;QJz#L`C&F{ zG^#gRB!%CBCxs^^dd0fOy2xs=`(&GDn`(R1CW8JjsyPB!DG+T{&SDC^_zG2H1dmbH z6KyQca0gi! z%-Fk_vzVipt(euAxtNKVftaqCmYCX@Pca2CnK6kmA7Vmd{A0XgoMNnFOk#9mRAOXe zL}U14IAT`O57GQiar`4*C^DQ!N-8oq5I0APE7G$PPolX*2=~%OqxYcMMu;q7IHNb9 zIT>O@FjUd|f8$SMSz?4KQT)bxLmZ$)?ts^X+s=x2ia8o3JB_)7Zqvs!ioS;S8A@M` z4o9nG#W}@rM_275V8woe5gbL)i^qyJjl~i{(u?DOFM>uJL7+&Xi^h%?_#2NIjW|Ne z5Q{JZtCwsr5>pYK6>p;#jTP%k&krBC&xFso&zR4s&xp^k&ydfc z&wx+APoGb(&v&05ADBmlzfWs`(NKtukUYL9_UMktNtYkZH-)?#i8uOXzd>b*XDUDV!4DuhG7bH259Ia)ld) z&T{{iaZ6cJ>Rv#tn82t<`xCDLq9h>U$?bcSt0Ap%^7NAe)f3U zw$)piBR`v)6?|6vS?sCTR$*m~tTs1g_>BEC+*7Wt$VwksX>L+bGc<4KDbSW}rHw2% zH)g1bnV0uuYD>0KN0ylz6;!LuGkX%Z#aJoV=-x4 zC8Q3w9kZ)hCI_=C((I+UIc7nm-b;;U(W@<}`Ry zmvKv{x-$Z&cV9X!ciM)=9c{l1*CvP$B{;?`Qr5<|4c<09T_8VNIc#8GAoKElZ9#Sv zmu(wT8`n0_+0d0e(D};DqRng||COEv(NWCL{)7fNH?VqibF?G-Dr7mb3<*URBMXuF zNC+|qnT5`XIfLo=A6OLbd8V z8($-V(T8fqdE1B5Tfbe%o961N?J+I4Br!rI?D4isVS}}W}7Kp zHK8dsUk#Tbd9`$)DQPv%d?DXwY*P%+YQ#MHXEbV4Ur#jZh})kYp0o31HA*I-mAkt2 zdGw;)gZdd6G_8KzqKPt`J*wUW!< zAkn%p)uexlopExII;|iIiI_s`MNxCJZC}EqdY*G zBRhRqbB{{>mOU_i+#@jyXl0U3s67&>)Cj7$jVV3i&-Jv*IcD4D_0-DQX6;rgS!YRk z3;r6Ddc+0j1@XoOsC=&Utcev<$*+7eOIQ;_rN>`RHH&|crwyFuWX7WJ(xf#R%3a1; zZPWNP$HbHpSbtUoOiNWWijffDzWdOrvn(})o!V8f!Ulxm@-)4bJGVgv-(?|?tL0IEC9Sxq5^pMWJ7MXwavil>dm z6!o{7Obm|zYkh{LnzWVp=L|Cs>3Q7lK8Y!A!Y9dm%%RAYYEoF_ilSE*aed^hqDKG@ zA7*moBoEO!_8~_03=I!q05<(6zG*5R0xER_DAzQ_eqx5A2GpxMF+ouUD%Yx}1Z6R# z*(qh7CfQFoF(lc?F;)9%NbHW2HPf+g8o(t@UKFRV;#QhDEwxV-fFq4w^j=@Vtt4}* zV{hJnR|-`aq%Y%ElsPH2m*qb!wOQz^FX>j0`J-dk)W1?{rqE4a)GaR)F11VLA1l>Y zXs<8imYX@zu`>^HmHJX>q0jG@ojESGlLb?+(7!t`lGJB#d)Hhfu20YX zZt{goq44BQv|rB)+rq!g2xq^B7h5Jnc?ea%{=N9QL9KB{mjn@&JSg0tlDj2yY;4Wh zQ$#cm0@tbJY{}3XD|FTt;mAvaYgBTyq>qi-IP-~6=6!^#RI;_CX^k>E6Nq5uMZgs* zSz1!ZM%J7Th0h?taG6S`mK3d#LZ@-zEyx?VWF^1jp2ujdJ;p->i2TVdi2&}67JqpONC#=ot%c-ll17UMI@ZPN0ySlP&;u9tH%^8 z>y&<)Hr(fKrR>E25pAc;54=RpeU4W0&Z@*|)qOTG1H#e|(~A2=k4ksYRrjJ3r8DUJ z0R2DE!08w7=A_&iTlMbt8Cr=uE0_0yn$E9un-FU9Zj66|nJWttG^o5#PB`+wtK~3l} z-$k}7ehR?O$B>JZ_rMRpqQ`hWOX`jrfZ172H%;h{EvCf{C7+fCmy=H8>@y0?V!LAm zXtv}n!aaAzMDuUp-mZe8nRz2{$KB_mkei8PmlF~HysvPp-DjdXH{;$e4I(CaRdC~7 z9?`6uv14Z+;PvLhwRfM2X55T=JD=>ai>8N+!l#sXSw+)AO5VfeciUfo-d;>%)dI3QE~WoscOa30rS zdEo+x4*ZX1Td4V{bZZ}}~xjjSx zzHjrY^0@&7f9!*^5{Q*9?Ox-O<0}1TG}KuR#7vhOrqRx^n0_$o?<@skq)WM1f6MVT zeR8x- zZg1Ho)AL4koH;=hbbwfsV7~&9=K(Tp*M$v4M)$FMLcy5@R8Th|=gj=qU~-44nS?Ia zS1K+^rQM|@Or@P|QKY~ISORl@iC)Gssua6C`AfJQ^K3%EndUEnRF2swuQQd%vywEu zQO<1(F1FOPMP~~dwzxE@p#n!ezse^jVZ7~B774sVnT}e1$(v+M`o1JF#95YD>SiuBZcE`U>-%D^#=czX=3_DqKjLMIT5+l z`{uV~6T>%RIF21=ZRA|9`)|=Ex^G0fILw%dkpsOBzlEA;zY)T*H!!>+TY4>j^EXj{ zBhbYz#?XoU)NA;gyNU7}J{-#qgFiB}SMxVV6ZyBe#9r0kY|lvvBR}*i{$_ea7Y)sY zjc+&$HD4`x9SdAigvP=8HynhTt`?7ukQaZ!L9osZ8=;1)1+Sy}i&?N6ta-yysP5|L z(c$ZhX0Qe9^M2}!`6YJ>gx}usZ6pEeN9Sh__KaVOvbI238pf2)iTFM)8DYF?Drh zXY0i8;#-J~>bv!MrF=Qg(jupD`v)Qa!3)(5rcrnkJX zEO~9xiubcJ2Lq{+Hm}NM)>(MkQXPG^NrO9iOmjAkAfu;7;FWxcBr9Lk1 zJ*3!aq+SJ;d>WZqg*ASkS=~P>u5X-~+dl|b7y6VsGbNYwsP?BU_l`y`v6%XMg2hl3 z!U$Cr3~V&TR0;{SLvaWtR7o(fcN9~8k6#>gLkOdCgMsa!m_i{wcTfYthDr_A2;(Rw z|NhZpkQ+gciVIc=V=E?8_((g5i$F)c4^{|cDJK0Mw>Yr3i8>7mmI-4jCQ*pX9T?f% zJoOEh3}YxJ{vK;FP_;R8>J}^-MpsOv5KB7{x7l}UA1o9`TTJ*pX0hLG^UJA4Fn<_z zF@Zu%ZokH6>8U|5cNk?c{`Y8$e(ufmQ;lGbuqoQj_*0eOV%mP(&B#-ke;=;;zM`am zec0>!cU}0ak8(8**?;LE)AX=-;`OKPY$XWUd1)il@UY# z+t#vTf&6@FD)aea-p5m@EptT!S$1hCQ~fY^;_;`|e?=0Rf2k`|`7rC_(b#IT!i~(h z)RZZIm^pFxZ53UiMJ8OT%9K4!`?w3WQm)`4qb?O?pbt|gZhu2xGt~xfpekU)R!uv#ym$Pm%~OF}QVQe&?4 zU@E{3ELxP;qia#gJ(qqk65s?DE&>XUwi0(P^5A=bJy@tHw?}8OVsF;(;A?<2SfB`~ zzubzE*;faR0Ty8XqU;`Ri;AjQql2md6EJTPP>!?}akEMXB>@It?xM^dt;KS;S>c1+ z039%A5m24E|+A!&c({=RxY z2l5DMWwd&H7L+47k+(AeatUc>d>Q^sGKY2|erEvmDx`_g@{#{4`)WL7rwe2o(!gjD z&i^%gaol641@tncj?w&)FDU!#_{*JIkXguQMze4}$?V*5t({LGqmXJw(?{N`tRSEt zF97L zkK9+8S7SHZY^Tf{2-Z{57l~(YNP}_1hJs$;hG8}!Jip<{ViqHwoDzP+W*p-GO?8TY z9k0{N@tf=o<@*SkUWVVClW6u4{JmPT=qYH6n;hY2k`a4v$b}*(dsP_YNPDq=vxW~~ z{zf}ZHR!5XpYd_OYFS-gKX!d9)RnzH=Hqf-v*Zs4EJ z!1WFvo2&ZO1>2*_z^P7`^)Eh_SGB7@*AHC-8#~R`YkW+vYF6iM54i$UJJr@peGIRv zR_E3ab_0An#nmBQOPUR z-UOFecdeXDsuE5z!y;NS*tL|`gJR|SG(B$9lPTPdPRGQp^yM*CA z-Ls7DjduC9>Zbgr>;`&Md{cOne*?M6xyic8xJkQ7xkB zz6rj0dlPU2x_NWsbK`yEdEwf=-QH`)#VnSAJVoZS~XbDKRcujw5hE&ndkBR zCJ*>fI`h7IWz0s6KYawsNQ|xy+i{HES1;#VgjUPUd%1(QFv49vmM`(fUDOk3w<$aZ+$3Jh+?4m!v~mNtbYr3k6oNy)oXpZylqzqcODlzC~k*ctCo{ zd-S-=yZ5-syY;xryY{%qyYx8AJNG!rJM}oqJNCSif7Rn4@6cl}Z{K4lZ`WfhZ`)%d zZ_{HfZ{1@hZ`Jcs{$-D)yk(DtyhV?>ym=2xTWV+h!V$B0>aE4zMT(V1@?ucYr(vVj zw#FW#)#{`C`Ua!9>I1O4z^4?WDRJO@MV93p(TFF4)tDCf4ynWw=o<{?xeh5VtTU@M z=Dpgi6RTC`VV(6#^DLh2yFZx^NiMAZc#>SiTGjvbBo2&~t{0xC^&}LxvZ&^rr#x)Q zG1RDLn|FPHzBQDrp2!EzYjeZQK)@`huN}End<~ua>1Ye(GK>>%U^XPI?z=U|twx{! zUOdc|EI`j7*6!v#xs?A!*vWdhGD(1rL8RTudw40~3%S!LWrNa+X(M-@bIKL`^=MaR zq0;PWE%&GAX!`lPC<&!psC_W#Gs(^3x?={U&>Uchp-Y!9!@XR^{qyO5gthb^*oEUZ9 z7H)nbl|-ce2r$5|QJZanW*Vu)B()v@`qhq_Zu2%%NhJ`eg#(nYU{rUTvzbCFK1o#` zz;Nq9hI{WX$;-H;iLZq!!)&X~20Z}D^UBj} zV4F+x2#I~BlEOA6LUa>ztemq$BSIJw!#KgWO(jA=mt!!>wN3GtKQm2Z)T^04F->K( z5TW;_@`XtfZx5Aq0{>7JLhDQU3u7YQa4MMuhM^>c+Ly8yMoB#KR167(L(vGOFVGi; zL_Cj_cky?FVF_0T*^^lv-b@gW)On z66G5HE|zbUcwe4_?kVRIg$;fsmO&I}U%G?lDaR7|8lEebP!x4vyo2f~+Y*@#9v2pA z6n0;vgW@U666qT5F6N)ei{20i*;A$^5*yq~%&EwoUO$HyrwmKPYdEf$jgdclJsiYN z>6VCWaJVp2BZqolIXpk5T_Rk=-o@~ZZ0mjL@a&X&iNFTC62l;}qSwfQ=ag~@e+|nO zLnt!4SIgn)DftrK+JKq^`zaYDE~Zz>f%TN+Uql{k!p_CFnRYq!Sp1SEG#)l!_sX}4 zcIocu;^Gn<0_(D~^=+VC3_WVQSO9y#TI^o>*3mB99R^;sgI~gG?aX{X)BX%S6u-y; zYr#I*8TnSz&fgteoWBK2!3ym3d@E_^LJ!)`&B45|Ogk;#a@yIu{lIexFdZz>PR+NB zb|!RR{G0}i5Bp%JEvB6c-D^8r2s!BXx0CWMq@BFm4LoZP zS?>0-6Zg%h{Smq=ewGt5*6m~`>Lu$Iw|G&Us zCWqV?8Mm5p<&Q>xs(t504{0wFZdK*V9u0i~f0FVL_af?6Q4acO@Tcm}+}Xj^Irvsq zuJ}>kx2kb&?qKg6bo)ZC@KNtirSDwJ!Q#2+t(aW?qpokI&|K!h$hqU~b2-SP&Yy}u zv;GHP&#i8s$>lt1`&Km0njBP}8{hKCWj$*BDfgWfJ;*)RzI`f}@u=xr{^yQeEPq>RRW2?3&--)v+jhG@q35!<_9a(9cP+Q^>s<=7hVNC+qq^^>#?ba2b%Re;cXY5M zd3A-W=eM*#6`WX~KLeZJ^Vw0CBxnqkAoNfb!GP6dN5ztWJM?l~`;|3KFOsM2w^?Pe7w;Wqu@^@v!#`RrFnst^x9UES9cV)uHCY?!F-@L|L z!D@|s{k0wKHQGw}s+es)vtArvITRiI2ynd>vCS*i)u?2dOFkG101D0)!mNu6I1uj+ zU;%cwg0{KEIwKXoW?vt)1pxJDo5QRV2N)8z2lWBww|utQ#o8JbC9`@56#+mw+Ga6p z;{yJK+(A)*{wNW1u_nGzm*8vi@^tQ>x>KdO) zW`69u1c=;5|GLRMO+5W@D*B!@oFtq$d?&Tfx6hz2^`7&Z?fM{m>rd}guh13cd+czm z@J-SyhEvK@%=cgIBUmDS{nG=ir5TX>P0QNLjc0B-LKE3LV<6|7FKd=|&$x3)Ct`N` zKn^#JYZe>)d)a@+19v(=HaGQa=63v5*;9ZI^#x>kQ@dui!RMCUIBvF612Vm-Su?fc z?3jo*JRjt9Orng612M`vNlz$VYIauwG!;#HC&X;fLY@ zwQe+hG*UlCc_88N;hLB<0<%{^9+~r2qpNg%%Ev9JrC=F$Z2p$JD}8;$$0_Jb!IIoD z&0E~A$o20&_CbvWi?E}Gz`aht^{+nGLG=X-az{CVBb~3-8+|N-Y72hC4$T9rI*rz= zd`yCB3g+bwX#(RqmDWpq41%f(=3oa40dAeb>$yHUK@|nF0s|?prX+~&VvzkgNE!*k z&={swzGEs00)}>jC9Y$NORvl(jU}&kuf!&mCAqFPr6m^ch23T5W0FhHKi(vlIM!`H zy@}uA$g~MB(Rvd~cv>{^E>RxOc z$#cUmOT68ZGl#WGMD^+1lKkag6$K69oJ z!Oe?;D{`@(r5=nNJN*^Df`H+&T+C-F9wT*5v%-525c~xf<5}_nz*L$6^VAbA#zlXY zctL&uIF;ja)Y_%kk`Y4#YZb2JvNf*8YjxTwwm2H@=|BAg4+ zhCk(^IQ!@^c>IH%i`@U?Sb;K}m5a<@@;yWzE@1*77g=~oUV)4*6PM41D+y(sM}hoV z>=;n;hkX9o-^Q&g>YL5AY75QA`JA$w#w{vpo6W{P+n9akGs&(VH>s#;Hq`y0 z3$RlN_%O30#uX|`nhnN)#dOGfmK8iMQ&H5cuT@oOGS0h|^=4eMqM%uCtkTBhGw)oM z`?zRDUbC(i;Jqa94rDou3svMc>x=;_&y%+$%W|B*BD+~ztD?|YoA*bwM1wPJR?^a@Sm zZ60HQBw+#iKrX#VQ-7Pwm@+SZP}PoIAU#`Cd;6&|0G6Twxxkd3tf{`uZVcchRX{Wl zr^je2Z?hT$kO@omk?kSvou>RYvoUF2+@Q(^8!|0WQ+k`xn1n7aTBV0=J?*uo#5TP# zabB#dN-^7yG#5>g?I*@Wbg@_};cT!pTTMY{OxqeKs5R6C$`ObkfEOUWUM&L^g-#r( z+M+X|+g1-8;Ot}VE3WI&D&i_$1%HIpv~dy7fXFvKSh zsK0Uotb}W?bMZ_1kmN3PTc-x{#bB>i@dbL13O^()^&Vwx`QMpke5&HR@!Qs ziNB{5aR*U{SR7jrn;To)F5foKHrMuKVYE#&V{~C&XqtbTR~o-R5r_~(0CE;oB2m)M zW$?!Z;ea{{A^d3^X@vgY)*=cc7$IFd7~5#u;Xe61fDBRwI-toxS9VdqylP!tE_TY) z_~AiEc7UJp0ICRs@_l^IpdGuAUv9MyoD)Ie8{a-?#SY*!9zZ1ZQ`pBh44Sj^`(;;a z>vEP+7{pf&8nXj*jR(+7QGmN%JgCpk?Uz}t1%K*DArzl8sLc+*HXcAiDFdxm%Ah(s zN4nZK_9PYRR|#ZlcyZjyl&|7XiM66R`rD?7)uP$@?UrI$`$^dt{wkBcihDyG#1{8P zr8?HLH1@d)B=$)^VQCB*F@H2wKmI12HXcVbnX+gYnKT|lw96=~4H+Na*mI?itn)EW z!K`9ryUN1P$SCjz$Y@FMu%jJ<2{EJ5`dRb`v+484aH{|(kRoK7g-bF3vyYci~BbY~^zee`3r zhTkOHXg+9WhQ#w|u4pD?n8E0+Xk`xAX&5iiWtH$9u*fjnqevBT9Wa})=pqOev03rC z(8wdC-(ZnO;Pf)`L}4ppIN%|A(V=K$SiTNuWN6oY9+Dnk-6h?>x=Ffyb(M7e>LTg# z)mhT{tCOVDS4T<5udgIueRYs@_-Zd{|J6>??rUe8AF?+;vTZH@o4G@Buhq9$YlbE+ z2V$G9dTdL~rpv|3j@dPLL9wg6BlzoG5&ztd@ik{bkt>K7Tws^NKdz&H%|Y<_RqoLQ za_287sH1buMo{o7$7`Z~XBOnv(Y$6U`0OhCX#Dj~GsvRj^O~t3-&K~^xWG;ZNTZ`{ z%}|i%D)VRzxeWqIcI2<=3UXa#c#YL>8-utzGS)N&pI!kr$m?wp5N$`onyMiCRhrkR zz%~U4wvQ(6_I6dbPE#ysWR2Fej{i_p{T}x~*=-tp2lvS>k88 zpZT6TZRIOQ_0>Ue_PfHkhGF8CaifkFYU?Dmb9W) zUnXIcSS>eC?@8DeZAIygf9ayt#IYp4;Gzb!9L%fDtV^WNCjMHIdgH!z33`V67PGe5 zvkpfBv!MCO5@8)SjSGJh)e`>km^KicCbP_Z*DDRU)0`6#w!M4~S`z4q5kJ3hghq(> z?pme`^inve(2S5cU^lUbI$<`UEluffX>F-(DQ(GZNo|R532zB(@osT$v28JJ(QQ$0 zk#7-i;csDYp>I7PZV(p;6atAjKU|k^P@A_*_>J=Ag8^j+M zIPkf_vo=sK1g}V%Pl!TVvLtEqvEs)ml|sRa%u>m0A^F6xnEO}1Ftxs@ku+uStZ%LnH>KyRpEPu-7M_0@Q`%!_ zTiv{(>{8lb=wud?udG(ue~AAx+DiGI8O0%9pkAOExhGy*RXZQv#fS0FvOhnpDB1oj z8dq9}e*T>QS?LF~k`L!1%2P@0CjlRJU=;W>#XKG#)=yKGYLa;z&l(X!+G+w*V?8Ll zVc=mkvmtS{l$j}EHP*c0Pz|OjI$!n1JQ^R`MeW>td5xOMhssOcA9+@y;;x=HvjR0a zVmb)`BDT`1EjOLzt;rJ8Cg6ORA+1$Vt~<+FlOd*+0Qd#;T6e9Q+E1A>{???4X%akL zpJS;>Z58-A$IP30p1XUHVycl)6_n30l)h3U1WRAhJjxQNPoLF12vFw(`k+bPq;s`H z_Ut(t-oyYk`YOIzD&7PteS>nYS&D;#3_#y`)f6OXs+7Z8^_0q4j3ss|m}f~2@=uIO z4q{F9ei{>d#ER(&m(v;(^5&as@|IK1ie{&&YqFKQ-m(U!NoY=FDRr`%rDb?TQ7L`a zIe_{b0)Z+jl3Sd>5A90JP%lGqcX}#fFpl_#9HlX+lcBKt6Dp!x?2D+kN*z#ZLjiXh zDnc-JTa>v{4b;St*PV)r;1(+oFwXO!I)evZUSbT5epczEpmZ}!3wh?a(17{6o($O$mq#F;XF3q zCjmp6eF8CcGbpbi6;Dzz5I5tTrsa%GQCu{nrH+hM40TV8QdEFG8&{EoGEGyjmoiQh zS0{WfrJp9^`Q$%Sv#%S#E=^h#JE_cF>Oal9PZWS5{jV^5Ql7iSf2w8=?vIfAQ}}jL zn!Cta+3lyK2XAb+O?P?ZGX0f+-!s1iuw zjSa0idJ4BdEaCi>)GY~GLxqmo!k-|9aPCUV79bL4WHoIDSNdDBE8{cBt{gY!1l$we&qQWXd`dOI%7Sp?GOtx7WK=lzxCGcLgikolTQU6V52sVvuV5l+CD;jJVxiQ@g9 zWj8L!nG4dXsel_h@-}9fkMnV6gS2@ol8lvkOS1IGxi|rgUsJ9N^y#Tt>f=4njKi_# zG(DVwk))fX03b=2mp-BC;w(Zz7m_5VPQwwOKA`!^`8mato5XV`D=FCSy9g-lXmcy=<{z=ownTLYvCjK0V9ZIDa zXzDq0QBa1&x2l@6^QLENYB@ippt$)M0HhG<(i1h+oY^VJLq3YB(y-&Ff6!ENW~Csz zi91)hWV=ZV)dT{G6r>?>ttty_2WkGAQqGJNBsZ}EKqPTF%}Z04ShzClQkof>o4(*a27pr6g4hP)aCf zprkDt6WaQs9z#X)jJo?)%gpkUx3h#GLw53*y7ShTnI%i_rxGL#G0A=E4qJ^gi%VYH z;wSWh$sOu8TlF&wmR{xJlk_ghU(_wPYG-~fc{+UKsGDxp%*TiyZ)t$Wb6Oi(!YqfR&H3WZmE>)^|>XTmvnK-j7iAZm4kz)BI${-`a>nNOh; zUp{D5#v`1C2RQFw0I(Gf>XmT`XMWV`eriR*AD=a-RrXXk0}n`MN&~ztNtI?Z;W>#} zG_Nuh8)Gq^UNmRFQg6lp$@(X5_RYgm9=y zQInNO#pVSo69aL zxQS?qaEY!6_XrmWM+m=7&yj1YV!s3OQE zh$9Fha3ioF&>*0NZvbg1$>qdjxl8)cMA+cQtHvhDCBox}i(7Cgtb4<@u|aY%@d$RY z0``KnZoF))lUyJ?dUw$YwuaShm^FTu{F!(tcaaa)ft7C-t4q19L}IZJr(?pzwo0883XYb=wTN!*t^rw0?lqBoQp0qjn=|8RC2 za@QTUA=g+eIhD8vJ6j1k>JHcd4r0kk!o7EAogu5;-W%eL`I0{pcjeCVLngYNH-sA@ zl5oP^hri&Eu5OzRfkq&rl(+-?YZX%4ZECl^^6jm|k6tSuJsbWebq4{PfqG&~$|jtK z^#zPNbjv%hIU5fjG1~k)c;K#te_qFgjf)T9Er!CycWM0NI|giC`8=nE+)Z5UT!KP6 zx@>HH1Zi_aC)#!vKpq_}HZOgi(dOI%9(OzFWk;=znGYXrb|_$5=YX_2KG_)g@X%)6 zja_WN1xa-j*y#Ci(PoB@wQZY&csnv}w0xe@X55ViZcBjZIudQveAsEzLr2B8X+ZcL zA8eF-SZUMlMlQB4{cgU6+Q|7Z)24=wv~4Z;9ene*k@8`rO}QHm+-mn*{`Q~2vi}U0 z{b#W3KZ9leI|s|q|C6TtUz(=$Y%{MFpQk%)9R`L!1?JstODhcxs;8`!nMz{~3oZcb zW*sW%1=uxKN_q?Uo>*-S%twG3AZ?Kt7@4ogwD1kAC;-JdvSxzf82$s(7l^gHdrvPF zf01(X8Xiu1O;;e&?(99fl=6kniGNr~+b`TuI)eJ$R{BrBa4)!=k#uJW&}*3i zd=N3cDX7b|@P1yb(s!usw8(mtE6|BJSz#ANsVfbf!OT?R%6T}6cQvf)~;02EywU7J#K z1#oCyNeo=cmGw;iALJeUb&LEJBn19lsw4<5?Hbmz$0(YzT>S1Vd3o4o%G*CVZ&-Iq z)RoRZDWv!nT*#HSMfNM?C7gdZ*c`x~M)1=xM+pFU{>mGab+jv3e>uqLc=7{D0@&F8Jwz;2RWH3|?x5kTY}tNO5urss|V2v7u2b>6BcJM1EWo`X*S z*iVEkB=`NO{I6hcuLdTnsbeL{H_e>Dh3 z$O;!<>w9xI0JqER4T2CagbT0rj=6jQ=U{Qr69K57Yh7LKZ&4y(b0l)J*VjSv4FiO@8^;{8DsraR%|Y}H9fU|0hXyk) zvai?PLFf%Fgs>a9KEN&VORt3k{~Kxufh=|ohDKy*uYm*i8%hX%H4_CO(#M~6N3@GAySYBg9!bymj(+x9PMQqE*MQF zg8v()R0M+}wJD}}1YIxHAetSTKm?B^dN~jtF)+k zEU-+xj9mJ$RKJwG^m<8eNnnY13Ay-Vv3@amF?r1XZ}$CDmNA9D$@h=6H>!_mQ}5Xn zMsfdM{oA^`er^9xPCoVD-OVplVA%aHg^>{y^3R=QPqO{{cKfZl>+=$n%l(u0kIx?y z{~#v554oW%?yqaFdF2yG|55%L6hi$nK%`!K{d!;z|M=?;iHlMQyFBZO`?cBl{z~ee z*#5!Oeqic!E_KuXnnwPXKaAjgzry!umQ9uG-S?+V_c-70i5{aGe`hN6X`BYWzsK!~ zEV)Ano#Nl0Zaqc_{r>Rp5;F|#-+S$cayw&|nz(s|L#B&lu=+!polyx8U1vEYzK8)U zKS1paS*mg8{~bJ>zXQuZ6x$h;RE^C20w((c!O{UdJ9712hBkJ!IKwvj77+3I|2!dSI@H z3_Gn7U?PX*Ao={AJur(ydT?W>$pTE_{CP;-(D=1*^bSZ}{X04f*J%7Sa@RFAv$c=h zSo{SH5e7~zEbe=p2Pk~Xof+BRV^;rH7Bn;QF0oi`D#3oJ7GZ*_zBcIw;>HFEEJKM1 z6%?>NbOQqp`!#U1d3G9nEz`}^Lt+qDGBB~Zb^7L7vYVlYcq-O@pmuZa z)csnto34k*AeLnyadY6*;aaGhwuf*k=2yShX3MGNHGemC4}n2U$$zzXrvFgD?Z3A# zDPxH+A*pO3$vR^fSu*xLlI&5|VVFTkvS&-m5+Y=mWyTVcEqjU@Au~+bn;6FO``%B_ zv-3Jn?sHCm!o21?T-RrLU)@Fv#ha!SF&uXX>u9WVx70$$<_(G{jLUuMqr<@m6ZenKf7|4lxTa7o0ls+HFM6s;uRO+zfv$ zIEnFUZnT+|Sjz}82rm(wIPyd`D$Vk*C5BuNeVfP>F{j9A&gsdz0J&AxcUk>$i!vr;prlE|HqxCOswts8xHK33#b&6ZE7gvPK)| zH;(B4DE>YEXC;&jYZGiwG0`t59s=+60a>XH+MYb3=UiL~Dvq9H`8JSiHPI{Bghg^; zI58X;;>P>Y_ciZp4&*4(6bVXtCxZrq2LAqmAcdboz%15urFO|4EMVv{beQ>#9kxyS zO=_oCYa`sAT{k;G1qj@EgGPjsZL`*LxFtK#tiuOkgcsWY zWCpiL0W!41pcEl<8?edX=Ij!YnXo}9A!+-1>npffinvB5bdZe@yA3QZxGB5%aRzmO zx)ia^5em8Q&p}~+@gb~*8Aso-2zx=_L2Ki$;7FfOi|{{5Vf;>)vdHt`1cl+C_u@R~ z1D2KqjV{6YQ=42KQLGtpXp+q7fI=1o~L`j(YZkTfQIt1 z3_r0B9_RmwcjU_y%#_IVFG2Z#WrFf{MhG1=`fYN(?C`A+{LW$f&HX=QdO87(gP54Rp zMwlcF5qbz%LIa_aP)x`pWDpVwF@$hJAi;;=N}!ri%fETIubcfTj|_I&C=Z?rMSoH8 zKHGGQ`U~2W+*UpF3%CH!t39T!{^I-V*i-I2CF9-dXVy?|Gv)vGm5Et#`NF=g?<*Oz zPK5-NZo4(M>9MLVzVe%Z?qQL;kJDPwx@pbdY9a{if3|-+ZR$}zDWHQcdS_Z33=Dm> zB6T0Y`w`IgorZWb2gkHk@lUaNGb-4LfxKE9stPj2^vh9tn$eps*!Jfy+NLOhs{6rF z0-A@vz3xqXeO0IB8s0}tddm-$f9G~@@axk$5m#8Bd(vAzsGOoC`Lz8B!MI1-8bvS1 zZhBrKbJ?@LjMdW?E=nF#yA^v$8&VWz&mtJhq`g6g%rfKkeg{FGGanWQqL|;t^pWyL zw0;))&g%at^c_?9r1z!JYwVgDu)#kTdI&Hb0t0*~i0wH*!dVNe}MF>A}ZUu79=#M8Ph#6IiyvB^8e>>sb$fQdDass&8jf?`NQe&rA zSjh%Pg`#5|o*5X>V1|xe{1oM+DOG5oN}C)tJuZrlQq)}bOqbE*F4TGc3FmGtUzR&Q z?BOmS2pV8;cV7AU-1o!I?y`Y7xkEU&>8l>Ob;CC9Qh`~ygK)QO5R@$%M!1UwX5`Xnmo<9I};dlq*0rIY4 zz9AkU?b@o;EdO|~XLtlR)AQZMX&FE4GIyo2VYbR;)pNHd4C^`p z8e~TQjU~}~&eoj7e%&RJtFmnAG5wlLw~he&@!Q>7>mq;BA_HVL(t`V2B8GsGS1(On zg4UmE&7N5T;K$r-kA4vL+`inC=G-rH*Uk^{c5M3Vs#12c$o%VSm|KZu9QC(?^xiRSd8$A*IkhZ8_SKMMY3|b#jPq;rkUeu{8YB4ny z7dp-Mg#JU+qDEao3u0Joj{A4QpWe_#Sl!DO6Jr1g7)tor`(RP2F0bY0FsO)B0#cCY zqI_L$3kXOG0ne+a_x7S}T}}%~M9u+|tD)C+QK~Mh1w%2?pFhdD)2+zB ziD8*ZJA-0@#rm|csl-CH6)f2lV)?tT^>R3|NXL5Gij;7eu#B2&^NGZAJYR?z)B1Yu zvJTV2LsP9Ak65)Cie%3A&=*BtIrGFNI!%-BQOg5O@pBA44t(brIF23&mjqp|TrSSRjnefT*vGc8;897sZCgqIZ8Iv2lV9+y$hMH&_8>~*Tq)I$iXNTbsCHXtcEnA zQ5>4w5j3ha(5Q?3G>#yLWN?M{G0jx8XeiA!8looWWg0FT9S&x88pfz|q4e}oG+huA ztO52qRtu|!RlzD?<*?FN3G79z2v!iwkLAH~VozaNv5Z(+>|xtp+uydWHn}~cj;94n zqI>!s=|7Io7xW8n@5wGP?`a+#MMemBuKn)(xV#mqaSZE70bAfQMY-b;ecac)^?P*3 zCqkg$NWLRU_~)o_U*y)Uh`naY<)bIUkvtk7^tTow7xO3=kEjL*oRMo-=U&KGc<14{ zqxezG-0mjq;O~ZdeCI-?^O_HZAG0ra!1kkCKQa!nfS@erY>Gy({5XJg#Qd1gDR}R^ z$rG_HMq%k#Pd)BZ+6;6)U{sG5#(WpvWQ^GSfr*05A0yrrOd0(W{hPM0-(K8dHVXbh zN&Vvf`*>fc9puLl<-)*br2iYWuhCx6ffz0O0gx_#$f5hN_Lm(dh_ZsoA<|FsgMFp; zypEfrubn3=Nt0yHeK3}H7$ZPyCy~@czP%6T@D8I0oM1KRFV#MyO+RPvuf5x`hk-Dvj8AG)&#d!W^^P?RbYnYP}-~%R&#z zxTV(X%oMDe>|BCnsNL&nI_1ed#`F$9OV>#2N%fr!@`<2lK}``^+B{)Lstf`; zgkToNP^mO$!hlp9WQGzK1g*v066AymDK7}f5`q~RgXR*O2?}Fy%-O zS=>B6v*Ht^fD#Z)#^@;l#bRKE8?%h)mhFagj~=K|R459REDfekMh!-d?VS1e`7&y9 zC%p!J=Xb;b4~3Hwr}6!M7orPtfX4te40Eg`os+FUR74mJ@Ca*oVb^ilCR=PMo$xWh zHLUK1tvc|`nTMhYodM2aH7{&B0H0`i(4X;5qNRyL$>j+iuNM_JEK0=9mDSRp-!Lk< zZwNk8b*Y~usB`R0C^CYNH|qPTGQr^w3hv@HMMB!yB2>m=P^Pp}(o}3Umsv1KeV&wLTYmnOqDW7>6Mi;Q*5)0gol2BN@-_$d4*@{ggs*^z|NHxnwg5Pa8Ch)zzx9X zvI-4MFRXA*vCI})7b)|4+y8o6HfN`5?H1s)`z%JwpL z1b`3Z$5>Ym0{2y@FCs?N%JSt$6P9s-vnsz|1dXVay^tSaU5*HBQ~CDd-iSikbNNpR zM0ntHm7y1RM}P+)Kg>!z2tcV|UpS6Pmu1TX+6ykgUZwJdY4ZK7OW#L@%TgQGrbdOtQvV={%PAHg6AE^7fA;#% zmTlntXO(|vtX26>D3b#FBz`9OkBBAx`9NCC9OV^D4E(@fAUSec>9Hsz|^w< zqyz>x%vH7Am&PWwMD_c6F)fS7a3Scb@Jk(m3T_l}d?+hqN+8{EYiAcSuJ3o_t_n{F|QMPv3@J7O(hD zddyu3%fcGoa;xUD39D?hnwIp+YB02OtKzcWs|d1sCux^eX=vqE!DSUzam^}Q(jcqY z(9Erz3v?&RmLZZVS$T#x-ClE9guQ9Bw3d{}$}lu=E8#NVD-W_%l;qD!G`#Niipwml z{F)`NBx_cTp_W@A7bseiEp{aKGQ$nk-14~)VP%aL(-Nf2KtmO`7hEQLuY)Y!Nz7#W z7%I3u=eim8`kF$;YGKnTma&8&3wl#eOv08o}p-Q`dB~6kJa-JBsqdibVhoAziSDgLl`8T z)jJ^?lit@yI((h0LU=aq1FRl-scTKPDwlNS|F7LM} z0qh8aCugreIj@sm*pDayJHbH0cU^B_3_a^ND!EwCWsvYf*ANtc()#sFz=|;N*il#V zJa_uzew`ALdJcofXLV1Ur%#XS*C+vSiGldQO%Amo_ zK}LANT0iCga1tVJ!8f5(A4C< zzB{6I{$IuKom8ZQMkaUkIf>Tze<}AFR#+6cpZu=RUi7X1w75@_Lbrf(a&@1zXr=$> za&KdWR|1IS*L~)qZ~Ori=cK|z0gdGRJ`>S0|H*Q%;cIsUWRtV|3`I-*C&ayyu4xMh zC#Ur3i5B~hmwOstJ1+o9e$=NeTI4?_?s@VWoxpJtrcYh8z<;#dV_1HXe32HcP}tC&fC>qR^~m)hGE1$ba-DiZ&fR)%#(CNoiXpB(fg8lvs!WG z(%5fVo_&+ zK4g0zJFiuHPMPB*%Mm>_n%WgHuU0Fh3`t_{p#K*AyDMm3rS_~c`$=XD0A+vcx;L*- zE1-NTiAfy*vxmCw&db&EDW5#abVL^(jqP%rm#*biW=mr1pmT_>?6RDfs6DNG;v^%6 z4i=r)WjueemP?s6i9sFcPvI>r#tcGqe`-0DSxz#{EeKy}K+aUI-rPAKp4#3wcSp7k z`BiH5)DB&EWP8V)lWYz0OXcrRe-}~r+uzOE%f3ZUOZ`s%+l_K=ub#7(twesV{B`s1 zD-@#r^_;ov8zdOdPyKy}(rC}0Gm$Mrf{FVR`3_39J$uejwiF45=wz}sO1M2`4$$+E z&j!_D6Go_k#p@o>OEx)G-b-r!HH79Iagb^k*?_0~a!eJ892XFUh<&o<49 z)q=M2wb>5bUrtnU+OmCl;6p#nP!$ba2pS)n>7&_eHqW7-;)v>jFIzu3x~~HoPu}o# zqEKLR>j%d>>sM@^L5CTMM*-;8c1Ndm8JnEEp$~+)0N>UoNBea!(Lh1dy(8dO>sv?b zbx9kbPG2YF2i$Bebu?cWw*i7PBLNej*80-XWc`v&M&7`OC8q%C)+|TE^$Rvnp#yPC z>H&hS$&PyKfR&ope|1)Yt}6JpdaIYcKG94d>s}29y!hSjA6$b#&p7YP^VGcC=Z0TtTse@bX9FdWkq2nOO2_W zQH@cpKBq9guZ?mTLe2SVz?LB-O zvhA={R@0Y!(wjcoyZY8;1HvEAG?PzsQ>VSNZ%sDfJAD33^!9IRws-J-n+@jptj~Sk zE=@J|HoleFR@e%w&#%4Bn#$}geBWdP93ao`s)pQ!)G~cND^<^EOLwn-@hfc=4e#Ci_h@)&s54%0FPtab6XyZ< zz`4WSac*!o++FxxoGaWFcL#n4=K^=Z-G<-BIm4ZCPH-pOE%+^*Bis?^0C&LI!|ic) za66nW+!kj8x4~J%t#MXxE1V_V5@!Loz?s9%ab|EcoGIKChkzq+CU6rRv@*HLX=YkL zDmlx^qw4B3-)~&Et#g%(b(n8my{%2vR>hs0b+2p}{`Pv`k*@2IN(k<4d)HR^SfO{; z;TuoyzKfg9+%K24+Unnagjzq_&sM&lYu#-pd3sPTO|`Fl;3muir)evrcpUHhMDcX(fANBOVLf4< zSPz&7)*a@Kb%VKK@51h4U16@+JFq)g7nlq7HtaUm8Rm?2f;nMt!ERw4VUAb_m;=@x zW{*8D$r=23uu^;KoFqrEO=P+Hl%f^jvB9Sbo>#&w1tqKJyxDo;y(`uKKJwOiYo*xq z$WW*BGk?WNg-5EDSEkNXuvu9+-h-4TeYI+rbgv8h9G2A{ia${`&7}+ zlr>`!#?&sW3Y;#ET;cOy^q&FY zTH1=~7oAnKVFjxtLxqXpXZi|Sg|a8!v{_LC70SU;lA4U4^8_>*3o*gZpI>Y8^d6ha zf1_gI^SaMC8JKUMpSlA?l#RKRS5~H>>cy<5dIm)$2y-#7j7)>J3sz4r8Kjr!n~QoC z@ERnP=$NbXJY_P7E(zZk{hR)-6g4YKNI#l@f?w)azw^se-wH z#LJNzZY6RPDskx@EdPslWd8bpgIO2N^pq)pNwRX=@;_p{6;cu?@)Z@{6`=n0yn=6v zt%8xqN~-+y6zi`yhTyIYhouD?Y$?$4(m~Bzg_5QQ3eitAKfftPX(|*FEzP9C%&5at z#%{`3UWl@wFQ=KRF|o*a&3p6I^I3TX5ARP?&))ch&D4J4vP3r7(EH-sr{)G_CFT?2 z5}9O!Ul)9zUIl|I#k^f)VB0BqG_>4J&Wk3Q|Ams)b?_b1&sBc*bV&WnR ztR(xQ`1H9O9wo99Jfwsm;QQi%wIo&onP4PE2WjpL<3ad$xp-sz?@Cw@d|wcs;-p_z zJUUKT@efkohvJjx^gN1d#y_uk1zp?c$0s@Ii4`L#FEv;@Su|KQ{_M^#&wrV>IpRCu zJve>P@BC2sVZpl65&OZ(g8+>!n|b~DGe-|PX*FmyHjz|D3N7XEN28;?kaT>u#?VW` zrvUf8dPlqb%lg^EL&<~>0eALl9c_g!GiM(Uq6zH*PJ7jkHu+K?vwi@MQB#2ZUX`P@ zkW_qD=in^@U;ZV3kR)?fH2(Jh4=!KgW2Q5J9)S}7 z9zeVaNyKMr3t_lNo+88rXzvv|nhJ?CXC4n6E};X|!&q?f&B`nX z^dpP#@APxD&Rc>bv_GOMTLh|UN2AJ=nGWduX!W;FC^JUTVHUYIS(F*R_X^LmZ!(1@ z$akHYKk(T#9R5sIk7?GA^Bptl_ck?$~!A5QA@UfW{ua zclXU-d%u3WW@m(W`+lvJ!>8%n9<>(G81(yhw~Ir7kgWDz-52Ou(1*UW|KT3dV;lLu zvfX3+2bli{nEwZu|9=iJPx}w7_z$f3UkX-Kx)eEJvz-xf;kaLEo9ev8y{Dso9CvU| zxc|p->c-Dz^x-ZAezdV1`SHuvR^;(n;g8Lo8$T$Dn8O)NGUDfCWM9sf>%m?RMYbbR z_L#TegY(wUn#D4TxcUJig1fUfQX@fQu?D@HND;woGz<40Z_bQTg)s4*NBHGUrHH?k z=u;X!2#rUP3l5Z(noWrV*5d;%Oe|y}@Ay4(6MC>MPhsnLh&W;pe*figOPIohSx@Zj z65H%arP6mqL5{vbHfaxzCH`ih4A3P)=Evp8DWzZhf5H6|G`I&73AiL@ezE?IL2022 zg-nmjb|-g9d*pDG8aiJH5n0we`GrIx2clHaFN92vUn3`*Ni$?0lmZ%1zav4MD~B{d z26vy(0P}qe@>@t!E7<`hjRsurND$bvA-y46pd`=$t;$iF4OwccoFhYEl@UnkR@4#fyk7&^m=nU{1?EfqaNr)Yt+=Qn(m~yCz|@ z*L6{L3v@%_7cqc+n%b+gD7=+I(SwU(IBOmwdPNr@TaPH(a1jiL>|;nT{o?T^hN2D^ z#z1P~M|*xQY;1;5RN;accG>vUo}Pstn|>5UI23cL<`JT&cwuzYopKe>1ItvAxCX`EXZVYQpETUUFmee=+DJO5SrqtQ%YegZZ)H3|!oY!*W z{aJ;z+z?}GDSpDqtE>?*dvPs2=I(mb$NRRtqxGem z?Rw|n5T!`t+v)pU2As0f0oBA2vb(B0I=31mR7EFKNqyuys^G!NfXjCDpsCnFc2bo^ z=Tw8h>d%S$q<3U{Rq*~~z@fVyP+Y7gTdPW;v#LRUwRi$SdQCQ01 bool: + with tempfile.TemporaryDirectory() as name: + directory = Path(name) + count = 0 + for index, frame, expected in frames(path): + plain = pyzstd.decompress(frame) + if len(plain) != expected: + raise SystemExit( + f"{path} block {index}: pyzstd gave {len(plain)}," + f" the index implies {expected}" + ) + (directory / f"{index:05d}.zst").write_bytes(frame) + (directory / f"{index:05d}.raw").write_bytes(plain) + count += 1 + result = subprocess.run( + ["dart", "run", "tool/verify.dart", str(directory)], + cwd=root, + capture_output=True, + text=True, + ) + line = result.stdout.strip().splitlines()[-1] if result.stdout else "" + print(f"{path.name}: {count} blocks, {line or result.stderr.strip()}") + return result.returncode == 0 + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("paths", nargs="+", type=Path) + parser.add_argument( + "--every", + type=int, + default=1, + help="when a directory is named, take every Nth container", + ) + parser.add_argument("--limit", type=int, default=0) + arguments = parser.parse_args() + + containers: list[Path] = [] + for path in arguments.paths: + if path.is_dir(): + found = sorted(path.glob("*.hgtz")) + containers += found[:: arguments.every] + else: + containers.append(path) + if arguments.limit: + containers = containers[: arguments.limit] + + root = Path(__file__).resolve().parent.parent + failures = [c for c in containers if not check(c, root)] + if failures: + print(f"FAILED on {len(failures)} of {len(containers)} containers") + return 1 + print(f"all {len(containers)} containers match pyzstd byte for byte") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tool/verify.dart b/tool/verify.dart new file mode 100644 index 0000000..5790d42 --- /dev/null +++ b/tool/verify.dart @@ -0,0 +1,52 @@ +import 'dart:io'; +import 'dart:typed_data'; + +import 'package:pure_zstd/pure_zstd.dart'; + +/// Decodes every `*.zst` in a directory and compares it to the `*.raw` +/// beside it. Driven by tool/oracle.py, which writes the pair. +void main(List arguments) { + if (arguments.length != 1) { + stderr.writeln('usage: verify.dart '); + exit(2); + } + final directory = Directory(arguments.single); + final frames = + directory + .listSync() + .whereType() + .where((file) => file.path.endsWith('.zst')) + .toList() + ..sort((a, b) => a.path.compareTo(b.path)); + + final decoder = ZstdDecoder(); + var checked = 0; + var bytes = 0; + for (final frame in frames) { + final expected = File( + frame.path.replaceRange(frame.path.length - 4, null, '.raw'), + ).readAsBytesSync(); + final Uint8List actual; + try { + actual = decoder.decode(frame.readAsBytesSync()); + } on ZstdException catch (error) { + stderr.writeln('${frame.path}: $error'); + exit(1); + } + if (actual.length != expected.length) { + stderr.writeln( + '${frame.path}: ${actual.length} bytes, expected ${expected.length}', + ); + exit(1); + } + for (var i = 0; i < actual.length; i++) { + if (actual[i] != expected[i]) { + stderr.writeln('${frame.path}: byte $i differs'); + exit(1); + } + } + checked++; + bytes += actual.length; + } + stdout.writeln('$checked frames, $bytes bytes, identical'); +} From 0cbe208eb0e0b76402a2388984f9470fbc5ca06c Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 21:53:21 -0600 Subject: [PATCH 02/18] Measure the decoder on the phone, which is the number that decides it 0.414 ms a block in profile AOT on the Pixel, so a 113-block sweep is 47 ms. The phone is about twice as quick at this as the laptop the first figures came from, which read 0.917 ms and 104 ms for the same sixteen blocks. Debug on device is 0.540 ms, so the shipped configuration is the fast one rather than the slow one. The blocks are real, out of N39W105 in srtm1z. The committed .hgtz fixture is no use for timing: its samples are a synthetic pattern that compresses to almost no sequences, and decode cost tracks sequence count, so it reads about six times too fast. It was measured that way once before this was noticed. --- README.md | 25 +++++++++++++++++++ benchmark/same_blocks.dart | 51 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) create mode 100644 benchmark/same_blocks.dart diff --git a/README.md b/README.md index 9e6f2e3..071939c 100644 --- a/README.md +++ b/README.md @@ -99,3 +99,28 @@ Run it yourself: ## Licence Not chosen yet. + +## Measured on the target device, 2026-09-21 + +Sixteen real 256x256 blocks lifted out of `path/to/srtm1z/N39W105.hgtz`, +131,072 bytes of output each, the same blocks in every row so the numbers are +comparable. Median of 960 timed calls after 320 warm-up calls. + +| where | mode | median | MB/s | 113-block sweep | +| --- | --- | --- | --- | --- | +| Pixel 10 Pro XL | **profile, AOT** | **0.414 ms** | 302 | **47 ms** | +| Pixel 10 Pro XL | debug, JIT | 0.540 ms | 231 | 61 ms | +| this laptop | AOT | 0.917 ms | 136 | 104 ms | +| this laptop | JIT | 1.125 ms | 111 | 127 ms | + +The phone is about twice as fast as the laptop at this, which is why the +desktop figures in the commit message read worse than reality. + +Native, for scale: pyzstd on the laptop is 0.175 ms on comparable blocks. That +is a Python C extension rather than Dart FFI, so take it as an indication. No +native-on-phone figure was taken. + +Do not benchmark against `test/fixtures/N51E000.hgtz`. Its samples are a +synthetic pattern that compresses to almost no sequences, and decode cost +tracks sequence count, so it reads about six times too fast. That mistake was +made once already. diff --git a/benchmark/same_blocks.dart b/benchmark/same_blocks.dart new file mode 100644 index 0000000..74bd159 --- /dev/null +++ b/benchmark/same_blocks.dart @@ -0,0 +1,51 @@ +// Times the same extracted blocks the on-device benchmark uses, so the two +// numbers are like for like. +import 'dart:io'; +import 'dart:typed_data'; + +import 'package:pure_zstd/pure_zstd.dart'; + +void main(List args) { + final bytes = File(args.first).readAsBytesSync(); + final data = ByteData.sublistView(bytes); + final count = data.getUint32(0, Endian.big); + final frames = []; + final sizes = []; + var at = 4; + for (var i = 0; i < count; i++) { + final length = data.getUint32(at, Endian.big); + sizes.add(data.getUint32(at + 4, Endian.big)); + at += 8; + frames.add(Uint8List.sublistView(bytes, at, at + length)); + at += length; + } + for (var i = 0; i < frames.length; i++) { + if (zstdDecode(frames[i]).length != sizes[i]) { + throw StateError('block $i decoded to the wrong size'); + } + } + for (var pass = 0; pass < 20; pass++) { + for (final frame in frames) { + zstdDecode(frame); + } + } + final micros = []; + for (var pass = 0; pass < 60; pass++) { + for (final frame in frames) { + final watch = Stopwatch()..start(); + zstdDecode(frame); + watch.stop(); + micros.add(watch.elapsedMicroseconds); + } + } + micros.sort(); + double q(double f) => micros[(micros.length * f).floor()] / 1000; + final median = q(0.5); + stdout.writeln( + 'samples=${micros.length} bytesOut=${sizes.first} ' + 'median=${median.toStringAsFixed(3)}ms p10=${q(0.1).toStringAsFixed(3)} ' + 'p90=${q(0.9).toStringAsFixed(3)} ' + 'MBps=${(sizes.first / 1048576 / (median / 1000)).toStringAsFixed(0)} ' + 'sweep113=${(median * 113).toStringAsFixed(0)}ms', + ); +} From fdab6db63401dcce13746d5d396ff94c71ed660a Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 22:11:05 -0600 Subject: [PATCH 03/18] Time the decoder across compression levels, interleaved Sequential runs of same_blocks.dart disagreed by 70% between the first and third invocation on this box: schedutil plus boost ramp is a larger effect than any difference between levels. Interleaving one block from each file in turn puts that drift on every level equally. --- benchmark/levels.dart | 91 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 benchmark/levels.dart diff --git a/benchmark/levels.dart b/benchmark/levels.dart new file mode 100644 index 0000000..3f08627 --- /dev/null +++ b/benchmark/levels.dart @@ -0,0 +1,91 @@ +// Times several extracted-block files against each other, interleaved, so that +// CPU frequency drift lands on every file equally. +// +// dart compile exe levels.dart -o /tmp/levels && /tmp/levels a.blocks b.blocks +// +// Each file is the format benchmark/same_blocks.dart reads: uint32 count, then +// per block a uint32 frame length, a uint32 decompressed size, and the frame. +// Every file must hold the same number of blocks, of the same output size, so +// the medians are like for like. +import 'dart:io'; +import 'dart:typed_data'; + +import 'package:pure_zstd/pure_zstd.dart'; + +const int warmSeconds = 4; +const int passes = 60; + +List load(String path, List sizes) { + final bytes = File(path).readAsBytesSync(); + final data = ByteData.sublistView(bytes); + final count = data.getUint32(0, Endian.big); + final frames = []; + var at = 4; + for (var i = 0; i < count; i++) { + final length = data.getUint32(at, Endian.big); + final size = data.getUint32(at + 4, Endian.big); + if (sizes.length <= i) { + sizes.add(size); + } else if (sizes[i] != size) { + throw StateError( + '$path block $i is $size bytes out, expected ${sizes[i]}', + ); + } + at += 8; + frames.add(Uint8List.sublistView(bytes, at, at + length)); + at += length; + } + return frames; +} + +void main(List args) { + final sizes = []; + final sets = args.map((path) => load(path, sizes)).toList(); + for (final set in sets) { + if (set.length != sets.first.length) { + throw StateError('the files do not hold the same number of blocks'); + } + for (var i = 0; i < set.length; i++) { + if (zstdDecode(set[i]).length != sizes[i]) { + throw StateError('block $i decoded to the wrong size'); + } + } + } + + // Spin long enough for the governor to raise the clock before anything is + // timed, otherwise the first file measured pays for the ramp. + final warm = Stopwatch()..start(); + while (warm.elapsed.inSeconds < warmSeconds) { + for (final set in sets) { + for (final frame in set) { + zstdDecode(frame); + } + } + } + + final micros = [for (final _ in sets) []]; + for (var pass = 0; pass < passes; pass++) { + for (var b = 0; b < sets.first.length; b++) { + for (var s = 0; s < sets.length; s++) { + final watch = Stopwatch()..start(); + zstdDecode(sets[s][b]); + watch.stop(); + micros[s].add(watch.elapsedMicroseconds); + } + } + } + + for (var s = 0; s < sets.length; s++) { + final xs = micros[s]..sort(); + double q(double f) => xs[(xs.length * f).floor()] / 1000; + final median = q(0.5); + final bytes = sizes.reduce((a, b) => a + b) / sizes.length; + stdout.writeln( + '${args[s].split('/').last.padRight(24)} n=${xs.length} ' + 'median=${median.toStringAsFixed(3)}ms ' + 'p10=${q(0.1).toStringAsFixed(3)} p90=${q(0.9).toStringAsFixed(3)} ' + 'MBps=${(bytes / 1048576 / (median / 1000)).toStringAsFixed(0)} ' + 'sweep113=${(median * 113).toStringAsFixed(0)}ms', + ); + } +} From 019423f08fd727a9ffad598898a489d279725b0d Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:01:56 -0600 Subject: [PATCH 04/18] Record what an audit found, including a wrong claim The table said several frames in one buffer are refused. They are not: the block loop breaks on the last block and never checks the cursor against the input length, so a second frame, a trailing skippable frame and appended junk are all accepted and silently ignored. Four more, each with a reproducer and none fixed yet. The content size sizes an allocation with no ceiling, and one flipped bit in a real tile's descriptor kills the isolate with an error no caller can catch. The interleaved FSE decode writes three symbols in a pass that checks room for two. RLE blocks skip the block size ceiling. And the checksum is never verified, which is the one that makes the others systemic: 34% of 600,000 mutations decoded into wrong bytes in silence. --- README.md | 41 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 071939c..8114a59 100644 --- a/README.md +++ b/README.md @@ -44,13 +44,52 @@ RFC 8878 is the specification. Of it: | Repeated offsets | Yes, including the no-literals shift. | | Content checksum | Parsed and **skipped, never verified**. | | Dictionaries | Refused. | -| Skippable frames, several frames in one buffer | Refused. Reading one frame is the whole API. | +| Skippable frames, several frames in one buffer | Only a *leading* skippable frame is refused, and only because the magic check fires. See **Known defects**. | | Streaming, or decoding part of a frame | Not offered. | It targets native Dart and not the web: the literals header reads a five-byte field with a 32-bit shift, and the bit reader carries 56 bits in an `int`. Both are exact on the VM and on AOT, and neither is on JavaScript. +## Known defects + +Found by two audits on 2026-09-21, confirmed with reproducers, and **not yet +fixed**. Nothing should depend on this package until they are. + +1. **`Frame_Content_Size` sizes the output allocation with no ceiling.** A + 16-byte frame can ask for 4 EiB. One bit flipped in the frame descriptor of a + real tile gives `RangeError` or `OutOfMemoryError`, and neither is catchable + as `ZstdException`, so the isolate dies. This is what the skipped + `Window_Descriptor` was for: RFC 8878 §3.1.1.1.2 allows a decoder to refuse a + frame asking for more memory than it will give. Require the declared size to + be under a stated ceiling, and have the `.hgtz` reader pass the exact + `height * width * 2` it expects. +2. **`fseDecodeInterleaved` writes three symbols in a pass that checks room for + two.** `fse.dart:210` guards on `at + 2 > outLimit` and the exhausted branch + writes a third, so a crafted Huffman weight stream indexes `_weights[256]`. + The reproducer is 53 bytes, and random mutation of a real tile found it + independently. The guard wants `at + 3`. +3. **Several frames in one buffer decode to the first and return silently.** The + block loop breaks on the last block and nothing compares the cursor with the + input length, so a second frame, a trailing skippable frame, and four junk + bytes appended are all accepted and ignored. Require the cursor to have + reached the end. +4. **RLE blocks skip the block-size ceiling.** `type != 1` exempts them, so 97 + bytes of input returns 44 MB of output that a caller cannot tell from a good + tile. +5. **The content checksum is parsed and never verified.** Over 600,000 + mutations of real block data, 34% decoded into wrong bytes with no error + raised. A frame may also claim a checksum and supply none, because the four + bytes are stepped over rather than read. `hgtz.py` writes no checksum either, + so a corrupt tile on a phone becomes wrong elevation rather than an error. + This is the one that makes the rest systemic: it is the only mechanism that + would catch a wrong decode from anything else. + +Over-permissive against libzstd, with no wrong output, so lower priority: a +compressed block over `blockSizeMax` is accepted, the Huffman weight table is +read with `maxSymbol` 255 where libzstd uses 12, and `Regenerated_Size` is +capped at a flat 128 KiB rather than `min(Window_Size, 128 KiB)`. + ## Correctness The reference is pyzstd, which wraps libzstd. `tool/oracle.py` extracts every From 489c898b14d1bc3b3864f19548f0f1054b15ce37 Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:09:25 -0600 Subject: [PATCH 05/18] Guard the FSE pass for the three symbols it writes Both exhausted branches write a third symbol in the same pass that the guard admitted for two. The Huffman weight decoder calls this with a 256-byte buffer and a limit of 256, so `at` reached 256 exactly, which is one past the end, and a crafted weight stream turned that into an IndexError rather than a refusal. A 53-byte input reproduces it, and random mutation of a real tile found it independently. --- lib/src/fse.dart | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/src/fse.dart b/lib/src/fse.dart index 6c2a067..4f25923 100644 --- a/lib/src/fse.dart +++ b/lib/src/fse.dart @@ -207,7 +207,11 @@ int fseDecodeInterleaved( var at = outStart; while (true) { - if (at + 2 > outLimit) { + // Three, not two: both exhausted branches below write a third symbol in + // the same pass. Guarding for two let `at` reach outLimit exactly, which + // is one past the end of the caller's buffer, and a crafted weight + // stream turned that into an IndexError rather than a refusal. + if (at + 3 > outLimit) { throw const ZstdException('FSE stream longer than its output allows'); } out[at++] = symbol[state1]; From 09b06ca9a0067761078a95c657a3f8e71f84eefa Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:09:28 -0600 Subject: [PATCH 06/18] Refuse a frame this decoder cannot account for Three ways a header was taken on trust. Frame_Content_Size is up to a 64-bit number straight from the input and it sized the output buffer unchecked, so a sixteen-byte frame could ask for four exbibytes. It is reachable by accident: one bit flipped in a real tile's frame descriptor moves the content-size flag from 2 to 3, eight bytes of compressed data are read as the size, and the eighth lands on the sign bit. That is a RangeError or an OutOfMemoryError, both Errors rather than Exceptions, so a caller writing `on ZstdException` catches neither and the isolate dies. There is a ceiling now, and a caller that passes expectedSize gets an exact check instead, which is what the .hgtz reader wants since a block is height * width * 2. Reading one frame is the whole API, and nothing compared the cursor with the input length, so a second frame, a trailing skippable frame and four bytes of junk were accepted and silently ignored. Two frames concatenated returned the first one's bytes and raised nothing. The checksum's four bytes were stepped over rather than read, so a frame could claim one and supply none. And an RLE block was exempt from the block size ceiling, which let 97 bytes return 44 MB that a caller cannot tell from a good tile. --- lib/src/decoder.dart | 59 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 56 insertions(+), 3 deletions(-) diff --git a/lib/src/decoder.dart b/lib/src/decoder.dart index b38663a..7aa109c 100644 --- a/lib/src/decoder.dart +++ b/lib/src/decoder.dart @@ -12,8 +12,35 @@ import 'tables.dart'; /// One instance is not safe to share between isolates, and a frame must be /// decoded to completion before the next one starts. class ZstdDecoder { + ZstdDecoder({this.maxOutputSize = defaultMaxOutputSize}); + static const int magic = 0xFD2FB528; + /// The largest output this decoder will allocate for one frame. + /// + /// `Frame_Content_Size` is up to a 64-bit number taken straight from the + /// input and it used to size the output buffer unchecked, so a sixteen-byte + /// frame could ask for four exbibytes. It is reachable by accident as well + /// as by design: one bit flipped in a real tile's frame descriptor moves + /// the content-size flag from 2 to 3, eight bytes of compressed data are + /// read as the size, and the eighth lands on the sign bit. That is a + /// `RangeError` for a negative length, or an `OutOfMemoryError` for a large + /// positive one, and both are `Error` rather than `Exception`, so a caller + /// writing `on ZstdException` does not catch either and the isolate dies. + /// + /// RFC 8878 section 3.1.1.1.2 lets a decoder refuse a frame that asks for + /// more memory than it is willing to give, which is the job of the window + /// descriptor this package skips. Allocating the output whole is why it can + /// skip it, and this is the bound that replaces it. + /// + /// A caller that knows the exact size should pass [decode]'s `expectedSize` + /// instead and get an exact check rather than a ceiling. The `.hgtz` reader + /// knows it: a block is `height * width * 2`. + static const int defaultMaxOutputSize = 64 * 1024 * 1024; + + /// The ceiling this instance enforces. See [defaultMaxOutputSize]. + final int maxOutputSize; + /// The largest block a frame may contain, and so the most literals one /// block can hold. static const int maxBlockSize = 128 * 1024; @@ -98,6 +125,17 @@ class ZstdDecoder { 'frame declares no content size and none was given', ); } + // Before the allocation, and before any of it is trusted. + if (size < 0 || size > maxOutputSize) { + throw ZstdException('frame declares $size bytes, over the maximum'); + } + // When the caller knows the size, the frame agreeing with it is the + // check. Left to the end, a frame declaring less decoded its own + // content happily and a frame declaring more allocated whatever it + // asked for first. + if (contentSize != null && expectedSize != null && size != expectedSize) { + throw ZstdException('frame declares $size bytes, not $expectedSize'); + } final out = Uint8List(size); _huffman.log = 0; @@ -118,7 +156,10 @@ class ZstdDecoder { final last = header & 1; final type = (header >> 1) & 3; final blockSize = header >> 3; - if (blockSize > maxBlockSize && type != 1) { + // Every type, RLE included. Its Block_Size is its regenerated size and + // the format bounds it like any other, and exempting it let a 97-byte + // input return 44 MB that a caller cannot tell from a good tile. + if (blockSize > maxBlockSize) { throw ZstdException('block of $blockSize bytes is over the maximum'); } @@ -153,13 +194,25 @@ class ZstdDecoder { } if (checksum == 1) { - // Parsed so the frame is consumed whole, but not verified. Every block - // in a .hgtz is its own frame and pyzstd writes none. + // Not verified yet. Required to be present, though: stepping the + // cursor over four bytes that were never there let a frame claim a + // checksum and supply none. + if (at + 4 > src.length) { + throw const ZstdException('truncated content checksum'); + } at += 4; } if (written != out.length) { throw ZstdException('frame decoded to $written bytes, not ${out.length}'); } + // Reading one frame is the whole API, so anything after it is input this + // decoder does not understand. Nothing used to compare the cursor with + // the length, so a second frame, a trailing skippable frame, and four + // bytes of junk were all accepted and silently ignored: two frames + // concatenated returned the first one's bytes and raised nothing. + if (at != src.length) { + throw ZstdException('${src.length - at} bytes after the frame'); + } return out; } From cc5d8b8914554083a299ef1890921026e0e6a616 Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:09:56 -0600 Subject: [PATCH 07/18] Mark four of the five defects closed Only the unverified checksum is left, and it is narrower here than it looks: the downloader verifies a SHA-256 over the whole container before it renames the .part, so a tile damaged in transit is already caught. What remains is corruption at rest on the phone. --- README.md | 59 ++++++++++++++++++++++++++++++++----------------------- 1 file changed, 34 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 8114a59..212acda 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ RFC 8878 is the specification. Of it: | Repeated offsets | Yes, including the no-literals shift. | | Content checksum | Parsed and **skipped, never verified**. | | Dictionaries | Refused. | -| Skippable frames, several frames in one buffer | Only a *leading* skippable frame is refused, and only because the magic check fires. See **Known defects**. | +| Skippable frames, several frames in one buffer | Refused. Reading one frame is the whole API, and input left over after it is an error. | | Streaming, or decoding part of a frame | Not offered. | It targets native Dart and not the web: the literals header reads a five-byte @@ -53,37 +53,46 @@ Both are exact on the VM and on AOT, and neither is on JavaScript. ## Known defects -Found by two audits on 2026-09-21, confirmed with reproducers, and **not yet -fixed**. Nothing should depend on this package until they are. +Found by two audits on 2026-09-21, each with a reproducer. Four are fixed; the +fifth is the one that matters most and is still open. -1. **`Frame_Content_Size` sizes the output allocation with no ceiling.** A +### Open + +**The content checksum is parsed and never verified.** Over 600,000 mutations of +real block data, 34% decoded into wrong bytes with no error raised. This is the +only mechanism that would catch a wrong decode from anything else, which is what +makes it the one worth closing. Two halves: verify the XXH64 when the flag is +set, and have `hgtz.py` start writing one, which costs four bytes a +block and a re-transcode of the archive. + +Narrower than it first looks for this project. the downloader verifies a +SHA-256 over the whole container before it renames the `.part`, so a tile damaged +on the way to the phone is already caught. What is left is corruption at rest +after that check. + +### Fixed 2026-09-21 + +1. **`Frame_Content_Size` sized the output allocation with no ceiling.** A 16-byte frame can ask for 4 EiB. One bit flipped in the frame descriptor of a real tile gives `RangeError` or `OutOfMemoryError`, and neither is catchable as `ZstdException`, so the isolate dies. This is what the skipped `Window_Descriptor` was for: RFC 8878 §3.1.1.1.2 allows a decoder to refuse a frame asking for more memory than it will give. Require the declared size to be under a stated ceiling, and have the `.hgtz` reader pass the exact - `height * width * 2` it expects. -2. **`fseDecodeInterleaved` writes three symbols in a pass that checks room for - two.** `fse.dart:210` guards on `at + 2 > outLimit` and the exhausted branch - writes a third, so a crafted Huffman weight stream indexes `_weights[256]`. - The reproducer is 53 bytes, and random mutation of a real tile found it - independently. The guard wants `at + 3`. -3. **Several frames in one buffer decode to the first and return silently.** The - block loop breaks on the last block and nothing compares the cursor with the - input length, so a second frame, a trailing skippable frame, and four junk - bytes appended are all accepted and ignored. Require the cursor to have - reached the end. -4. **RLE blocks skip the block-size ceiling.** `type != 1` exempts them, so 97 - bytes of input returns 44 MB of output that a caller cannot tell from a good - tile. -5. **The content checksum is parsed and never verified.** Over 600,000 - mutations of real block data, 34% decoded into wrong bytes with no error - raised. A frame may also claim a checksum and supply none, because the four - bytes are stepped over rather than read. `hgtz.py` writes no checksum either, - so a corrupt tile on a phone becomes wrong elevation rather than an error. - This is the one that makes the rest systemic: it is the only mechanism that - would catch a wrong decode from anything else. + `height * width * 2` it expects. `ZstdDecoder` now takes a `maxOutputSize`, + 64 MiB by default, and a caller passing `expectedSize` gets an exact check + against the frame's own figure instead. +2. **`fseDecodeInterleaved` wrote three symbols in a pass that checked room for + two**, so a crafted Huffman weight stream indexed `_weights[256]`. The + reproducer is 53 bytes, and random mutation of a real tile found it + independently. +3. **Several frames in one buffer decoded to the first and returned silently.** + Nothing compared the cursor with the input length, so a second frame, a + trailing skippable frame and four bytes of junk were all accepted and + ignored. A frame claiming a checksum could also supply none, because the four + bytes were stepped over rather than read. +4. **RLE blocks skipped the block-size ceiling**, so 97 bytes of input returned + 44 MB of output that a caller cannot tell from a good tile. Over-permissive against libzstd, with no wrong output, so lower priority: a compressed block over `blockSizeMax` is accepted, the Huffman weight table is From 6b48be1bfd91f5477a0f16e02e39c9c8c8f77ce5 Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:10:44 -0600 Subject: [PATCH 08/18] Add XXH64 so the content checksum can be verified The decoder parses Content_Checksum and steps over it without reading it. An audit mutated real block data 600,000 times behind a valid frame header and 34 percent decoded into wrong bytes with no error raised. A frame can also set the checksum flag and supply no bytes at all. This is the only mechanism that would catch a wrong decode from anything else, and the package decodes elevation tiles on a phone. RFC 8878 section 3.1.1 stores the low 32 bits of the XXH64 of the whole content, seed zero, little-endian. Xxh64.digest32 is that value. The incremental form is for the decoder, which writes the output block by block and should not walk it a second time. Every operation relies on int being a wrapping 64-bit two's complement integer, which holds on the VM and on AOT and not under dart2js. The package already targets native only. Verified against the xxhash Python package, version 3.0.0, over both byte patterns at every length class around the 32, 8, 4 and 1 byte boundaries, over 400 random inputs one shot and in random chunkings, and end to end against the trailing four bytes of a real zstd -19 --check frame. --- lib/src/xxhash.dart | 169 +++++++++++++++++++++++++++++ test/xxhash_test.dart | 241 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 410 insertions(+) create mode 100644 lib/src/xxhash.dart create mode 100644 test/xxhash_test.dart diff --git a/lib/src/xxhash.dart b/lib/src/xxhash.dart new file mode 100644 index 0000000..93de96a --- /dev/null +++ b/lib/src/xxhash.dart @@ -0,0 +1,169 @@ +/// XXH64, the hash zstd uses for `Content_Checksum`. +/// +/// RFC 8878 section 3.1.1 stores the low 32 bits of the XXH64 of the whole +/// decompressed frame content, seeded with zero, little-endian, after the last +/// block. [Xxh64.digest32] is that value. +/// +/// The algorithm is defined on unsigned 64-bit words that wrap. Dart's `int` +/// on the VM and on AOT is a 64-bit two's complement integer whose `+`, `-`, +/// `*` and `<<` wrap on overflow, and `>>>` shifts the raw bits in, so every +/// operation here is exact with no masking. None of that holds under dart2js, +/// where `int` is a double: the multiplies, the rotates and the wrapping adds +/// would all be wrong. Like the rest of this package, it targets native only. +library; + +import 'dart:typed_data'; + +// The five primes, written as they appear in the reference implementation. +// Dart reads a hex literal above 2^63 as its two's complement, so _prime1, +// _prime2 and _prime4 are negative here. The bits are what matter. +const int _prime1 = 0x9E3779B185EBCA87; +const int _prime2 = 0xC2B2AE3D27D4EB4F; +const int _prime3 = 0x165667B19E3779F9; +const int _prime4 = 0x85EBCA77C2B2AE63; +const int _prime5 = 0x27D4EB2F165667C5; + +/// Hashes [data] in one pass. Equivalent to `(Xxh64()..update(data)).digest`. +int xxh64(Uint8List data, {int seed = 0}) => + (Xxh64(seed: seed)..update(data)).digest; + +/// XXH64 over as many chunks as the caller has. +/// +/// A multi-block frame can feed each block to [update] as it is written, +/// rather than hashing the output again once it is whole. Reading [digest] +/// does not end the hash; more may be fed in afterwards. +class Xxh64 { + Xxh64({int seed = 0}) : _seed = seed { + reset(); + } + + final int _seed; + + /// Whole 32-byte stripes are consumed straight from the caller's bytes. + /// This holds only the remainder between calls. + final Uint8List _tail = Uint8List(32); + late final ByteData _tailView = ByteData.sublistView(_tail); + + int _tailLength = 0; + int _length = 0; + + int _v1 = 0; + int _v2 = 0; + int _v3 = 0; + int _v4 = 0; + + /// Discards everything fed in so far, back to the seed given to the + /// constructor. + void reset() { + _v1 = _seed + _prime1 + _prime2; + _v2 = _seed + _prime2; + _v3 = _seed; + _v4 = _seed - _prime1; + _tailLength = 0; + _length = 0; + } + + /// Feeds `data[start..end)`, defaulting to all of [data]. + /// + /// The range form is so a decoder can hand over the part of an output buffer + /// it just filled without taking a view of it. + void update(Uint8List data, [int start = 0, int? end]) { + final stop = RangeError.checkValidRange(start, end, data.length); + var at = start; + _length += stop - at; + + if (_tailLength > 0) { + final take = stop - at < 32 - _tailLength ? stop - at : 32 - _tailLength; + _tail.setRange(_tailLength, _tailLength + take, data, at); + _tailLength += take; + at += take; + if (_tailLength < 32) { + return; + } + _stripe(_tailView, 0); + _tailLength = 0; + } + + final view = ByteData.sublistView(data); + while (at + 32 <= stop) { + _stripe(view, at); + at += 32; + } + + if (at < stop) { + _tail.setRange(0, stop - at, data, at); + _tailLength = stop - at; + } + } + + /// The 64-bit hash of everything fed in so far. + int get digest { + // Under 32 bytes the accumulators were never touched, so the hash starts + // from the seed instead of from their merge. _v3 still holds the seed. + var h = _length >= 32 + ? _merge( + _merge( + _merge( + _merge( + _rotl(_v1, 1) + + _rotl(_v2, 7) + + _rotl(_v3, 12) + + _rotl(_v4, 18), + _v1, + ), + _v2, + ), + _v3, + ), + _v4, + ) + : _v3 + _prime5; + h += _length; + + // The remainder is consumed eight bytes at a time, then four, then one. + var at = 0; + while (at + 8 <= _tailLength) { + h ^= _round(0, _tailView.getInt64(at, Endian.little)); + h = _rotl(h, 27) * _prime1 + _prime4; + at += 8; + } + if (at + 4 <= _tailLength) { + h ^= _tailView.getUint32(at, Endian.little) * _prime1; + h = _rotl(h, 23) * _prime2 + _prime3; + at += 4; + } + while (at < _tailLength) { + h ^= _tail[at] * _prime5; + h = _rotl(h, 11) * _prime1; + at++; + } + + h ^= h >>> 33; + h *= _prime2; + h ^= h >>> 29; + h *= _prime3; + h ^= h >>> 32; + return h; + } + + /// The low 32 bits of [digest]: the value zstd stores as + /// `Content_Checksum`. + int get digest32 => digest & 0xFFFFFFFF; + + void _stripe(ByteData view, int at) { + _v1 = _round(_v1, view.getInt64(at, Endian.little)); + _v2 = _round(_v2, view.getInt64(at + 8, Endian.little)); + _v3 = _round(_v3, view.getInt64(at + 16, Endian.little)); + _v4 = _round(_v4, view.getInt64(at + 24, Endian.little)); + } + + static int _round(int accumulator, int input) { + final v = accumulator + input * _prime2; + return _rotl(v, 31) * _prime1; + } + + static int _merge(int h, int accumulator) => + (h ^ _round(0, accumulator)) * _prime1 + _prime4; + + static int _rotl(int v, int bits) => (v << bits) | (v >>> (64 - bits)); +} diff --git a/test/xxhash_test.dart b/test/xxhash_test.dart new file mode 100644 index 0000000..0cda652 --- /dev/null +++ b/test/xxhash_test.dart @@ -0,0 +1,241 @@ +import 'dart:typed_data'; + +import 'package:pure_zstd/src/xxhash.dart'; +import 'package:test/test.dart'; + +// Every digest below came from the `xxhash` Python package, version 3.0.0, +// which wraps the reference C implementation. The two byte generators are +// written identically on both sides, so the inputs are reproducible here +// without carrying the bytes themselves. +// +// The lengths are chosen around the boundaries of the tail, which is +// consumed in 32-byte stripes, then 8, then 4, then 1. Each of those is a +// separate branch and each gets a length either side of it. + +/// A repeating pattern. 251 is prime, so it does not line up with the stripe. +Uint8List _pattern(int length) => + Uint8List.fromList(List.generate(length, (i) => i % 251)); + +/// Bytes with no structure, from a deterministic generator rather than +/// `Random`, so the expected digests stay valid. +Uint8List _pseudoRandom(int length) { + final out = Uint8List(length); + var x = 1; + for (var i = 0; i < length; i++) { + x = (x * 1103515245 + 12345) & 0x7FFFFFFF; + out[i] = (x >> 16) & 0xFF; + } + return out; +} + +const List<(int, int)> _patternDigests = [ + (0, 0xEF46DB3751D8E999), + (1, 0xE934A84ADB052768), + (2, 0x4865A00BA85B7CD6), + (3, 0xE5C7BB4533BC65DD), + (4, 0xFFCED8604453CC1E), + (5, 0xDD0274386E26030C), + (7, 0x14CC643F630C72D2), + (8, 0x884A173614B81B8D), + (9, 0x67D85784A7C78C5B), + (15, 0xA948F5F0F6ABAC2D), + (16, 0x44B6EF2FB84169F7), + (17, 0x5603E60C527599B6), + (31, 0xC346D2B59B4D8EE1), + (32, 0xCBF59C5116FF32B4), + (33, 0x0C535D1ACAFB8EAD), + (63, 0xE26AA9E2A95F8E4F), + (64, 0xF7C67301DB6713F0), + (65, 0xC31EB63B2AE4465B), + (127, 0x464D085810CE0199), + (128, 0x7A7FE14647B9AB92), + (129, 0x0BA25DFD6E891FCF), + (255, 0x566D96B832B967C1), + (256, 0xF33944343EE85824), + (1000, 0xF306F04AA88B54D3), + (4096, 0x122A8C8D994AD3EC), + (5000, 0xA6833D648FD6A332), +]; + +const List<(int, int)> _pseudoRandomDigests = [ + (0, 0xEF46DB3751D8E999), + (1, 0xD2DA77930F69647C), + (2, 0x9F33D4BD50F99C34), + (3, 0xF7C331CF2042B940), + (4, 0x1B042F821FC4A793), + (5, 0x6E1F98107D9DB571), + (7, 0x41D39023940006C8), + (8, 0xEF7823FCE4AD9AFB), + (9, 0x81CF88E9D8340150), + (15, 0x6F0095B1C00E82FB), + (16, 0x29965039DF6047BB), + (17, 0x5236FCB96D8FB228), + (31, 0x68C3DBB752F3FA97), + (32, 0xAD7DC5A569BB047C), + (33, 0x475B642AC692E380), + (63, 0xC18C4743DE1B99A8), + (64, 0x06872C5D6370DE25), + (65, 0x8B5F4E34A32C5A09), + (127, 0x77DC651915D836B5), + (128, 0x2E9C4CB0B29BC8EC), + (129, 0xB5E3A45BFF735CFA), + (255, 0x9AD1F660B88ABC38), + (256, 0x9AE30993C3F4B05A), + (1000, 0x7AD5B044FB6FA81D), + (4096, 0x4AFD74BF8BD52DF5), + (5000, 0xCAEF4049E5C55187), +]; + +/// `Mt Herman 2764 m\n` 241 times, through `zstd -19 --check`. The frame is +/// embedded whole so the trailing `Content_Checksum` is read from a real +/// frame rather than asserted from a constant. +const String _checkedFrame = + '28b52ffd64010fcd0000884d74204865726d616e2032373634206d0a01' + '00db4ffe5c02fe13e790'; + +Uint8List _hex(String text) { + final out = Uint8List(text.length ~/ 2); + for (var i = 0; i < out.length; i++) { + out[i] = int.parse(text.substring(i * 2, i * 2 + 2), radix: 16); + } + return out; +} + +void main() { + group('xxh64 in one shot', () { + for (final (length, expected) in _patternDigests) { + test('matches the reference over $length repeating bytes', () { + expect(xxh64(_pattern(length)), expected); + }); + } + + for (final (length, expected) in _pseudoRandomDigests) { + test('matches the reference over $length unstructured bytes', () { + expect(xxh64(_pseudoRandom(length)), expected); + }); + } + + test('honours a seed, including one that is negative as a Dart int', () { + expect(xxh64(_pattern(0), seed: 1), 0xD5AFBA1336A3BE4B); + expect(xxh64(_pattern(5), seed: 1), 0x27CE45508B78A340); + expect(xxh64(_pattern(32), seed: 1), 0xD74E6766CE9DBA94); + expect(xxh64(_pattern(200), seed: 1), 0x20BD094800CB1DFA); + + const seed = 0x9E3779B185EBCA87; + expect(seed.isNegative, isTrue); + expect(xxh64(_pattern(0), seed: seed), 0x6EC6D05F61C7E7A7); + expect(xxh64(_pattern(5), seed: seed), 0xF60DC54C180A9098); + expect(xxh64(_pattern(32), seed: seed), 0xBFB3E4EF6096C49C); + expect(xxh64(_pattern(200), seed: seed), 0x6EA98426E0B64F4A); + }); + }); + + group('the incremental form', () { + // Lengths either side of every boundary, plus enough stripes that a + // chunking can straddle several. + const lengths = [ + 0, + 1, + 3, + 4, + 7, + 8, + 15, + 16, + 31, + 32, + 33, + 63, + 64, + 65, + 127, + 128, + 129, + 1000, + 4096, + 5001, + ]; + + for (final chunk in [1, 3, 7, 8, 17, 31, 32, 33, 64, 250]) { + test('in $chunk-byte chunks equals the one shot', () { + for (final length in lengths) { + final data = _pseudoRandom(length); + final hash = Xxh64(); + for (var at = 0; at < length; at += chunk) { + final end = at + chunk < length ? at + chunk : length; + hash.update(data, at, end); + } + expect(hash.digest, xxh64(data), reason: '$length bytes'); + } + }); + } + + test('ignores empty chunks, leading, trailing and interleaved', () { + final data = _pattern(300); + final empty = Uint8List(0); + final hash = Xxh64() + ..update(empty) + ..update(data, 0, 1) + ..update(empty) + ..update(data, 1, 40) + ..update(data, 40, 40) + ..update(data, 40, 300) + ..update(empty); + expect(hash.digest, xxh64(data)); + }); + + test('accepts a whole list with no range', () { + final data = _pattern(100); + final hash = Xxh64() + ..update(Uint8List.sublistView(data, 0, 37)) + ..update(Uint8List.sublistView(data, 37)); + expect(hash.digest, xxh64(data)); + }); + + test('reads the same digest twice and keeps hashing afterwards', () { + final data = _pattern(500); + final hash = Xxh64()..update(data, 0, 200); + expect(hash.digest, xxh64(_pattern(200))); + expect(hash.digest, xxh64(_pattern(200))); + hash.update(data, 200, 500); + expect(hash.digest, xxh64(data)); + }); + + test('reset returns it to the state of a fresh instance', () { + final hash = Xxh64()..update(_pattern(999)); + hash.reset(); + expect(hash.digest, xxh64(_pattern(0))); + hash.update(_pattern(65)); + expect(hash.digest, xxh64(_pattern(65))); + }); + + test('rejects a range outside the list', () { + expect(() => Xxh64().update(_pattern(10), 0, 11), throwsRangeError); + expect(() => Xxh64().update(_pattern(10), -1), throwsRangeError); + expect(() => Xxh64().update(_pattern(10), 6, 5), throwsRangeError); + }); + }); + + group('a real zstd frame', () { + final frame = _hex(_checkedFrame); + final content = Uint8List.fromList( + List.filled( + 241, + 'Mt Herman 2764 m\n'.codeUnits, + ).expand((e) => e).toList(), + ); + + test('declares a content checksum', () { + expect((frame[4] >> 2) & 1, 1); + }); + + test('ends with the low 32 bits of the content digest', () { + final stored = ByteData.sublistView( + frame, + frame.length - 4, + ).getUint32(0, Endian.little); + expect((Xxh64()..update(content)).digest32, stored); + expect(xxh64(content) & 0xFFFFFFFF, stored); + }); + }); +} From bd4ccf19b5608eb67ccbd6079174f031cc860255 Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:12:23 -0600 Subject: [PATCH 09/18] Verify the frame checksum instead of stepping over it Skipping it left the decoder with no integrity check at all. Over 600,000 mutations of real block data with the frame header held valid, a third decoded into wrong bytes and raised nothing, because a damaged match or literal length usually still produces plausible output of the right length. Every other guard in here catches a frame that cannot be parsed; this is the only one that catches a frame that parses and is wrong. Hashed only when the frame carries a checksum. The .hgtz reader would pay for it on every block and has no use for it: pyzstd writes none, and the container's own SHA-256 is verified before the download is kept. --- lib/src/decoder.dart | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/lib/src/decoder.dart b/lib/src/decoder.dart index 7aa109c..15b61bf 100644 --- a/lib/src/decoder.dart +++ b/lib/src/decoder.dart @@ -5,6 +5,7 @@ import 'exception.dart'; import 'fse.dart'; import 'huffman.dart'; import 'tables.dart'; +import 'xxhash.dart'; /// Decodes zstd frames, holding its entropy tables and scratch between calls /// so that decoding a second frame allocates nothing but the output. @@ -45,6 +46,7 @@ class ZstdDecoder { /// block can hold. static const int maxBlockSize = 128 * 1024; + final Xxh64 _hash = Xxh64(); final ReverseBitReader _bits = ReverseBitReader(); final HuffmanTable _huffman = HuffmanTable(); final FseTable _literalLengths = FseTable(literalLengthMaxLog); @@ -146,8 +148,14 @@ class ZstdDecoder { _repeat2 = 4; _repeat3 = 8; + // Only when the frame carries one. Hashing every frame would cost the + // .hgtz reader throughput it has no use for, since pyzstd writes no + // checksum and the container's own SHA-256 is verified on download. + final hash = checksum == 1 ? (_hash..reset()) : null; + var written = 0; while (true) { + final blockStart = written; if (at + 3 > src.length) { throw const ZstdException('truncated block header'); } @@ -188,18 +196,30 @@ class ZstdDecoder { throw const ZstdException('reserved block type'); } + hash?.update(out, blockStart, written); + if (last == 1) { break; } } if (checksum == 1) { - // Not verified yet. Required to be present, though: stepping the - // cursor over four bytes that were never there let a frame claim a - // checksum and supply none. if (at + 4 > src.length) { throw const ZstdException('truncated content checksum'); } + // The low 32 bits of XXH64 over the whole output, little-endian. + // Skipping it left the decoder with no integrity check at all: over + // 600,000 mutations of real block data, a third decoded into wrong + // bytes and raised nothing, because a damaged match or literal length + // usually still produces *some* plausible output of the right size. + final stored = + src[at] | + (src[at + 1] << 8) | + (src[at + 2] << 16) | + (src[at + 3] << 24); + if (stored != hash!.digest32) { + throw const ZstdException('the frame checksum does not match'); + } at += 4; } if (written != out.length) { From 724e9b0433c6787c84ebe54558d2d053b83cfb71 Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:12:46 -0600 Subject: [PATCH 10/18] Close the last of the five defects --- README.md | 34 +++++++++++++++------------------- 1 file changed, 15 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 212acda..a4c8a4d 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ RFC 8878 is the specification. Of it: | Huffman weights | Both the direct nibble form and the FSE-coded form. | | Sequence tables | Predefined, RLE, FSE-compressed and repeat, for all three of literal length, match length and offset. | | Repeated offsets | Yes, including the no-literals shift. | -| Content checksum | Parsed and **skipped, never verified**. | +| Content checksum | Verified. XXH64 over the output, compared with the stored low 32 bits. Hashed only when the frame carries one. | | Dictionaries | Refused. | | Skippable frames, several frames in one buffer | Refused. Reading one frame is the whole API, and input left over after it is an error. | | Streaming, or decoding part of a frame | Not offered. | @@ -51,26 +51,16 @@ It targets native Dart and not the web: the literals header reads a five-byte field with a 32-bit shift, and the bit reader carries 56 bits in an `int`. Both are exact on the VM and on AOT, and neither is on JavaScript. -## Known defects +## Defects found and closed, 2026-09-21 -Found by two audits on 2026-09-21, each with a reproducer. Four are fixed; the -fifth is the one that matters most and is still open. +Two audits, each finding with a reproducer. All five are fixed. -### Open - -**The content checksum is parsed and never verified.** Over 600,000 mutations of -real block data, 34% decoded into wrong bytes with no error raised. This is the -only mechanism that would catch a wrong decode from anything else, which is what -makes it the one worth closing. Two halves: verify the XXH64 when the flag is -set, and have `hgtz.py` start writing one, which costs four bytes a -block and a re-transcode of the archive. - -Narrower than it first looks for this project. the downloader verifies a -SHA-256 over the whole container before it renames the `.part`, so a tile damaged -on the way to the phone is already caught. What is left is corruption at rest -after that check. - -### Fixed 2026-09-21 +One decision is still open, and it is not a defect in this package: +`hgtz.py` writes no checksum, so nothing in a `.hgtz` is covered by +the verification below. Making it write one costs four bytes a block and a +re-transcode of every tile. It is narrower than it sounds, because +the downloader verifies a SHA-256 over the whole container before it keeps the +download, so only corruption at rest on the phone is uncovered. 1. **`Frame_Content_Size` sized the output allocation with no ceiling.** A 16-byte frame can ask for 4 EiB. One bit flipped in the frame descriptor of a @@ -93,6 +83,12 @@ after that check. bytes were stepped over rather than read. 4. **RLE blocks skipped the block-size ceiling**, so 97 bytes of input returned 44 MB of output that a caller cannot tell from a good tile. +5. **The content checksum was parsed and never verified.** Over 600,000 + mutations of real block data, 34% decoded into wrong bytes with no error + raised, because a damaged match or literal length usually still produces + plausible output of the right length. Every other guard catches a frame that + cannot be parsed; this is the only one that catches a frame that parses and + is wrong. Over-permissive against libzstd, with no wrong output, so lower priority: a compressed block over `blockSizeMax` is accepted, the Huffman weight table is From 3c9f035b7fbf463c5148bb16b7898c19d62e69e1 Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:20:38 -0600 Subject: [PATCH 11/18] Build a zstd corpus for the branches real tiles miss The only fixture was one .hgtz container written by one producer at level 19. An audit ran 16,490 real blocks through an instrumented decoder and found thirteen branches of RFC 8878 that corpus never reaches at all: raw blocks, RLE blocks, RLE literals, treeless literals at the three-byte size format, frames with no content size, frames with a checksum, the one and eight byte content sizes, the three-byte sequence count, a block with no sequences, and RLE sequence tables for literal and match length. make_corpus.py builds frames for them, either through libzstd or by hand and then past pyzstd for a verdict, so legal and malformed are libzstd's opinion rather than ours. The expected plain bytes of a legal frame are whatever pyzstd decompressed it to, gzipped to keep the fixture small, and the manifest records the exact command behind every frame. 46 KB in all. --- test/fixtures/corpus/A.raw.gz | Bin 0 -> 5023 bytes test/fixtures/corpus/B.raw.gz | Bin 0 -> 82 bytes test/fixtures/corpus/checksum_good.zst | Bin 0 -> 5014 bytes test/fixtures/corpus/dictflag.zst | Bin 0 -> 5010 bytes test/fixtures/corpus/direct_nibble.raw.gz | Bin 0 -> 8744 bytes test/fixtures/corpus/direct_nibble.zst | Bin 0 -> 7974 bytes test/fixtures/corpus/fcs8.zst | Bin 0 -> 44 bytes test/fixtures/corpus/fcs_2p40.zst | Bin 0 -> 27 bytes test/fixtures/corpus/fcs_max_u64.zst | Bin 0 -> 27 bytes test/fixtures/corpus/huf_weight_overrun.zst | Bin 0 -> 53 bytes test/fixtures/corpus/manifest.json | 157 +++++++ test/fixtures/corpus/mixed_rle_raw.raw.gz | Bin 0 -> 100 bytes test/fixtures/corpus/mixed_rle_raw.zst | Bin 0 -> 51 bytes test/fixtures/corpus/nofcs_windowed.zst | Bin 0 -> 160 bytes test/fixtures/corpus/rlelit_fmt0_R31.raw.gz | Bin 0 -> 23 bytes test/fixtures/corpus/rlelit_fmt0_R31.zst | Bin 0 -> 12 bytes test/fixtures/corpus/rlelit_fmt1_R4000.raw.gz | Bin 0 -> 40 bytes test/fixtures/corpus/rlelit_fmt1_R4000.zst | Bin 0 -> 14 bytes .../corpus/rlelit_fmt2_R131072.raw.gz | Bin 0 -> 163 bytes test/fixtures/corpus/rlelit_fmt2_R131072.zst | Bin 0 -> 17 bytes .../fixtures/corpus/rlelit_fmt2_R70000.raw.gz | Bin 0 -> 104 bytes test/fixtures/corpus/rlelit_fmt2_R70000.zst | Bin 0 -> 17 bytes test/fixtures/corpus/text.raw.gz | Bin 0 -> 3221 bytes test/fixtures/corpus/twoframes.zst | Bin 0 -> 5048 bytes tool/make_corpus.py | 387 ++++++++++++++++++ 25 files changed, 544 insertions(+) create mode 100644 test/fixtures/corpus/A.raw.gz create mode 100644 test/fixtures/corpus/B.raw.gz create mode 100644 test/fixtures/corpus/checksum_good.zst create mode 100644 test/fixtures/corpus/dictflag.zst create mode 100644 test/fixtures/corpus/direct_nibble.raw.gz create mode 100644 test/fixtures/corpus/direct_nibble.zst create mode 100644 test/fixtures/corpus/fcs8.zst create mode 100644 test/fixtures/corpus/fcs_2p40.zst create mode 100644 test/fixtures/corpus/fcs_max_u64.zst create mode 100644 test/fixtures/corpus/huf_weight_overrun.zst create mode 100644 test/fixtures/corpus/manifest.json create mode 100644 test/fixtures/corpus/mixed_rle_raw.raw.gz create mode 100644 test/fixtures/corpus/mixed_rle_raw.zst create mode 100644 test/fixtures/corpus/nofcs_windowed.zst create mode 100644 test/fixtures/corpus/rlelit_fmt0_R31.raw.gz create mode 100644 test/fixtures/corpus/rlelit_fmt0_R31.zst create mode 100644 test/fixtures/corpus/rlelit_fmt1_R4000.raw.gz create mode 100644 test/fixtures/corpus/rlelit_fmt1_R4000.zst create mode 100644 test/fixtures/corpus/rlelit_fmt2_R131072.raw.gz create mode 100644 test/fixtures/corpus/rlelit_fmt2_R131072.zst create mode 100644 test/fixtures/corpus/rlelit_fmt2_R70000.raw.gz create mode 100644 test/fixtures/corpus/rlelit_fmt2_R70000.zst create mode 100644 test/fixtures/corpus/text.raw.gz create mode 100644 test/fixtures/corpus/twoframes.zst create mode 100644 tool/make_corpus.py diff --git a/test/fixtures/corpus/A.raw.gz b/test/fixtures/corpus/A.raw.gz new file mode 100644 index 0000000000000000000000000000000000000000..b17235202118d0d6d58ecb2e95527e5812778b62 GIT binary patch literal 5023 zcmV;Q6JYEgiwFP!00002{{e^-ckI_(NQQ9dqL)JHf1~_O^gJA2*ym%vYLJ2S>0v(> z4-h@_8`#+n(EmjCq(oJq3bTB^7Z>z-$B6$;opsjKZfuK_UOI|`l`o$;tB5-o!yMa2 z3m_^qYPY1Op0A}3AEVyGAk?_@)ljOSw#QDuwTLCVq}qOW-|BURFQE~Ox?Tq-sDBmR zCIII;`+2MGa3NLgJpgtUfVw%vN!Wla|}8uzZ3Us|3o#M&FG<^9xq&YJij$*$6aR=bdP3$t_wFi^I4F(Z*zUbVbTSo#bdNy(%EL9JzN9z zI|?Q;#kTVck%2+~#Q0+sP0O;}-8iWHy6|7NcE~MS&?Mv|EVHFnC|{;*U+8x}Jks?G z80V&gzw==te&5hL%Hqhi?E*&7F)-EnE^}C$(vjcVjBRmlb(8vPRrhgH(UsC={sSY% zG$*RME%?1kF|jnui_^w~Nc94<>(U2%c9;tO%&Kn$+qnI{0Jt<-wDPeUIeEo5hGfd6 zT4W^>36^8IiR)>kpTM07I~pvp*QSkQbcJELz%Bah2Gv2vgzf6#Sc3l}M=Drd=gDqR znf~L&90AO98E(|f47cdh%&>m^tke5aJo|Kwh?He(g3leKjE2O6<~g#Uc`-)$%qxo@ zZc%B4h$O2o_urG}D(|JORd7fHS1mOiemg(E3d`+$oS zPF}#yH?i(})aOXU1f@{&e+%CRZCI?jf5d3Z`qNsL>2zpZbD?`kL*-B>VKV<}Ierz6S&JscmvO?vIAja`JV<~>p;w>RGf{^wZX_{-WX-*?-#1~na#|;s4 zZ@9No_dCB@cuTmXo2t08PD*}Zr5sRj;g6s}1qOQfuWF6v0!P$LCez`9cGns4+!_ym zjv7h>KC+!ZBZ%*G!7m7*+ssJ*#xOgs8lNKxV`%-#>hUBEg3&TWsPoDjZjULL{LmIR zKM6*?Xv8JgCpy2L_}?x=;j7P@T1jas(RLTWCP^D3#kc}uvSfgd;{s5R4pGhnCpTlj zZOPg~DY{I_p`f<{QG9=TAWYuSVyBnS6EtsmcrUsF)KR>r>Jr zZCHXSD|~GYKf5$w?_k{P6f~ z2)Xv_pKY76tu6;`n;DlZkq(f|o&MS~5!ogr|x z{XRFbA=EMYn}5kL7!ukQ8z@R~ZrFOclR9O*n2q~V;W+z_bzappBUPF+mbJxdc+$eL zS9inaO0g56%99@>*Wffs{#eQj0xN9K6x{iku)* zNYf6PxH@^Tp`70&aXvMmX3Z!-jm%Ns)_Yj7>(+adlf}eDv^j&0xk%x9%y^jZLQ7|$kOXqI-Qq0*G2?@MU z1YgC!NB|Hj$-G{DCk}!C%x;=BY+)@nlSUo6gfaQrTw$};+TGagoJgLyR;I+j@TwU^ z6m}yYEMTsUmOw7FySX3JnN+uA9x8>M0Y*U2UE2ec2_5*y3D+IVGau!48~*Iu3Y7(u`^Rbca$kXxiwLuUTTdTIjO*NYV?Wd zPpoMir5QDZ4h>950Ks3~=VEn>?&g8sm zLW|>voS3V5RYhuN;eyHc+P1i0PZp8yv5!(z>pl+gH0gAlxemZ12BC#qLi89 zlK@QpwDStZo6r6>v@o3vntAr%kSTL;&GsQQ?ZW-c$$}(wfT+>EQ#>V1cuRi+mWl#V zCU(lrX;-FNXhXsW83hFU>gvQz4`}Og`uKO^rBQwumQOm}x)2j)92mp%AiIS@Yqf^o zX{yWOb%9>5Ihp-H2IHv&KU!qP5o928_2 zLxki7Sh?LaqWAP!8GfZy?8i&FwTre!xk7;g+9!V1!;e~^_FuUXXT71pf-{*J5~~)> zwU|{BYMG{;5t5Bg{lA@L_Tg+B%C60zFujKiaS}og@lOz6Lo;ok>UGFP7bGZjDEd8V zO4PmwueyKAmfIj=$PXr{ATDmtAXCJ?#+yVqdN*2VS+Ywp9s||FaY?hg2DI|0UCf!% zus%{p<-c>mW_MKH*2DSc7L||1I49;9gU@e(EV}j6Wf!G;X2=2mw`Gy$yinbFS69z& zS5lv)o>hXqnoAd;9MN{N5#&(QKMjzOc-46!YzcOii729G_MH`r^<_@(jbI)UBoZv;J zIanP3K03M|ij?4TdKXU8v2t4SmQ3u0(VlBNJHXNye}j@FEg?@Q6y@|GA0S-d^rI=X zf70z z-tO`gxqZVA6d_mF9WUcZ0n1tSR%Qp>T4ilBR^(6iZf&3ua}MIpu!hE3 zSCx1Z`|@*yquQZ80_@c2cy-*(l;Dbc@>&^6TmfY+j}b5j|D+|s5ohKcZi8SMS&>CL z;_TJp|ImtBcL_OW{eL_5mOzL;?{gxnWs;S2!Pdx@#H*bd=uy3la{B+z;;zQ|YYD5!>rS-H5jcifmX9v9D;CHG#XU@489jkwUtOp=bVyHHt^ zoZOVmqrL*hgG?g@xfG_t)p)R;QY{j<5Ad@e|A^0?2nH5|qY5oBc5-n5Kx#UBoR?au zju=imUqZy}MH>dOi&4M?V-;Jn?0dFS5PX620&wt;qiSiyJ`ljM*8$F%WNPFjDgWtXy1A3oBcZetw82z(Bw|GOe z&9~(SCfVJiI4g|ut@Iyr6hA7($ERLJ4)l`jY+x{i{X#{|=b#&pGS#IuglHW31U|xE zzkGzzZ`RF>>tKHT!%ZTLxMG%9FjPsENB*H3?n>9=%}fd}fB+$zn+H(%$01R&<^|WX zN)n(OMC0n3xFJ73d{odStHd4bb`>AH%n|NY<{kUc^9Dlpfyle`!IVp5oY?16&$Bv&zQOH6 z14SQ`R*nGJJE;1;IpRl?{W8*@mzpz|G)Zaze`cZE1mIf9_z1RVa59aLf^a5eJ9>HU z47vz(mh+~Lx>PI~SVF%wgAQ!_A=9&p-cMP$-!B{Yv`m_fe+UTdA^iQExDN2dkiJZj zXBESo5UBg#0Y>};dXlZ?o{LNbm~#uDYO@oSKGSq`rZjH(GtWf=_KBoxMb}BAW-Y7B z_IKN>$rktOnh?VL;bHHkU&rgxh@yTuYhej?&+GgFvUCJl&Jdjq5x~<+qq;*es~x3k~;UTa7*&E|KutkV!ASpHsywOX=4)T^~?N5|(UB zuq zg){Zgrm6J8@G!T9;jU(a;=(OzeEm9?NpVFIdu$sy5njV>8P&IylbMk5)U{lNEo2sr zZ$;)F5~@wX^=Lx_)V$@PTD7>Mdz`8R@+ z+Ur+5kc27Ld4G&&Q0`H%IdyLVJ+xrqU+K^x*9;Qj{Oi>K&+a#rgMvF#!7hg$Cp>>Fc> zmIGZ<-c;E0-Y;KLoJ>1mtv^J3s|pqCX)6uNbL8;&zpVUpB|*&0R%rln6A=xpg5Cz( z50l5|@M$W!%z-{~j&r7~dkdrRt%e&Z9bSWIvS?&^!r`qG#0g_BKztsT zimZmIf*!Q}`vv;NUq=wk7B{@MZ*3~N7>q_8m145LHhf#^Vlmbe?i65b?g9J~KYPPN#6w=Y zHM5m1^5Sh$)1Bk-$`6<~irsnEVT?HXGkeNb8BhrS_3?qsS!cPhogc@O1(hXH?U!}V zgr_QrJ<Asw*2_EbP_8KEwOiKiG?tmTQUimvoQx7Ww!u!*yiZ?95 prWdQf)VEeI01-UZ=`R_Gs(xKvpd6K7vQ9IOehSYXl)s1*005IV&{zNf literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/B.raw.gz b/test/fixtures/corpus/B.raw.gz new file mode 100644 index 0000000000000000000000000000000000000000..6d146b9a7795b0411d0895e2f5be771ea793a614 GIT binary patch literal 82 zcmb2|=3oE;rvGnGT;yy}5MekFdN0Q{h{KlW=&^FXhwncBs&$n{qN=! VpR;_9N#vGs?=85)BB0K|008G#Cfxu4 literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/checksum_good.zst b/test/fixtures/corpus/checksum_good.zst new file mode 100644 index 0000000000000000000000000000000000000000..a5b91178f98edc81b327ebe50068dba14c457882 GIT binary patch literal 5014 zcmV;H6KU)ywJ-f-h!R1Z0M}edhH&SimqO`(qx?)vwXf67xa0@i2qHUb=K5wY>Sg#I*NjoFP}N9h&vd=9NR_UD)Lp%II^UI!9@5cL8__|Qz zQouotwNv~?r<}&cW%w2wIVnl8a&NUvE?6A<*8uFe`yi*Oa8jZa+DdvmNGXs% z<1%}ON7pays`0+Sj&9*}k7j_b3pYFSS&+JKbA7~N(gmZ%W3*n<*=C?UTm$tx3MMke zw(|>-fkFSo_+u4K%d*_vIH>%(@L#re$SqpXB;+G3v!zxjU#4td=yyIm()9}%=ca?d z^I;->-_SeC;>fk_0!Gj=FxB}kb6A_wk>A>kZEuIE)z?}&@8Z5Ebrj28Cg<-kCE&A*R)j`ID?dsuJg8w5&Dp+0T$!<`Y{^P|Q z0nBt6Zq&>Sx9HQ%uzvik)B939`*e+nlx1sz&mE+UhQx#BIkKR6F-G~!D~lg)QE7#U zB&#p?-;?Jm@1?C(a7Y7L&PJ6(sfs zh4U6Fg~Bn5KkW?PG+3->0d&D&0EvjS^_7fD4l#e~O>> zumz*M+LSK!Hp_*D=6aO8G`_yrmD6~+W<31|qay`Hp;vxMEmR7s*t}+=Saf@rBL#J3*QE9Sgg8##AwU<(^{74bZA_2p?gR}eU15QbL=~5<(5KLgkDg#_>91DSp!8Eg>|5koFmAnrpCWP9to@7g?Fd4H0v1xVKXG zJHJ|ZOSq()s<^XGN`7Ib98hrKkDx&X2736fYK`UsN7PIv)8T@4*BSBL8V`St8cG8` zvYkI8i0^d4F9@OA%t-#mFgvaqpCbulX#L9S@gxm`(K1A+^U51;k13e^&=xm82}ZqW z#3k1!I=`Oy-!4PptIwKRNogt3b{D`VNgE@@xB_CbWPp$30#J_*QO*M=H)FtU$=X6G zx=hKTptk~1e1Cc%Oy2!b;2ZaCP^CG@M?sx%QzUqjyGkoM1j1<(8S?goH|e50p6^F^ zwIPGTEN?{?7?!)WW028@Xn!AwA7*HIcXw6lWWc4Fe0Bh-$_w+Tm=hf9Q_>@Em(p%^ z55KSUY{2y-lhLZxlE<}VFbHq_dK;cnmwFfXW2`He+NhwGC@c3;Ax%TUy zZJV;KE(dL!8J8@P4v@^9{@OAT+p2_S`TQe5dFz5u^A_7#uB8|(Mm9Fgde+YjjO4+< z8zCk)m5W+Gau1FwH9uCVZzyN;O5qY^zQ$*Pr$XZk)3e+WvKVK@^;PlgROIVqN9lLQucKiU`ysXvqUddcs?9^=+Xm zW&FHR-2>$ufQKXPkNn5niSQ6F2WtPFA#k?+J~y!; z)G_*-f5|Wy6516TC`xf|*m}8>I%T|=jr&sJIQx!uUez=sRhl!FwZ&?9(!#MvYiAv!kvV#P`IpIA`L>ps$z-<=&HjS&T3*tDlu2_^i#<^sywUrLoFG$3(+-)q zI(e|6oZlpIJ~g0b(O2#QdQE6&^Bq2j6(y0f<6ro_PTsu@HSb|W7w zV6KgpKrXbqxgXP+RJUXvDutZ^Mys+(Nns(=nXW-yh~YVNJzT|`49%*GbIo198A_wk z?wAS4rjfiEe;e6V&(8cAXu!D;cl0#+SuQe-PbErnfZ8|Lc}z%8 zQ?(d&S0|;q%)EGE4htC!#$(&AT;->+GgCr$lqo^EHBq8oYK;g7r9 z3goO={W|wyFnu|ql$qj_08IU~ z^9seA&;B;FFr5sVdG_FtDRXem_8~Ow!u`z2f+Tc+sL{PsJS9wcOMe5FiULt4cFN6Z zSEgEML&66c1qA!*>cmYCXzOwM_;=!^QGOVfPdeSY5EEt`7{l`*yM;k(wT9nms>|Ya zfnKjUnf*Wp>DA_U^xr@@Wx0ALWi~R`+;&MzYJ#3U4Bkpc3680ui3CTyANiAYL!X!- zYXHZGRBk*LwtySJCiX$~k#Wf4^1ql}vSnzEQ>Q>#- zg1`KDlWf6fzUD-Wj+tT?fC1Z-#>RjGJLwANUW!qn;%Np=_+F{{rPjOgdzUE;SYwId zcjX4rk*r^gKd=9D9f*A9e7CsTRLZx?Z$ByVkg@B8SVuf&LODzkU(AKo5nCw|q#k6NJiU%3%yy`jN^Gnp9@s}{_)m{k&L znWmi)l8sLNznx_E;cOequFaq@y@w2O5<(C0PY_>2Gi{&hb;v~*Bq($!`aNk%)V>F= zx_`=++aO}d4<@J}E^f~tQ^dZ;n?yKzH(F>}vP&@@1J%NDNwd2KwDPB2%$d@#K2k^J zzjMK6cU0ci!};bGm5;?ZC*~M~&u@S%y7kj#7o~e<$N~SiWs&B*P~CY~SI=%&QlF)s zRf4^mOBbLV*6DPzTB|~GF7*64cGt4KYo*7FP4sQ3Sl!X)oYF>QiY~3^FZ|9)9!ln+ zVK*}T-X)gNY?@TQu*`B4^X-L9O`UZdpH=_*;?PU-48x?^mKD^BMp43Q$VblDEuplO6871~q zIz&16ozKpQb6H}4Po;?8hzC^j{?11K@s97vRL_ov$YX@Z=~yv8#t@%zH2TarPm;w1 zZ@5L{B%Du)vM40PAPjd`JQu&sfq~W$

&p7v+?<5E1AN3XR^z`_|ve+vKs{?(!75 zeZvnFAy?NOFXKo7%UShSuPaVC$8II-zUd5FWo+?X^T7tfd__gl8&*vP+60l+?33t zz5>RBOd|!k6sE$}c(9&QEfTj6@UtKPh|it~1{Q;(3N0{pa&Z7aYC3zIms+Wg7*0E1 zLd5Jv8wRn9QNRRa6m{j)*0ctf#2xH*6!f1JvdbA-fPOAcgsR7k;GG#riafF-$ojyo5R=@8eWF4X z8^1#A8qaE*cw&A1xMndNf4kAVYznnGDRf(Gk?dFGvq;I0n#2g}v*f+L!RRi%bNVa|@tqvlEp*({yyEG;aAb&qV_EiKJ^q*GZ#hEvw7+ciXGU z7We9!5W@W7Veh41$LrFFqJBARVF`B6>-+(-bOc$>5T;uZ5nOuJ$ojKZ{n@?pvf3xS z$u~HS>pV&2x0q+xEU4lO4fkVPjXoPLk?`A)NiV&hQ^hPx>DM=1A5cmXmTXJpsC;an zGq>+xWV-Qc12LV`A8JN~T#0!+I=aWmb@_0wOLYXq-2h5XRS2W7(EUk8W&_%VGxgA> zsr16|Ft>%_u4aPb!Yyii{W_OPaYYh)Y#TWdUc+q})wh+CnUL|+wOoZQWEPEYMdlt7 zs!hT5XhQ?kyyc-sLIG zgelf}e~f2P?oqHgb#DRW-xtc*NQ;}Sg0I{fxqBKsKQTlD;Qv)F>B%nEKZ*%c64uLK zruNt?y3Rz{UcMaPaS=eN?|MlzFXSf^kI&!)Aq|QS>6Ea~e%-_WB?xGgu48i^&L%;` zCBIE9L7-1RM9&*eh@b4Ju>^RZyZ4#Z4{m4>-GNTPb8u@9TtYiUT`|18xsx&}f0aoA zFmeFmC0n|Q=B^N#0OB9Hi2VD-FtXZ9;jI(I31crnd>)sItcIzA z9<=@Y1^UHbM-a>wH@voQZ7RALj7A-mVzR$Bd|T>bG1e3A6ku%b0sIG`oB0R?hcFh6 zB~}e9a|r(zxd%}A1Ex!~==AYA1>{)OL6zl_WUxLxPprbs&Y;3Sd&5J-LteWzvz0CK z;%!pXo#XM!512QK-Fenwj5zu;d&*WBPze9^@qx@)XSuMQAIFmgl_gQ_mvzpBrz(j( z(gMZinD<<4ov176zMQHF9_$778Y5dwO9XW8fF0sq`8!Ed4=Vz~`_rk4H!Q)X7puS2 gw^lCz5j@rDFByoceqCLl9F)vwXf67xa0@i2qHUb=K5wY>Sg#I*NjoFP}N9h&vd=9NR_UD)Lp%II^UI!9@5cL8__|Qz zQouotwNv~?r<}&cW%w2wIVnl8a&NUvE?6A<*8uFe`yi*Oa8jZa+DdvmNGXs% z<1%}ON7pays`0+Sj&9*}k7j_b3pYFSS&+JKbA7~N(gmZ%W3*n<*=C?UTm$tx3MMke zw(|>-fkFSo_+u4K%d*_vIH>%(@L#re$SqpXB;+G3v!zxjU#4td=yyIm()9}%=ca?d z^I;->-_SeC;>fk_0!Gj=FxB}kb6A_wk>A>kZEuIE)z?}&@8Z5Ebrj28Cg<-kCE&A*R)j`ID?dsuJg8w5&Dp+0T$!<`Y{^P|Q z0nBt6Zq&>Sx9HQ%uzvik)B939`*e+nlx1sz&mE+UhQx#BIkKR6F-G~!D~lg)QE7#U zB&#p?-;?Jm@1?C(a7Y7L&PJ6(sfs zh4U6Fg~Bn5KkW?PG+3->0d&D&0EvjS^_7fD4l#e~O>> zumz*M+LSK!Hp_*D=6aO8G`_yrmD6~+W<31|qay`Hp;vxMEmR7s*t}+=Saf@rBL#J3*QE9Sgg8##AwU<(^{74bZA_2p?gR}eU15QbL=~5<(5KLgkDg#_>91DSp!8Eg>|5koFmAnrpCWP9to@7g?Fd4H0v1xVKXG zJHJ|ZOSq()s<^XGN`7Ib98hrKkDx&X2736fYK`UsN7PIv)8T@4*BSBL8V`St8cG8` zvYkI8i0^d4F9@OA%t-#mFgvaqpCbulX#L9S@gxm`(K1A+^U51;k13e^&=xm82}ZqW z#3k1!I=`Oy-!4PptIwKRNogt3b{D`VNgE@@xB_CbWPp$30#J_*QO*M=H)FtU$=X6G zx=hKTptk~1e1Cc%Oy2!b;2ZaCP^CG@M?sx%QzUqjyGkoM1j1<(8S?goH|e50p6^F^ zwIPGTEN?{?7?!)WW028@Xn!AwA7*HIcXw6lWWc4Fe0Bh-$_w+Tm=hf9Q_>@Em(p%^ z55KSUY{2y-lhLZxlE<}VFbHq_dK;cnmwFfXW2`He+NhwGC@c3;Ax%TUy zZJV;KE(dL!8J8@P4v@^9{@OAT+p2_S`TQe5dFz5u^A_7#uB8|(Mm9Fgde+YjjO4+< z8zCk)m5W+Gau1FwH9uCVZzyN;O5qY^zQ$*Pr$XZk)3e+WvKVK@^;PlgROIVqN9lLQucKiU`ysXvqUddcs?9^=+Xm zW&FHR-2>$ufQKXPkNn5niSQ6F2WtPFA#k?+J~y!; z)G_*-f5|Wy6516TC`xf|*m}8>I%T|=jr&sJIQx!uUez=sRhl!FwZ&?9(!#MvYiAv!kvV#P`IpIA`L>ps$z-<=&HjS&T3*tDlu2_^i#<^sywUrLoFG$3(+-)q zI(e|6oZlpIJ~g0b(O2#QdQE6&^Bq2j6(y0f<6ro_PTsu@HSb|W7w zV6KgpKrXbqxgXP+RJUXvDutZ^Mys+(Nns(=nXW-yh~YVNJzT|`49%*GbIo198A_wk z?wAS4rjfiEe;e6V&(8cAXu!D;cl0#+SuQe-PbErnfZ8|Lc}z%8 zQ?(d&S0|;q%)EGE4htC!#$(&AT;->+GgCr$lqo^EHBq8oYK;g7r9 z3goO={W|wyFnu|ql$qj_08IU~ z^9seA&;B;FFr5sVdG_FtDRXem_8~Ow!u`z2f+Tc+sL{PsJS9wcOMe5FiULt4cFN6Z zSEgEML&66c1qA!*>cmYCXzOwM_;=!^QGOVfPdeSY5EEt`7{l`*yM;k(wT9nms>|Ya zfnKjUnf*Wp>DA_U^xr@@Wx0ALWi~R`+;&MzYJ#3U4Bkpc3680ui3CTyANiAYL!X!- zYXHZGRBk*LwtySJCiX$~k#Wf4^1ql}vSnzEQ>Q>#- zg1`KDlWf6fzUD-Wj+tT?fC1Z-#>RjGJLwANUW!qn;%Np=_+F{{rPjOgdzUE;SYwId zcjX4rk*r^gKd=9D9f*A9e7CsTRLZx?Z$ByVkg@B8SVuf&LODzkU(AKo5nCw|q#k6NJiU%3%yy`jN^Gnp9@s}{_)m{k&L znWmi)l8sLNznx_E;cOequFaq@y@w2O5<(C0PY_>2Gi{&hb;v~*Bq($!`aNk%)V>F= zx_`=++aO}d4<@J}E^f~tQ^dZ;n?yKzH(F>}vP&@@1J%NDNwd2KwDPB2%$d@#K2k^J zzjMK6cU0ci!};bGm5;?ZC*~M~&u@S%y7kj#7o~e<$N~SiWs&B*P~CY~SI=%&QlF)s zRf4^mOBbLV*6DPzTB|~GF7*64cGt4KYo*7FP4sQ3Sl!X)oYF>QiY~3^FZ|9)9!ln+ zVK*}T-X)gNY?@TQu*`B4^X-L9O`UZdpH=_*;?PU-48x?^mKD^BMp43Q$VblDEuplO6871~q zIz&16ozKpQb6H}4Po;?8hzC^j{?11K@s97vRL_ov$YX@Z=~yv8#t@%zH2TarPm;w1 zZ@5L{B%Du)vM40PAPjd`JQu&sfq~W$

&p7v+?<5E1AN3XR^z`_|ve+vKs{?(!75 zeZvnFAy?NOFXKo7%UShSuPaVC$8II-zUd5FWo+?X^T7tfd__gl8&*vP+60l+?33t zz5>RBOd|!k6sE$}c(9&QEfTj6@UtKPh|it~1{Q;(3N0{pa&Z7aYC3zIms+Wg7*0E1 zLd5Jv8wRn9QNRRa6m{j)*0ctf#2xH*6!f1JvdbA-fPOAcgsR7k;GG#riafF-$ojyo5R=@8eWF4X z8^1#A8qaE*cw&A1xMndNf4kAVYznnGDRf(Gk?dFGvq;I0n#2g}v*f+L!RRi%bNVa|@tqvlEp*({yyEG;aAb&qV_EiKJ^q*GZ#hEvw7+ciXGU z7We9!5W@W7Veh41$LrFFqJBARVF`B6>-+(-bOc$>5T;uZ5nOuJ$ojKZ{n@?pvf3xS z$u~HS>pV&2x0q+xEU4lO4fkVPjXoPLk?`A)NiV&hQ^hPx>DM=1A5cmXmTXJpsC;an zGq>+xWV-Qc12LV`A8JN~T#0!+I=aWmb@_0wOLYXq-2h5XRS2W7(EUk8W&_%VGxgA> zsr16|Ft>%_u4aPb!Yyii{W_OPaYYh)Y#TWdUc+q})wh+CnUL|+wOoZQWEPEYMdlt7 zs!hT5XhQ?kyyc-sLIG zgelf}e~f2P?oqHgb#DRW-xtc*NQ;}Sg0I{fxqBKsKQTlD;Qv)F>B%nEKZ*%c64uLK zruNt?y3Rz{UcMaPaS=eN?|MlzFXSf^kI&!)Aq|QS>6Ea~e%-_WB?xGgu48i^&L%;` zCBIE9L7-1RM9&*eh@b4Ju>^RZyZ4#Z4{m4>-GNTPb8u@9TtYiUT`|18xsx&}f0aoA zFmeFmC0n|Q=B^N#0OB9Hi2VD-FtXZ9;jI(I31crnd>)sItcIzA z9<=@Y1^UHbM-a>wH@voQZ7RALj7A-mVzR$Bd|T>bG1e3A6ku%b0sIG`oB0R?hcFh6 zB~}e9a|r(zxd%}A1Ex!~==AYA1>{)OL6zl_WUxLxPprbs&Y;3Sd&5J-LteWzvz0CK z;%!pXo#XM!512QK-Fenwj5zu;d&*WBPze9^@qx@)XSuMQAIFmgl_gQ_mvzpBrz(j( z(gMZinD<<4ov176zMQHF9_$778Y5dwO9XW8fF0sq`8!Ed4=Vz~`_rk4H!Q)X7puS2 cw^lCz5j@rDFByoceqCLl9F;e=`}zZ3kJnGn$K&;Qy`(<#vi-lRXtPrT^-#dlJ;kOc-;9YrYshX-mT&(5Q?hG@=0cbQ($*`t2ACw6+Qb?#5X&o4)>%gi=VaU;2R2Cd z?vbmFoUKIA2<)OJ_#Se0_4P2MS(gj)lHIWV4Ur#;B>@RQKV?F^X%OC zm1NR^K5I==mV~OpRt9}%K0d~GhN(6T0{JAVtOEA5AhBJGRWaUbEBANw=sTBKJ) zd?;m?paP;H&{~r3b&XGI?#7$bhCB&#<@fpCqL1&mpfsgw%82730Mo@4i|LRHLsZUD zXA7aNYYMCN@4A|~v?~}VB*;%TD0dU2@TSxl0i(%JKt3kW|<|iHUx<3o_ z@vL8BI}qG1bNchLhNVJ2arD+`v|NRs*Eh-ixCC7SGH(+_i{Nfz_kNQz?s*S4M;N9K zk`c~J9jEt^7ltc&D$3?adUesHZ?V1YAbRY63BCN3zG3os){Igg!sgW8Hr7>Q#DsDZ z>TQsuF~OycNb_p~Jqt`;n+ybjUcUD_B~^i$polGIZ7meP(Yq&o%>GHBr%Aog)Js`z zH?+@Q)8YwY;M#4B%T^x70Z_tP4#k-5hDd79dHT4=gQSX*&k=gQ&r$gr%U!*PjAQX1 z{91R=AjjKhCi;v{D=RUlb%Z+e0gaQ%w`pG)dJbUI+0$jE+`HoKRqbt$-xgx z(R*2s%5)vVgjJqc#aS=H-upeOxGyyb-c6+lIj#Zfj3gR}`&cFkrtowZ8@g6+$i1=N zv{@M|-z)MdKN9~KJw06AwhjCPpwte|@syO=7>CYXS~)n;SUvQ8%Q?(+qU)`#u)J?4 z!{VDqpX}F3Z*ebO5pH*~)x{JQhEp`$JC6UjGjwOv(Ddq#O1V= zUo8u4hKQHt>{33Ep_)X}$OEVp%LCoIRQI9ks;*!2h&E@CIi7di#k07aB&3u#rw0rV z`6%0(XijiCw;+!^5_Ny-dx>&VdDP_wRh>v{=Zffa%V|y%$7v(Ro8nzr=C(Rvd*-JtTmQ7;6BI?JWJs_XI(p({-xPjNPRJ}$}7Qs zt`|M`C)VjPx;ci_DQ**VO;Q`Z{<2oqqh|6(V5?0QTCJY-ESMo<;t4d`5_S4PdSi&q zKIl^JMjhbwqeQzP)y4->GI?(seZcr@zOq=)N>fpnul5s#oqVP+7;LBI8Pbpr`LZ+cnF6+Y>zV{AGJZo5fC-su{PNd49`1%3Mdz;N znK3R^5IQYV^ysMM(s*k#@3h{ZweuG!_3;<83INmB1#-Otd&Z*F0ZR4ZH%0btBx007 z;S`6PM`MM{k))@p8q9b=NLbG(#u zJww~O3~WY=figc(CkKtn%ge-KuN?Hr4f#XS8-kC+wk~ttsJ5zusBaY3yNJzhJ%XzX z4`n%J?RnN;h;AR*!{9383yAo=uk(Y!qr3KvOiICCm-q{HN9~aia?n{N$#Oo0Zyelm zCOA(o_6U2EsPL8u}VIEakilG z7UG-3qsEC=S$;lOIP`p_h0O#0rjfbX1Df$tNlE(2pr0^Ai;a}I!W)T_-XK3(=J$u+Z=evHh%DvC*H=I824`>svchNI$Cd>Dw#PN>y0-ElL#}b*ps{Y8W`GQtj@xa5oXbe}CXT2*=bTr_2bk6f*}^1~PwjcbxG zfIkBJPo-(MwFMPV*(y?TSTh+6e`dnRm#g-w!Q-!Llt;L_CzMtfZ`yMcY7#1+3H<8C z*be`zkD4ZZ;oZ|e93OGM7%bDVHogMLAW8L#)L=9Zc|6%`mi`=pHuc(Ym>H5B)BmIn&>YN~#TJBAYhq%+w0WVNh zDlQMV`#ZuZkw(&Z0Qfr9zLUeaHu)Sf>?3&j)DI+$(&4nFtK{tO5mqP;#h z=?#jT^B%e3#wjjAN{)TECO@jOs|q3i_zsR_WljO~5&5m}i9olzb(T!9<;<@A`ta;< z5#p_{J5eV-y>MMptk1K=Yt0#FU&>gV*j^I5wdq!cHOYSYJlg}@^G{zrgYncSq0R;2 z)!em~wh2CN`zn~&`|lRrEI_AU+V0himU@Ln-F@8v>7aAq`@qe=1O2>wKG>LghMdHe zGuoPz&dHeLjpU=Qs49lk&czqU&r(OfkCWU)5zhnD!@{h!Y+u=`wySs1JkqHpB-;f~ zL+&@m122B%CvU2m&I(s04T9Pa2|CH@+fFJXLZbyof8cer(6P9^01cl`FU_d-H|L`6 zeDXE!AwIewM*3=;d9Bp$lvA`@O>b}n++_Q0WDeJkn@R-J;g(#V|NNt)^vM>XJARtl zv&x0oH24j{?rTj{4s)jiy)4pfo%cuGUbINjv+2 z{;Q8I&R%YuhJu;JHUm9FK6Tv3k#P5z6(G&Bkjq+ZYpqToB=5|FiyvYl#`fd-g{K6l zX=~fl?tkwct*PeL9z4ZNjvA0#+|&h~7^)-Uf8X6(c{hOb@$1Z+Ig?=lo#5_*8*;r! zkD83)`+kzYZ`8D|A?6hhxD%yme@glS&Q^_La>xX178M*ZX9OxTn}G z13fe@foK*9Wa@aR>eA1srnSHz)-KRl|_wuK-z9hWg>N6eU zl@9XQt{Lck_3xh#?*LqFI(h{-AZ0rKZz=ZUj{nye_7v+uPllN6k|A)(a=V!>;H&0R zYf+P6+%v4~OvESD@w>7A0rbe8oB*GU(q9vL_c~ciFs4`>2A60-3b^j~ zj%VUMB4Z%}kN<`FS>FbmBov*jYHTrg#AJr5L8$RHgA6SxU5)`{kVho}0^h zXjj4U99w@xWZM3avz!JuFzk@dTUQQf4xtAx;sVM1z{Cv`ua}P}=3J!I4^PMXC|8dh ze)#x2QoB$!fTV5uZEk_1?!F|P)2vwUfw$oCM&sY74?r|8dGebke7Ae3h?*?p!JOLE zDM`1+30pNV?LY0OzvAKV%=#QX9WTD>vT_uHB+<@ZDs z#O+4oOF4DWpJ{ir8W<^Av|Y4^9UtJ8KnU>L4ZnX}v*-O_ZoXPRRJ__>(Hc+dQp%ag zyjGObOsTC!SIKp=1}-DNjU4Dvg}c4J6ld&tH}O{72xUUY&SCqZ2Mza6e1&qhDlyI3 zj;H@4`~V{YPE_pYxJ@5A{2F;C<8E~ULZR@SQM-aJ_ik4ss9$Cb2Py)o<8%`f+|~ls zfk)Yh9lTxpXS+HyTVt$X^2a0wnN~c0zAn6NguwvrJA;@tAL<1*6DMU%VO{r_U!38o zsK4{LByL&J{ZtI3%9#~hrSVnGxo7*~Na$3n;s(Y08KyWWFu`eK|Hq*!gn4VD7Us-A zA{?hD`2fhwp{5eJc>IZJURd|&uFu7qr<=vnFjgh*%&uY;g79+egJ;$Y_m%n9Crd|-A#KtCeZS7-36R{T+{*z&YWMk%ww zrNUk>I4>o_g-Tgh^#;x?=Nocg)L)I1v5Q3^kP-BYK$rG?{1G#YF6HC(^B&M><6PB( zG*{N~u2wMiJQwZ@R&W{wK{_DxU<#VdxDB?y+@;eyv^#PAi{Kcv@8CXZBJ)tBj%{LlA+`_ovZtu*JNb;`>v>o5VJ=EFcL(4d^SLzr+HVgzLBLh9Z*1HYOY%RfR9gE!NQhOjA{Xc63iJ*PA9 zgrE@cJy0CGCdZ{l^T`<;|C9mvMORo+b>;o{eizbS8gQrEn$eJuk+ppg_ca3)@2J|9 zO(BZ5OPFETL{jW)_1k~-^M4$yxrrqrSY>2AX6LUHlEh~6Fknn1=5}kP&aF4!ZMWog z4PU%I#;-3Z`+Tvn1~%!F*!{7<9(+;>mzK%+6+8{4Wl{^-x%XS2#9A@<`=+H3F^0KdgLFvoqj}II9%xvnv9M$`i9}yxh8@sUu ze5c-2L-Cwn0n{V4xn_9W1=-}TC|1Hrz$L1uYqq=k>tCasVm`%?|MJVam~yWDBv@mm zSq~KOg=4kqwa&LOS1pKqz05X-s+iZ6Q<$bkictpjz17E&o<}sd9XO6;IFexhVKo?WU-hBzdz1 z7yB5h*UJvQ?x*L{hrUvnZ~QOsqxOp@Jsw|zds>*M{dG>CY;d!RDdcIR z+jkyw*zv#JdJ_)6sNBwRGqB>N(wfd>Zipn$STd_l{cp=-`Crnduk+=8P^u%gMU^kEOp#$o;i)I_-ir~Hh!zWRSZUYG{-PEgfECzR)iTm2#o42QQ!(>@z{ttO` zHuUA1sMPVx(o0#d`7!<1GFjbB*22F3*9-s4W)g;xYQ}sE0by&L)){MM?fd060T0Zf ztS0TuYQzRC!c9KLj15S7UTE{G!3K4xI~9l(_QY(L7HS<79{2Jc=I91ZbL4!yS%KG* ze4BjKJk>|t?rx!)LWIkb)t@i*J|}HTG<~sH>V8e}{f~w!JmuR2Mh7nH0o0Ea%RzHd ze^O2$jIqIt429S-KmQ;1ew;q^HBvb)4$$HgjJ?&Cxdcd^^h9BbF*Ox_K! z+n>YT@6!GCF340PZnOYB&ep-#1gQH}Co2;$cs&Tasdbg8;rAI;l@dB8bB%+^+Q6-N ze8-lv=Qs&B7#!1L-Qo>Li|g#oG12{$BPZnDlE!C~p2~@%_%GiiES)uC#sR~jLk?B6 z-=jdD6+&y)(=#;Mxb8&~whxhuRt(+YlF$po89dF`OWjsof2H4fgr4mZQw>lPM;EAJ z$mQV-M@tSnrS-b4(;Wwfo*SdNayI5`sVOq>WL023>>dFIHyErNi|8b08jJLtMag#L zEF)T-CHXRUiGP2Jl=y#rC-jbEsTyLz-wI=R{>Cg$bE3p5x&d^!V4^PHyB}s{X?n?x zz%-b0nbH|#l-KmCRx9j~jj$=Uk!q608FGwl(lYpWab$b0 z>2ZfrN>9ril&qw|><(L01eG+>stt}L&>PgqGzdr;W_S zdz|@R9>RtIx_(QF*JJvDuCD2TwP=aXcUs%4H1gTB1KQeuLAi(eSqsVlD4&6axVfU) z_?r|3chA$^uSdALX7utsVeEe0+NyryLE&0L_lCo!+Eq*$MlEprw$BKnkW-Tdf++mT z;0bf>J~L;+=*@SquQGFAs}?^7c$sf(#Z~2*Bq_9QGZqGYs5WOC2OAJW*|uz+BL{6? zKB-8|FtSebjDOgSlnxOZ` zm)2ri1=+y5;TWIzm`mQ#2W2>NYG}}%?*!eN>v^n`C|?ElI$vA|1ZdAer`^CXV5(Ob z&dMb2vzR+InwBJAo8aImS)})Y4%QSLrPYF#XU*MpJzg4Yg)>N#@)*-iTQ0`gCWoDJ zpB!=-XZ-BxmMW<`{T$GJBxe(w&(pe1wW-eG7@bylYWqJ9f`(P2YCg%UA9kA7%G=t+ zLM5<*r#mUIdh;BoTYJ6&6;tQPjKwShuMy3B*EkGn{EtXuHU*fw2$oqW83+{Z zYfK^rH`eh}BfE`IfDzkUa(APY#shcsO}zCPkb4kc*d)DK$b`+uv0*!dQ$k|-q|Kt^cDBFpsq=; zD@s3~K;XY6cg0oB2>24#sykxt;%)c-sf#BCe{+7*{dlrw^N)gWcj~*)R(RB7AmkC` z0}RVFk2!1NKRIZDeYQ-6qYqEqBTL|_zqW#AWmGcz4o)$Jk#%%GYz($D)F z5L=s+?s&+**y(lyt^(j)Pf*1N52XX@wbyh4V3I1KsbP$^L}c8`XKFykXC8)QUaK78 z^9xK49e!9|K8i}azyl;ns$vUgzxol@_2*El&2szaVma!Xcw+RGxez$Cp5o_rTzDR(O$$Y$z6T9193YhWwUOYNdijz711=gB(za!h#rv;>Dk>+j&1dvM%V-HEqvN?-?I z-J-P##&N_A#uAO0aw$yduL7&%GSb9xLi=j34e!|FM%THfdTKisutP#V zlBziq8ZAHM#bmuzLJ>N*fmmu}P&i9nsr5(ZqX~LA7|1e?^sS(^9WXO?l!$i>-g#z# z+|Bfu)OW19jg+izci#@?F;j{Npj>LD0!}?=Zme}WhXo<>iY!~!ET>o9fj>{ohA|kH z7LJ|nlR#iCiB%Y^QuS%D>ky{hIWAyC_ZGAqeSF~vLac}d$!u>W5By!6I*?ifM4Gu%3C<=t>zFfXd@p+p} z%hHKQi*Q+?u%GG;S{|?k075CGYmb7BN5+FeIZ9p<=l?h9g&oKE?mvDj3^+VH-=`lV zDpDX#U&=mko_?AFY&vZxKn;p}&Ow2&jxFY$Q<){77ZujI|Iu0?!EwMpJB#Nv_HHdz z@;4-QX99?YoFaY^+ZIz4LJS>wo`j8HXc#gZVObA2kDd&C@T5s8XI{?-Jf1jBInd10 z^|Zc?Euo7FO@O0pma)8!=-V^McS|dzHn?WFobR#u+e8f<8iZjBoe^;R0t zJH=f0h6qu|br&$n*jCBM@1^ovy`zQy)ngRq7fBOv0#bYt@87c4MzoEtUkdm`?+qg< z3N=LOyM#N23Hy^Za@dVj7JL`N?4jfM^LtD@sfX$2yk5V=`QBdo z3>cbi|BD0jc^}Iq>JGWKyBUX`-p~$H)SL?20ZRNj&_{d+5V$tvW(#0@=ikD%_;%F? zHL@c2RrKcfQOYlz&cx4gym)Y2`jqLs9&|+m+$-hCs?O#3B+$U4oiH5c#%N$o`Sz&O z@;N>oDnu~`GOjTs1nSbVXQqE8viPKi0;or+KU*%Vp;;$tERJ^zsj&bie~g@7YsaH` P7Ms3W@cRGWA#DE#aT|F^ literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/direct_nibble.zst b/test/fixtures/corpus/direct_nibble.zst new file mode 100644 index 0000000000000000000000000000000000000000..cc5ebffe53fbb8d0ec744be84d8012644c282965 GIT binary patch literal 7974 zcmV+>AKBn2wJ-gk00062)%XBr*(j1g0000ILqtV}U?l(;04xAc?{1@-q&ibGoY|S# zrrFZDq?u+)(ljH@NE&I-3Ka$g1%*K&ljh#G?e<^TzW1j6{_7V4?(={B`dU*dVtx)e zb4oeIh%jUd(F_e4Vo*isB8Y2NSz;a}I2e#IMp6zr4HL-9QcBqY0A)^h$QDzeT!W*5 z0%G81;I@=;^LS&3x+l1ZhRB#h+*A!DLyQ?SCXM!}q`VLN0b`+93o4gz2R$Or%+05bt>2ml1|1pok40D}MmOn^ac5T@;*RDIh3xHEpZNfiJG ztPkYp_Wx5a>jaza39kGBdjG#CSV(>*Y<5XnfRCWq2(Ye0Y=eu${M%-M1b5*pXA}1W z`Wn`NQ=hOFIQqZL2Kp7q+$7Ql#rvlaqU}oK6 zvjox(xW6zEYZbq*Kl`#?qcs0#L4KMye{oC`=rqm&J3{+?B}TbFtIGYX3uzY!%ie!_ z3_g<*6o&%g;)Y_1{jETOUw&u>{q_G+0sVYfutARorep&s^|Qn0L^1aNK7ou}An*Mw z7yy2FLjp&ES|{hk4uI7Bzwcu9AU6O#|4aJ~7lLo1tZ}H{0x8sRO5>UKMCIYa~@ETLjjH- zA9z@AghscL0P>mmEWj8y0rma*1&Ck(;aA#g4J;LWFQoMT0P%ks_upFKH$b}sq9qVe zAh8fIJ8ieZ{o}pVW&g_O=ktqvuRywI7%*`IN?X5A6JUoPuCpj3lWIr9eZdUSetj?n zW?Ur?whw4jEfG)rkw5T97iMC*lY%}Dd4EnT0k~F|Sp!WE0AGMI(g3+{1ly>=p8-rc ze?6X3T%iy6{bxX@^&21%yWRzA+Xl$l909=O1b49g0Sh31js*YF0EYwAOknYVri8xr z?~Mtn4VVv>3}74}y1*as+!pvVpr!0jZ$O|Gyvp|&jnN48C8Yl~8oa=X0Js7swI=uf zi8n8R8Sq8GeElKojoEjlKi*kH(|}=~4$mAQ{ND}033PBAi{bCX-LoH~182Nfu9R+!Ga$Eiv!DChkb#*v3y|5EzlRhxPxOp`cprV2Jq!KKH%U$pYI0n{WiaH zz~4c4buL{1H2Q}{*Lzw13+h*k=)*J!5C6K zQ^!49vcCbi3BaFLf(NQ4*=X?3oz|MjP!mb{(g9CjezWOnEU7BlLhZ9&hrJw@B|4Pd+B~nH-HXsN?gUXU~}Js`B*?zJZ;qdTCR`J|J(dC z4gmT28qv;B)z6~_=jj1i;U8ZlLD%-*A-&D(xc~8~^&OKSt@}%mxOxOSd|cN5 z%>L;H_}5#SdrI!sFTsB|it*INcEJH7WSs7Q2`y`Q_5mDy~yP^c0=d}L0KRB2K@8) z+y|gb{jV0x0uLCa(d%?+C3zumtp9xd^KQg{KQGLHx&P6jK%*t_`dW3{waq_Tex3Up zK7x<<2cRE_$bWDFjE_soX|nNTHmX0`T=HJPrrL}bvd6=_EB*|ilLz8u_x*ikVqx*; z|Kfj6i$%MN$>*jcd0<8d|m)j9lC_Ju^kgUz`OuG zz`XUb^|sdki~pvfN=0P=t3%a{D#ZGSx03_Tu1S|<-W_1A;mB?t+Fv_hp7=Y+mEUy) zCm=*G-e9r=X&GR!Yu}Xe_RyXFB-|2UGY-~^GQ6EES*;DFJf|f=^&Jp59H24|S-iFz z#F}Gj%1n6x85T$2-`SuLpBKx9?!VOVC-uX zwSqTh-(Op83*@rGO#?ui;EGFdJ_n?iZ46Gl{WhN1f7t=50e%52<8tt#+6K#|UP;aB z+5xbA{JswP2@l8`fJ5C5I|xS6$U2B<0kBgcFu>PUt5r43g{8GY-xvW518BTe*=3As zY!kpv=l%3WHo%2TEN54CzImV?5_8@Het^87<#zVk)z$a`;Qasz5Jn8fX-d%ZvAr)f za*ieopaIo@y!-WH{$Ik(XJ|TP-;3E*W}W%(6cFj&;p2A4h#=-nu;Gga4x(Bnn_1^5nF43Jpq z^614Va9q#;NEGKF>p9N_7D#X>FFrm%OJMP=lRyme807(C{2$V2;Z!W#1y*1Bg8Tx1 z0M#D=pp3N}6?5uzUeX)-08v#Uj{B(pi5X!E5CgDnasWthsAqkDak~j_<8|VyYYib7 z?mNFHI;Eqc27m*(0uJua7qjMq?E@&DpC$ofqUe>9*)-onfo_Kj2l^F_(!vL>udIC4 zYEoI}yr}1gck6i<0D3b3DgzokP%D6T6;wC69Q~0`k+9= z0fNlp1F;Ju#$)PwUpDM5Kk?a8kl&CYk#BAEf&2TKF`28g0sx)&$2hk^B6ommRuU5za9 zy>#~gMiKb;frSHr1D*l&zy$|1Yj&~L?Na}r>F)g@T1){75&1kJ2^sZ(4e^K9YPa7ZFdfhQroy34M;6B1p;LW`0#Ranm(BV*= z1N48h7qKz%M=!$*j9G%XlL9-~0qO&&4fySN$swKJKlkMi&^lH^_)m*329tv)(7sXq zJ%b$G=vsYmwnYc@dkX!gpLYt!>TP?M0#Z`lVvrT#jS#w(0*cTF{QnQ9ylV2lqB^}V z`^{gj|K$yU@Z~_phBY>z3TQen{!^>``j4fMjV%LX@@)HmnD*%S_yO2H3|Qn9=>fJ1 z_%z$V2cR|#G|)DmKhfvNzWVYUwr{}6=Aj0&-_3x{_}JOezwv3jad@8}06PGj1^Qp`WFH{QzfJF5 zx(*QH_z{s6u&=&_H$P;!1AhRR0;Z+88E(Q)G8slZkCZ{sJ9hiqJAW$wa^}Cfr5lYdJ^qgRb zA5Z`PIH0ky0>lPLf?RC+F71WN_&Drc?ZaRRV_1uMp?9f7olRVY2XwRmRBZ_K0`c69 z{e=$~TA9l8-Lkbgc|U;O<3i@DOlp@_)8i zU`!>HJ;dqzZ=1dM*bjgYaQA>azOdZb-~1v>7P#*okO0C;@>OQ_8SMn3q|$CxJ;?*s z12riLoc*H)`+fWq+b-lnM{^0wn4>^T!=D#v*`r@7_?ZK;F3|i2KR>1m^CYk!H-cXU zXq&(8_vQ-aFB_{6_jnH=IT0v*5KDo-0R83<$^KN=yIMd*VWdVpZZ+Aoj}sm+3TztW@A{2zzdY-^|44T5f!i)JB}O@>H$a>n@lA#4+omY@H80>A(EjlE z$N4^({(qHqyN?1cZ*d5UR@crNtLP!2Awy6gP=dGy;3xfTyS9llzdBhKFR|F ziU|T%0F?JFPZLxE*}x2f!Nze8>Ns0T#bS3*nUZsoi*|{HMM25ueh_Q1ENzbDWJ+} z!hK2r$v?dzxW?}L4Q2Q`>=udE2YnD&ACpfxSwMdXfj58uV*uO$*G(d705kc|k3JA; zW8a$PQyC9ac4q#7Kmv{ic+W)rr~kA%XdIrCvGZ|b%S3S#^)G#5+pxdfELq{d14FU~ zZ0187?6@5Qgs)vOu&e;k_IK`Z*w9kx$Ki$)4;2h`D?Ng#HJ90FPw<`ET0YPL+9g9i%WGdEm~i0u z_yF?wN2oo3c*GWb?atJv*uww0V?Yp~$LAD${nJ`Nrfyz6zYfsNhxu!O@c$p)zsw6< z-bh^s$!)e5KYnQkzzt}}GXVQhrXbNpAZcE}?)~=9*8@Y+3|Acw7aEuwO&6?cnf&hd zyuoh__fsEv>z~^o)t`-zPM~erpgFJ;;!M^mL|)bYU$#NAzfZxf3n&KU(Vc3d)oZX$ zc@crX=l@}TNPzE@g!h0oR;8_t ze`W}is?RY)%flG!eJT8T8A?dblmI~nD*PCBsxrB#z{Yf2N z*0N-~7(Cg*EayLOz@w(_bLI&k!fOc<)o<6(BX7}^#9mG-J>ZAO0WlGP>i7kstd#-< zM78H`aP-wZB?=P$uhM|8H`g}YcMww&&9meO!hHpd-_~qD`^M*9sG?F9H#A=UZNL0a zkkt^GJB05hkE|Ab@#@*i^3=urNB=&apFpkF2A)FcGy41%u41yW z+I!!)3zWbsqw=bp#JI(C0o>FPF>Ocu7XL{REStTE2CWGd&h-ZNgBp)n5(V_R@oGZG zsHEV;)h^d96L9b;JZH6xdVOW3Usb2WOp)%{rauK7@YfRr8u%GBv?x`r5&8uylqksQ zJ7zw?jYkx%jw~BllI^l_#KL=GyNF#L(7dB}A5>B5oJ!dU+3$E#UYfetKl9*lcZ%BZ z1JfcNsl0jca+aD;R~kpADbmlWB1Ghszb8NntNu~x^zBJN^nnjrGqkkunvw^&T`zi8 z=jbDjihZLd!iiitVi=ro+Kj4Bix>N%6@uuR;073H>e*fzK2s59kv?gV_a%TAQz9_#n zdCOQw15D|QZA)iUa@R7$&vG8@87FZx)^Gju^L4rAz zcT6e!N+mjYd^2vA(qcg0VO_iQ&xWySgU0hIh`)nT{!dHVrrG@irA>aQelzEvm|()c z^q;H_-a}Qs7&_D_%{jm^g(n%1hd;V^91hr zjTwiUs7%}zIGL$4Qv;@V#n;#Eq=}Z2+yd^!?RvLS+^iaNcNBS^Jr5E#YW%0lCeVkt zG5PY(A0T-6?UC0PA?lcm=Z&WD#QhX8=#=Mk69acv_ya@nt&G<%&|G&opT^YjNqk{a zr@)P0;*7z)qjO|M$(^K>`DMn&@wZa40{5%=dVqxThokY(NAzUaqharh;M=Ax^kH-r zqScOU%!dN+oncRTvuKNtj*bA4m*q$aNp|_9`it7>&;*_YPuIHBQn;~=XQIvPLCxnn zf!=J3W^DgA8%J~c8<6$Q-Y3d~?Q(_rJQ$$h_S(2~*Q?bs*;b`)XLbf$=eVVt4RCcU z>z9WV2+E~m1gQG-0M}^Va!R8QN%rWStlS#Ab?4c{Pu_2IwAnT8oqr6AoQ?J3_T({4 z^-$!r@oTDBP9z1jVrQih(-7I(>{_p} zb@G(JE6|Z)D&8)6b&C#)BfKKUSP|ZWn$lbDTq6G<`~{|Eb+wKRy(>5xyi$BE9R{q) z5CW9TT&f>e{-0pHl0X~&3j1dto?r0-%)Ar+DH#B+!a>7Le+b0O06eO+HHS3Rc=7mS z*_UpV7KxQ)QDmo&0!B&6?8X3@WU;5-$quRyeoUn6V7d@zXgxZL8JF$zspdsLXBltD z`TKl7BtqB#wEP29)!^BT-P8N}ZO5S&(JvMQqe4g3S9BHNrdvGWTst;D3W zc1Ngti0i+vE?K)`S@F*>?{7}jtNU}30J0)B^S{tpc=;rKN|7@BzqvtJII~vE&D#Om zl-ZkLpdOKY5>NdV(0@jL(u6%x9K@pWR@SBefI;<4yT;Hgao#&6pC-6t_V$DNa6_*K z7C=t;C9!PawAe##q9UvJb3i?jA0&9GsUNqZJKUICgl_ndfQR@$+gopc{?xL9*^Mwv zR}p`8#~(>DochteMZ5ogm;~_=dG7_62Z(I%7Zuv4y(dNUgrmxW-`B*rZ<6o`q*B<>5fr0&)@}0p!@BQyh z-~wF#hOx=kE8XQ*UT&NI&|(=G zx=QGfAmK?S_eS&6<(=C6@pd+R@vX@!X}7W~V5_cXF2SLEpK`Zjk^3iK7WwDK;2KFb z=(#g!Y%b;>jUUuBuJMD@*R`=8d2`H z=b2v)@p}usCE@bZKkx}_m6?-s4bbQz&C}rn>i_D|$A%6=4VsW$N+ka|4kW^>Y ze{LOR!FGXDKn6lE3t5Nnn{372Umnk1lP;`Jl6Jl==`fK2ga<=6M z68BA*kcZx5!0uBMGklWg$5R@C?=8PKWJcRV)&QSLi5XOsaeZfc5Q4__19;6EfR$J` z0qj?K=`_I@;IT$ij0L@C(i~LSsEz_o>4^80W!Vo8m^Q=l; z15hcszqkW#fOI2bhmKcEQ)>HRH<$+a=)M3yP*VP0(gCkN!Ulb!xfry!*)HIs-il*L z{byaG;^h-soTD+b$b-xa)Gx@=B3P9kS&|Vc9eaxss(raZ-u^YBlMc0ee7)Wmx_I z!TY-m@Bv!)F|CZ-k)M3uDRVrat^(aHrw4y-)7k0n>WezcP5{jBjvtR43;b^w-*kx5 zRSuI#m3mDO$siazy#V_Pd8vhJAO@om6vZDf15|t6}89hz|zqKK3Iea|FIp z14moloN~TA?@^^Di>xeQCj`Jd0NDeH6{Qc$$^>~F%a6<-rWHBq1$$WqA8%&KC+x59 zvjq?XnC58bk&o$-OSseg8c^fBwFHZP8v&RBd_4jRR2F!;)ri4&=+IrSEr627RS9-l zllJ(i#h&y5;}KPV>Ih=R!MFw5|4;+YD@VA1mAidF8sN@Q_U1V&p28g%NDLD2!8*qL zKn1MQ0!rW?iM(PwL=Bserx6ut{+zeD7gwXNqyCyS;7{vtG=PZw0ABpBB85pReSdf! zQ<3ymnz{z)dVhO!6x0X2tuN?({WL=K;j%pg;_w&7`B4dG=WL}XC13aky$oUoySJx8 z`yoCTmge7q1j@N~%<4#dTIS5aY1T)EIEQmUNc}lk75$=eZwgQ*z^CZHlWO)6hFY(p z#2`4p_BBuu8&C`xw2IC7oNtllV!~)hI zxR2}nQ*Qc3xIOg?FM3&g3<@se=;P1;53h;8f&Pr{wwk)>Q=V>M= z;$qnaO@Us^Sa(~5zxe~-h)+*{k}5JfLCU#FuG?zKC!&}NDXs)Q5J6y?2I3ARYHOsU{F5&_LueB&PF~9X`(=FY`QGI-SlAd c1+dGfXQrO^L^%)JTvUdeWRYJ9_6i~U|9RA9fB*mh literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/fcs8.zst b/test/fixtures/corpus/fcs8.zst new file mode 100644 index 0000000000000000000000000000000000000000..0b1067092773c1dc2f3bc94017abadda686dd95c GIT binary patch literal 44 wcmdPcs{i+afI0&hJY`^5P?C|VP*|FooUM>llwY2wkd|Mez{v3F?8nKR06aVoMgRZ+ literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/fcs_2p40.zst b/test/fixtures/corpus/fcs_2p40.zst new file mode 100644 index 0000000000000000000000000000000000000000..ec03b055f1321e86062de98f0863dfd817ea4591 GIT binary patch literal 27 gcmdPcs{i)^0|+oOFhnviWTfWg>hpa1{> literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/fcs_max_u64.zst b/test/fixtures/corpus/fcs_max_u64.zst new file mode 100644 index 0000000000000000000000000000000000000000..6f90fe371214574d61cff9f48158e69c16ab28e5 GIT binary patch literal 27 dcmdPcs{i-Fe<+A#V8}?#$;nqJ&o9bJ0RZ`K50?M{ literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/huf_weight_overrun.zst b/test/fixtures/corpus/huf_weight_overrun.zst new file mode 100644 index 0000000000000000000000000000000000000000..ac3797da43d16e4f4bdcb8b61b6c9d7a1a915a43 GIT binary patch literal 53 ecmdPcs{dDkE0vL95d)Wsfc+{$fQgZjkpTddXBnsf literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/manifest.json b/test/fixtures/corpus/manifest.json new file mode 100644 index 0000000..a9c8195 --- /dev/null +++ b/test/fixtures/corpus/manifest.json @@ -0,0 +1,157 @@ +[ + { + "name": "twoframes", + "frame": "twoframes.zst", + "command": "python3: pyzstd.compress(A, 3) + pyzstd.compress(B, 3), A = 5000 xorshift64 bytes, B = b'the quick brown fox ' * 500", + "verdict": "legal", + "plain": [ + "A.raw.gz", + "B.raw.gz" + ], + "size": 15000, + "trailing": "a second complete frame" + }, + { + "name": "checksum_good", + "frame": "checksum_good.zst", + "command": "zstd -q -c -3 --check A.bin", + "verdict": "legal", + "plain": [ + "A.raw.gz" + ], + "size": 5000 + }, + { + "name": "checksum_flipped", + "frame": "checksum_good.zst", + "command": "zstd -q -c -3 --check A.bin, then the last checksum byte flipped", + "verdict": "refused", + "why": "pyzstd refuses it: ZstdError", + "derive": "the last checksum byte flipped" + }, + { + "name": "checksum_zeroed", + "frame": "checksum_good.zst", + "command": "zstd -q -c -3 --check A.bin, then all four checksum bytes zeroed", + "verdict": "refused", + "why": "pyzstd refuses it: ZstdError", + "derive": "all four checksum bytes zeroed" + }, + { + "name": "checksum_truncated", + "frame": "checksum_good.zst", + "command": "zstd -q -c -3 --check A.bin, then the last two bytes cut", + "verdict": "refused", + "why": "pyzstd refuses it: ZstdError", + "derive": "the last two bytes cut" + }, + { + "name": "fcs_max_u64", + "frame": "fcs_max_u64.zst", + "command": "python3: pyzstd.compress(b'hello world', 3) with the descriptor rewritten to Frame_Content_Size_flag=3 declaring 18446744073709551615", + "verdict": "refused", + "why": "a 27-byte frame asking for a size no allocator will give; pyzstd refuses it: ZstdError" + }, + { + "name": "fcs_2p40", + "frame": "fcs_2p40.zst", + "command": "python3: pyzstd.compress(b'hello world', 3) with the descriptor rewritten to Frame_Content_Size_flag=3 declaring 1099511627776", + "verdict": "refused", + "why": "a 27-byte frame asking for a size no allocator will give; pyzstd refuses it: MemoryError" + }, + { + "name": "fcs8", + "frame": "fcs8.zst", + "command": "python3: pyzstd.compress(B, 3) with the descriptor rewritten to Frame_Content_Size_flag=3 and an 8-byte size", + "verdict": "legal", + "plain": [ + "B.raw.gz" + ], + "size": 10000 + }, + { + "name": "huf_weight_overrun", + "frame": "huf_weight_overrun.zst", + "command": "hand-built: a compressed block whose Huffman weight description is [0x10, 0x3F] and whose FSE weight stream is 33 x 0xAA then 0x02", + "verdict": "refused", + "why": "the weight stream decodes past the 256th weight; pyzstd refuses it: ZstdError" + }, + { + "name": "rlelit_fmt0_R31", + "frame": "rlelit_fmt0_R31.zst", + "command": "hand-built: one compressed block, RLE literals Size_Format=0, Regenerated_Size=31, byte 0x5A, Number_of_Sequences=0", + "verdict": "legal", + "plain": [ + "rlelit_fmt0_R31.raw.gz" + ], + "size": 31 + }, + { + "name": "rlelit_fmt1_R4000", + "frame": "rlelit_fmt1_R4000.zst", + "command": "hand-built: one compressed block, RLE literals Size_Format=1, Regenerated_Size=4000, byte 0x5A, Number_of_Sequences=0", + "verdict": "legal", + "plain": [ + "rlelit_fmt1_R4000.raw.gz" + ], + "size": 4000 + }, + { + "name": "rlelit_fmt2_R70000", + "frame": "rlelit_fmt2_R70000.zst", + "command": "hand-built: one compressed block, RLE literals Size_Format=2, Regenerated_Size=70000, byte 0x5A, Number_of_Sequences=0", + "verdict": "legal", + "plain": [ + "rlelit_fmt2_R70000.raw.gz" + ], + "size": 70000 + }, + { + "name": "rlelit_fmt2_R131072", + "frame": "rlelit_fmt2_R131072.zst", + "command": "hand-built: one compressed block, RLE literals Size_Format=2, Regenerated_Size=131072, byte 0x5A, Number_of_Sequences=0", + "verdict": "legal", + "plain": [ + "rlelit_fmt2_R131072.raw.gz" + ], + "size": 131072 + }, + { + "name": "mixed_rle_raw", + "frame": "mixed_rle_raw.zst", + "command": "hand-built: [RLE 5000 x 0xAB][raw 33 bytes][RLE 40000 x 0x07] in one frame", + "verdict": "legal", + "plain": [ + "mixed_rle_raw.raw.gz" + ], + "size": 45033 + }, + { + "name": "nofcs_windowed", + "frame": "nofcs_windowed.zst", + "command": "zstd -3 --no-check from stdin over b'the quick brown fox jumps over the lazy dog. ' * 24000", + "verdict": "legal", + "plain": [ + "text.raw.gz" + ], + "size": 1080000, + "expectedSize": 1080000 + }, + { + "name": "direct_nibble", + "frame": "direct_nibble.zst", + "command": "block 183 of path/to/srtm1z/N00E042.hgtz, lifted out as its own frame; its literals carry the direct-nibble Huffman weight form", + "verdict": "legal", + "plain": [ + "direct_nibble.raw.gz" + ], + "size": 131072 + }, + { + "name": "dictflag", + "frame": "dictflag.zst", + "command": "python3: pyzstd.compress(A, 3) with Dictionary_ID_flag forced to 1 in the frame header descriptor", + "verdict": "refused", + "why": "a dictionary this decoder does not have; pyzstd refuses it: ZstdError" + } +] diff --git a/test/fixtures/corpus/mixed_rle_raw.raw.gz b/test/fixtures/corpus/mixed_rle_raw.raw.gz new file mode 100644 index 0000000000000000000000000000000000000000..a437cad8c8372a4b119118947e7bda8040a0fd63 GIT binary patch literal 100 zcmb2|=3oE;rvGn`E#zcS5MVg)LW*yBLWO`1-&gNjK4G_Gclm%6JlK9wIqhz->+193 SATD9>z)Gk)<@C$-3=9BUV;5-v literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/mixed_rle_raw.zst b/test/fixtures/corpus/mixed_rle_raw.zst new file mode 100644 index 0000000000000000000000000000000000000000..c6b7058638aa4494aee358717a14813f717190c8 GIT binary patch literal 51 ocmdPcs{c3P9XV$wpEq(z%0|NlvPYV_R literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/rlelit_fmt1_R4000.zst b/test/fixtures/corpus/rlelit_fmt1_R4000.zst new file mode 100644 index 0000000000000000000000000000000000000000..0fcb01330df6c9efde6cd1374fa26e25fbc8ad43 GIT binary patch literal 14 VcmdPcs{c1(0iP-Z1M9CS1^_1u1eO2* literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/rlelit_fmt2_R131072.raw.gz b/test/fixtures/corpus/rlelit_fmt2_R131072.raw.gz new file mode 100644 index 0000000000000000000000000000000000000000..443bbfd9f4510d13756fd1bb5cf2e89e82b94cb1 GIT binary patch literal 163 scmb2|=3oE;rvGmb8ZrVIhZf9xEq~;&lmmz}fZz_pzRMFX03F5v00bHgPXGV_ literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/rlelit_fmt2_R131072.zst b/test/fixtures/corpus/rlelit_fmt2_R131072.zst new file mode 100644 index 0000000000000000000000000000000000000000..8ccc14fe38862a0dcf9204c7740eced84bc7d755 GIT binary patch literal 17 YcmdPcs{eNZ0|OI-E&~HEgF+Mo04b3JU;qFB literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/rlelit_fmt2_R70000.raw.gz b/test/fixtures/corpus/rlelit_fmt2_R70000.raw.gz new file mode 100644 index 0000000000000000000000000000000000000000..2fd8780c1b1c6d1869809b27518add20464809b4 GIT binary patch literal 104 scmb2|=3oE;rvGmb8ZrVIhZf9xEq_F4o&$(ODadEe*C>BeAjrr70QN@z>% literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/rlelit_fmt2_R70000.zst b/test/fixtures/corpus/rlelit_fmt2_R70000.zst new file mode 100644 index 0000000000000000000000000000000000000000..d27068ecd28ad3953cdacdbdfb32b1b172d95b52 GIT binary patch literal 17 YcmdPcs{eOEfgmG;E&~IvxL_0m057ov=l}o! literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/text.raw.gz b/test/fixtures/corpus/text.raw.gz new file mode 100644 index 0000000000000000000000000000000000000000..9ca005468670ad6314e32fec132797382e4eb647 GIT binary patch literal 3221 zcmb2|=3oE;rvGnG-4<*z5OFvdmdE;Ki2zIBBE^I5U#+G+eDi$f=2EZ1{NnvbHa)vwXf67xa0@i2qHUb=K5wY>Sg#I*NjoFP}N9h&vd=9NR_UD)Lp%II^UI!9@5cL8__|Qz zQouotwNv~?r<}&cW%w2wIVnl8a&NUvE?6A<*8uFe`yi*Oa8jZa+DdvmNGXs% z<1%}ON7pays`0+Sj&9*}k7j_b3pYFSS&+JKbA7~N(gmZ%W3*n<*=C?UTm$tx3MMke zw(|>-fkFSo_+u4K%d*_vIH>%(@L#re$SqpXB;+G3v!zxjU#4td=yyIm()9}%=ca?d z^I;->-_SeC;>fk_0!Gj=FxB}kb6A_wk>A>kZEuIE)z?}&@8Z5Ebrj28Cg<-kCE&A*R)j`ID?dsuJg8w5&Dp+0T$!<`Y{^P|Q z0nBt6Zq&>Sx9HQ%uzvik)B939`*e+nlx1sz&mE+UhQx#BIkKR6F-G~!D~lg)QE7#U zB&#p?-;?Jm@1?C(a7Y7L&PJ6(sfs zh4U6Fg~Bn5KkW?PG+3->0d&D&0EvjS^_7fD4l#e~O>> zumz*M+LSK!Hp_*D=6aO8G`_yrmD6~+W<31|qay`Hp;vxMEmR7s*t}+=Saf@rBL#J3*QE9Sgg8##AwU<(^{74bZA_2p?gR}eU15QbL=~5<(5KLgkDg#_>91DSp!8Eg>|5koFmAnrpCWP9to@7g?Fd4H0v1xVKXG zJHJ|ZOSq()s<^XGN`7Ib98hrKkDx&X2736fYK`UsN7PIv)8T@4*BSBL8V`St8cG8` zvYkI8i0^d4F9@OA%t-#mFgvaqpCbulX#L9S@gxm`(K1A+^U51;k13e^&=xm82}ZqW z#3k1!I=`Oy-!4PptIwKRNogt3b{D`VNgE@@xB_CbWPp$30#J_*QO*M=H)FtU$=X6G zx=hKTptk~1e1Cc%Oy2!b;2ZaCP^CG@M?sx%QzUqjyGkoM1j1<(8S?goH|e50p6^F^ zwIPGTEN?{?7?!)WW028@Xn!AwA7*HIcXw6lWWc4Fe0Bh-$_w+Tm=hf9Q_>@Em(p%^ z55KSUY{2y-lhLZxlE<}VFbHq_dK;cnmwFfXW2`He+NhwGC@c3;Ax%TUy zZJV;KE(dL!8J8@P4v@^9{@OAT+p2_S`TQe5dFz5u^A_7#uB8|(Mm9Fgde+YjjO4+< z8zCk)m5W+Gau1FwH9uCVZzyN;O5qY^zQ$*Pr$XZk)3e+WvKVK@^;PlgROIVqN9lLQucKiU`ysXvqUddcs?9^=+Xm zW&FHR-2>$ufQKXPkNn5niSQ6F2WtPFA#k?+J~y!; z)G_*-f5|Wy6516TC`xf|*m}8>I%T|=jr&sJIQx!uUez=sRhl!FwZ&?9(!#MvYiAv!kvV#P`IpIA`L>ps$z-<=&HjS&T3*tDlu2_^i#<^sywUrLoFG$3(+-)q zI(e|6oZlpIJ~g0b(O2#QdQE6&^Bq2j6(y0f<6ro_PTsu@HSb|W7w zV6KgpKrXbqxgXP+RJUXvDutZ^Mys+(Nns(=nXW-yh~YVNJzT|`49%*GbIo198A_wk z?wAS4rjfiEe;e6V&(8cAXu!D;cl0#+SuQe-PbErnfZ8|Lc}z%8 zQ?(d&S0|;q%)EGE4htC!#$(&AT;->+GgCr$lqo^EHBq8oYK;g7r9 z3goO={W|wyFnu|ql$qj_08IU~ z^9seA&;B;FFr5sVdG_FtDRXem_8~Ow!u`z2f+Tc+sL{PsJS9wcOMe5FiULt4cFN6Z zSEgEML&66c1qA!*>cmYCXzOwM_;=!^QGOVfPdeSY5EEt`7{l`*yM;k(wT9nms>|Ya zfnKjUnf*Wp>DA_U^xr@@Wx0ALWi~R`+;&MzYJ#3U4Bkpc3680ui3CTyANiAYL!X!- zYXHZGRBk*LwtySJCiX$~k#Wf4^1ql}vSnzEQ>Q>#- zg1`KDlWf6fzUD-Wj+tT?fC1Z-#>RjGJLwANUW!qn;%Np=_+F{{rPjOgdzUE;SYwId zcjX4rk*r^gKd=9D9f*A9e7CsTRLZx?Z$ByVkg@B8SVuf&LODzkU(AKo5nCw|q#k6NJiU%3%yy`jN^Gnp9@s}{_)m{k&L znWmi)l8sLNznx_E;cOequFaq@y@w2O5<(C0PY_>2Gi{&hb;v~*Bq($!`aNk%)V>F= zx_`=++aO}d4<@J}E^f~tQ^dZ;n?yKzH(F>}vP&@@1J%NDNwd2KwDPB2%$d@#K2k^J zzjMK6cU0ci!};bGm5;?ZC*~M~&u@S%y7kj#7o~e<$N~SiWs&B*P~CY~SI=%&QlF)s zRf4^mOBbLV*6DPzTB|~GF7*64cGt4KYo*7FP4sQ3Sl!X)oYF>QiY~3^FZ|9)9!ln+ zVK*}T-X)gNY?@TQu*`B4^X-L9O`UZdpH=_*;?PU-48x?^mKD^BMp43Q$VblDEuplO6871~q zIz&16ozKpQb6H}4Po;?8hzC^j{?11K@s97vRL_ov$YX@Z=~yv8#t@%zH2TarPm;w1 zZ@5L{B%Du)vM40PAPjd`JQu&sfq~W$

&p7v+?<5E1AN3XR^z`_|ve+vKs{?(!75 zeZvnFAy?NOFXKo7%UShSuPaVC$8II-zUd5FWo+?X^T7tfd__gl8&*vP+60l+?33t zz5>RBOd|!k6sE$}c(9&QEfTj6@UtKPh|it~1{Q;(3N0{pa&Z7aYC3zIms+Wg7*0E1 zLd5Jv8wRn9QNRRa6m{j)*0ctf#2xH*6!f1JvdbA-fPOAcgsR7k;GG#riafF-$ojyo5R=@8eWF4X z8^1#A8qaE*cw&A1xMndNf4kAVYznnGDRf(Gk?dFGvq;I0n#2g}v*f+L!RRi%bNVa|@tqvlEp*({yyEG;aAb&qV_EiKJ^q*GZ#hEvw7+ciXGU z7We9!5W@W7Veh41$LrFFqJBARVF`B6>-+(-bOc$>5T;uZ5nOuJ$ojKZ{n@?pvf3xS z$u~HS>pV&2x0q+xEU4lO4fkVPjXoPLk?`A)NiV&hQ^hPx>DM=1A5cmXmTXJpsC;an zGq>+xWV-Qc12LV`A8JN~T#0!+I=aWmb@_0wOLYXq-2h5XRS2W7(EUk8W&_%VGxgA> zsr16|Ft>%_u4aPb!Yyii{W_OPaYYh)Y#TWdUc+q})wh+CnUL|+wOoZQWEPEYMdlt7 zs!hT5XhQ?kyyc-sLIG zgelf}e~f2P?oqHgb#DRW-xtc*NQ;}Sg0I{fxqBKsKQTlD;Qv)F>B%nEKZ*%c64uLK zruNt?y3Rz{UcMaPaS=eN?|MlzFXSf^kI&!)Aq|QS>6Ea~e%-_WB?xGgu48i^&L%;` zCBIE9L7-1RM9&*eh@b4Ju>^RZyZ4#Z4{m4>-GNTPb8u@9TtYiUT`|18xsx&}f0aoA zFmeFmC0n|Q=B^N#0OB9Hi2VD-FtXZ9;jI(I31crnd>)sItcIzA z9<=@Y1^UHbM-a>wH@voQZ7RALj7A-mVzR$Bd|T>bG1e3A6ku%b0sIG`oB0R?hcFh6 zB~}e9a|r(zxd%}A1Ex!~==AYA1>{)OL6zl_WUxLxPprbs&Y;3Sd&5J-LteWzvz0CK z;%!pXo#XM!512QK-Fenwj5zu;d&*WBPze9^@qx@)XSuMQAIFmgl_gQ_mvzpBrz(j( z(gMZinD<<4ov176zMQHF9_$778Y5dwO9XW8fF0sq`8!Ed4=Vz~`_rk4H!Q)X7puS2 zw^lCz5j@rDFByoceqCLl9F> 7 + state ^= (state << 17) & MASK64 + out[i] = (state >> 33) & 0xFF + return bytes(out) + + +A = xorshift(0x9E3779B97F4A7C15, 5000) +B = b"the quick brown fox " * 500 +TEXT = b"the quick brown fox jumps over the lazy dog. " * 24000 + + +def build(): + OUT.mkdir(parents=True, exist_ok=True) + for stale in OUT.glob("*"): + stale.unlink() + + a = plain("A", A) + b = plain("B", B) + + # 1. Two frames in one buffer. pyzstd returns A+B; a decoder that reads + # one frame must refuse rather than return A and stop. + legal( + "twoframes", + pyzstd.compress(A, 3) + pyzstd.compress(B, 3), + [a, b], + "python3: pyzstd.compress(A, 3) + pyzstd.compress(B, 3), " + "A = 5000 xorshift64 bytes, B = b'the quick brown fox ' * 500", + ) + cases[-1]["trailing"] = "a second complete frame" + + # 2. A frame with a content checksum. The three malformed variants are + # derived from this one in the test; pyzstd's verdict on each is + # recorded here so nothing is asserted on faith. + checked = cli(["-3", "--check"], A, "A.bin") + legal("checksum_good", checked, [a], "zstd -q -c -3 --check A.bin") + for name, label, variant in ( + ( + "checksum_flipped", + "the last checksum byte flipped", + checked[:-1] + bytes([checked[-1] ^ 0x01]), + ), + ( + "checksum_zeroed", + "all four checksum bytes zeroed", + checked[:-4] + b"\0\0\0\0", + ), + ( + "checksum_truncated", + "the last two bytes cut", + checked[:-2], + ), + ): + try: + pyzstd.decompress(variant) + except Exception as error: + verdict = f"pyzstd refuses it: {type(error).__name__}" + else: + verdict = "NOTE pyzstd accepts it" + cases.append( + { + "name": name, + "frame": "checksum_good.zst", + "command": f"zstd -q -c -3 --check A.bin, then {label}", + "verdict": "refused", + "why": verdict, + "derive": label, + } + ) + + # 3. Frame_Content_Size_flag = 3 declaring a size no machine will give. + # Built by rewriting a real frame's descriptor, so everything after the + # header is a frame libzstd wrote. + base = pyzstd.compress(b"hello world", 3) + assert base[4] >> 6 == 0 and (base[4] >> 5) & 1 == 1, base[4] + for label, declared in ( + ("fcs_max_u64", (1 << 64) - 1), + ("fcs_2p40", 1 << 40), + ): + frame = base[:4] + bytes([(3 << 6) | (base[4] & 0x3F)]) + frame += declared.to_bytes(8, "little") + base[6:] + illegal( + label, + frame, + f"python3: pyzstd.compress(b'hello world', 3) with the descriptor " + f"rewritten to Frame_Content_Size_flag=3 declaring {declared}", + "a 27-byte frame asking for a size no allocator will give", + ) + + # 3b. The same eight-byte content size on a frame that is legal, so the + # branch is reached by something that decodes and not only by two + # frames refused on the way past it. + legal( + "fcs8", + rewrite_fcs8(pyzstd.compress(B, 3), len(B)), + [b], + "python3: pyzstd.compress(B, 3) with the descriptor rewritten to " + "Frame_Content_Size_flag=3 and an 8-byte size", + ) + + # 4. The 53-byte frame that walked fseDecodeInterleaved past _weights[255]. + # Hand-built: a compressed block whose literals carry an FSE-coded + # Huffman weight stream long enough to emit 257 weights. + weights = bytes([0x10, 0x3F]) + bytes([0xAA] * 33) + bytes([0x02]) + body = bytes([0xA2, 0x00, 0x0A]) # compressed literals, 10 regenerated + body += bytes([len(weights)]) + weights + body += bytes([0x01, 0x01, 0x01, 0x00]) # sequence section, never reached + frame = MAGIC + bytes([0x20, 0x0A]) + blockhdr(1, 2, len(body)) + body + illegal( + "huf_weight_overrun", + frame, + "hand-built: a compressed block whose Huffman weight description is " + "[0x10, 0x3F] and whose FSE weight stream is 33 x 0xAA then 0x02", + "the weight stream decodes past the 256th weight", + ) + + # 5. RLE literals with Number_of_Sequences = 0, at every size format. + # libzstd never emits these: it writes an RLE block instead. + for fmt, count in ((0, 31), (1, 4000), (2, 70000), (2, 131072)): + body = rle_literals(count, 0x5A, fmt) + name = f"rlelit_fmt{fmt}_R{count}" + expect = plain(name, bytes([0x5A]) * count) + legal( + name, + single_frame([blockhdr(1, 2, len(body)) + body], count), + [expect], + f"hand-built: one compressed block, RLE literals Size_Format={fmt}, " + f"Regenerated_Size={count}, byte 0x5A, Number_of_Sequences=0", + ) + + # 6. RLE and raw blocks, which the real .hgtz corpus contains none of. + content = b"\xab" * 5000 + b"hello world hello world hello wor" + b"\x07" * 40000 + assert len(content) == 5000 + 33 + 40000 + blocks = [ + blockhdr(0, 1, 5000) + b"\xab", + blockhdr(0, 0, 33) + content[5000:5033], + blockhdr(1, 1, 40000) + b"\x07", + ] + expect = plain("mixed_rle_raw", content) + legal( + "mixed_rle_raw", + single_frame(blocks, len(content)), + [expect], + "hand-built: [RLE 5000 x 0xAB][raw 33 bytes][RLE 40000 x 0x07] " + "in one frame", + ) + + # 7. No Frame_Content_Size at all: the shape the CLI writes from a pipe. + frame = cli(["-3", "--no-check"], TEXT) + assert frame[4] >> 6 == 0 and (frame[4] >> 5) & 1 == 0, frame[4] + expect = plain("text", TEXT) + legal( + "nofcs_windowed", + frame, + [expect], + "zstd -3 --no-check from stdin over " + "b'the quick brown fox jumps over the lazy dog. ' * 24000", + expected_size=len(TEXT), + ) + + # 11. A real block that codes its Huffman weights as direct nibbles. The + # .hgtz corpus has 249 of these and the committed container has none. + tile = TILE.read_bytes() + offset, length = struct.unpack_from(">II", tile, 12 + 8 * TILE_BLOCK) + frame = bytes(tile[offset : offset + length]) + assert direct_nibble_weights(frame), "that block does not use direct nibbles" + expect = plain("direct_nibble", pyzstd.decompress(frame)) + legal( + "direct_nibble", + frame, + [expect], + f"block {TILE_BLOCK} of {TILE}, lifted out as its own frame; its " + "literals carry the direct-nibble Huffman weight form", + ) + + # 13. Dictionary_ID_flag set on a frame that carries no dictionary id. + base = pyzstd.compress(A, 3) + illegal( + "dictflag", + base[:4] + bytes([base[4] | 1]) + base[5:], + "python3: pyzstd.compress(A, 3) with Dictionary_ID_flag forced to 1 " + "in the frame header descriptor", + "a dictionary this decoder does not have", + ) + + (OUT / "manifest.json").write_text(json.dumps(cases, indent=2) + "\n") + total = sum(f.stat().st_size for f in OUT.iterdir()) + print(f"{len(cases)} cases, {len(list(OUT.iterdir()))} files, {total} bytes") + + +def rewrite_fcs8(frame, size): + """The same frame with an 8-byte Frame_Content_Size in place of its own.""" + descriptor = frame[4] + single = (descriptor >> 5) & 1 + at = 5 + (0 if single else 1) + width = {0: 1 if single else 0, 1: 2, 2: 4, 3: 8}[descriptor >> 6] + return ( + frame[:4] + + bytes([(3 << 6) | (descriptor & 0x3F)]) + + frame[5:at] + + size.to_bytes(8, "little") + + frame[at + width :] + ) + + +def blockhdr(last, btype, size): + return struct.pack("> 4, byte]) + else: + assert count < (1 << 20) + body = bytes( + [ + ((count & 0xF) << 4) | (3 << 2) | 1, + (count >> 4) & 0xFF, + count >> 12, + byte, + ] + ) + return body + b"\x00" + + +def single_frame(blocks, size): + """A single-segment frame declaring `size` and carrying `blocks`.""" + if size < 256: + descriptor, field = (0 << 6) | (1 << 5), bytes([size]) + elif size < 65536 + 256: + descriptor, field = (1 << 6) | (1 << 5), struct.pack("> 5) & 1 else 1) + at += {0: 1 if (descriptor >> 5) & 1 else 0, 1: 2, 2: 4, 3: 8}[descriptor >> 6] + header = frame[at] | (frame[at + 1] << 8) | (frame[at + 2] << 16) + at += 3 + if (header >> 1) & 3 != 2: + return False + literals = frame[at] + if literals & 3 != 2: + return False + at += {0: 3, 1: 3, 2: 4, 3: 5}[(literals >> 2) & 3] + return frame[at] >= 128 + + +if __name__ == "__main__": + sys.exit(build()) From b9553d9566483f2de2fbee24c20e4b9455bebe25 Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:20:51 -0600 Subject: [PATCH 12/18] Decode the corpus against pyzstd, never against ourselves A test that asserts what the decoder currently does proves nothing. Every legal frame here is compared byte for byte with what pyzstd decompressed it to, and every malformed frame is one pyzstd also refuses. The refusal has to be a ZstdException. RangeError, OutOfMemoryError and IndexError are all Error rather than Exception, so a caller writing "on ZstdException" does not catch them and the isolate dies. --- test/corpus.dart | 209 ++++++++++++++++++++++++++++++++++++++++++ test/corpus_test.dart | 92 +++++++++++++++++++ 2 files changed, 301 insertions(+) create mode 100644 test/corpus.dart create mode 100644 test/corpus_test.dart diff --git a/test/corpus.dart b/test/corpus.dart new file mode 100644 index 0000000..4d125fa --- /dev/null +++ b/test/corpus.dart @@ -0,0 +1,209 @@ +import 'dart:convert'; +import 'dart:io'; +import 'dart:typed_data'; + +/// Loading and generating the test corpus. +/// +/// The committed half lives in `test/fixtures/corpus`, built by +/// `tool/make_corpus.py`. Every frame there was either written by libzstd or +/// hand-built and then accepted by pyzstd, and the expected plain bytes of a +/// legal frame are what pyzstd decompressed it to, gzipped. `manifest.json` +/// records the exact command behind each one, so a fixture is never a blob +/// nobody can reproduce. +/// +/// The half that is too large to commit is generated here instead, from the +/// seeded generator below and the `zstd` CLI. +const String corpusDirectory = 'test/fixtures/corpus'; + +/// One entry of `manifest.json`. +class CorpusCase { + CorpusCase(Map json) + : name = json['name']! as String, + command = json['command']! as String, + frameFile = json['frame']! as String, + legal = json['verdict'] == 'legal', + why = json['why'] as String?, + derive = json['derive'] as String?, + size = json['size'] as int?, + expectedSize = json['expectedSize'] as int?, + trailing = json['trailing'] as String?, + _plain = (json['plain'] as List? ?? const []) + .cast() + .toList(); + + /// The case name, which is also the `test()` description. + final String name; + + /// The exact command that produced the frame. + final String command; + + final String frameFile; + + /// True when libzstd decompresses the frame, false when it refuses it. + final bool legal; + + /// For a refused case, what is wrong with it and what pyzstd said. + final String? why; + + /// For a refused case built by damaging another case's frame, the damage. + final String? derive; + + /// Length of the plain bytes, for a legal case. + final int? size; + + /// What the caller must pass to `decode`, for a frame declaring no content + /// size of its own. + final int? expectedSize; + + /// What follows the first frame, for a buffer libzstd reads as a stream of + /// frames and this decoder reads as one frame and then leftovers. + final String? trailing; + + final List _plain; + + /// The frame, with any recorded damage applied. + Uint8List get frame { + final bytes = Uint8List.fromList( + File('$corpusDirectory/$frameFile').readAsBytesSync(), + ); + return switch (derive) { + null => bytes, + 'the last checksum byte flipped' => Uint8List.fromList( + bytes, + )..[bytes.length - 1] ^= 0x01, + 'all four checksum bytes zeroed' => Uint8List.fromList( + bytes, + )..fillRange(bytes.length - 4, bytes.length, 0), + 'the last two bytes cut' => Uint8List.sublistView( + bytes, + 0, + bytes.length - 2, + ), + _ => throw StateError('unknown damage: $derive'), + }; + } + + /// What pyzstd decompressed the frame to. Only for a legal case. + Uint8List get plain { + final out = BytesBuilder(copy: false); + for (final part in _plain) { + out.add(gzip.decode(File('$corpusDirectory/$part').readAsBytesSync())); + } + return out.takeBytes(); + } + + @override + String toString() => '$name ($command)'; +} + +/// Every case in `manifest.json`, in the order the generator wrote them. +List loadCorpus() { + final text = File('$corpusDirectory/manifest.json').readAsStringSync(); + return (jsonDecode(text) as List) + .cast>() + .map(CorpusCase.new) + .toList(); +} + +/// xorshift64, the same three shifts `tool/make_corpus.py` uses, so an input +/// built here and one built there are the same bytes. +class Xorshift64 { + Xorshift64(this._state); + + int _state; + + int next() { + var state = _state; + state ^= state << 13; + state ^= state >>> 7; + state ^= state << 17; + _state = state; + return state; + } + + Uint8List bytes(int count) { + final out = Uint8List(count); + for (var i = 0; i < count; i++) { + out[i] = (next() >>> 33) & 0xFF; + } + return out; + } +} + +/// `count` bytes of pseudo-random noise. Incompressible, so a compressor has +/// to fall back on raw blocks and raw literals. +Uint8List noise(int count, {int seed = 0x9E3779B97F4A7C15}) => + Xorshift64(seed).bytes(count); + +/// A stream built by drawing `count` tokens of `tokenLength` bytes from a +/// pool of `tokens` of them. +/// +/// Short tokens from a small pool are what puts tens of thousands of +/// sequences in one block: every token is a match, and at three bytes each +/// there are more of them than the two-byte sequence count can hold. +Uint8List tokenStream({ + required int tokens, + required int tokenLength, + required int count, + int seed = 0x2545F4914F6CDD1D, +}) { + final random = Xorshift64(seed); + final pool = random.bytes(tokens * tokenLength); + final out = Uint8List(count * tokenLength); + for (var i = 0; i < count; i++) { + final token = (random.next() >>> 16) % tokens; + out.setRange( + i * tokenLength, + (i + 1) * tokenLength, + pool, + token * tokenLength, + ); + } + return out; +} + +/// `count` bytes drawn from a lopsided alphabet: `a` eight times as likely as +/// the rare letters, and no matches to speak of. +/// +/// This is the shape that makes libzstd build one Huffman table and then +/// reuse it for the blocks that follow, which is the treeless literals form. +Uint8List skewedAlphabet(int count, {int seed = 0x853C49E6748FEA9B}) { + const alphabet = 'aaaaaaaabbbbccdefghijklmnopqrstuvwxyz'; + final random = Xorshift64(seed); + final out = Uint8List(count); + for (var i = 0; i < count; i++) { + out[i] = alphabet.codeUnitAt((random.next() >>> 16) % alphabet.length); + } + return out; +} + +/// True when the `zstd` CLI is on the path. +bool get hasZstdCli { + try { + return Process.runSync('zstd', const ['--version']).exitCode == 0; + } on ProcessException { + return false; + } +} + +/// Compresses [input] with the `zstd` CLI, as `zstd -q -c `. +/// +/// A file argument rather than a pipe, so the frame carries a content size. +Uint8List zstdCli(List args, Uint8List input) { + final directory = Directory.systemTemp.createTempSync('pure_zstd_corpus_'); + try { + final file = File('${directory.path}/input.bin')..writeAsBytesSync(input); + final run = Process.runSync('zstd', [ + '-q', + '-c', + ...args, + file.path, + ], stdoutEncoding: null); + if (run.exitCode != 0) { + throw StateError('zstd ${args.join(' ')} failed: ${run.stderr}'); + } + return Uint8List.fromList(run.stdout as List); + } finally { + directory.deleteSync(recursive: true); + } +} diff --git a/test/corpus_test.dart b/test/corpus_test.dart new file mode 100644 index 0000000..84847fb --- /dev/null +++ b/test/corpus_test.dart @@ -0,0 +1,92 @@ +import 'dart:typed_data'; + +import 'package:pure_zstd/pure_zstd.dart'; +import 'package:test/test.dart'; + +import 'corpus.dart'; + +/// The committed corpus, which exists because `hgtz_block_test.dart` decodes +/// one container from one producer at one level and so proves the sequence +/// loop and Huffman literals on level-19 terrain and nothing else. +/// +/// The frames here reach the parts of RFC 8878 that 16,490 real `.hgtz` +/// blocks never touch: raw blocks, RLE blocks, RLE literals, a frame with no +/// content size, a frame with a checksum, the eight-byte content size, and +/// the direct-nibble Huffman weight form. +/// +/// Nothing below asserts what this decoder happens to do. A legal frame is +/// compared against what pyzstd decompressed it to; a refused frame is one +/// pyzstd also refuses. `tool/make_corpus.py` records both, along with the +/// exact command behind each frame, in `test/fixtures/corpus/manifest.json`. +void main() { + final corpus = loadCorpus(); + + group('a legal frame decodes to the bytes pyzstd gives', () { + for (final entry in corpus.where( + (entry) => entry.legal && entry.trailing == null, + )) { + test('${entry.name}: ${entry.command}', () { + final out = ZstdDecoder().decode( + entry.frame, + expectedSize: entry.expectedSize, + ); + expect(out.length, entry.size); + expect(out, orderedEquals(entry.plain)); + }); + } + }); + + group('a malformed frame is refused, never crashed on', () { + for (final entry in corpus.where((entry) => !entry.legal)) { + // A `ZstdException` and nothing else. `RangeError`, `OutOfMemoryError` + // and `IndexError` are all `Error` rather than `Exception`, so a caller + // catching `ZstdException` does not catch them and the isolate dies. + test('${entry.name}: ${entry.why}', () { + expect( + () => ZstdDecoder().decode(entry.frame), + throwsA(isA()), + ); + }); + } + }); + + group('two frames in one buffer', () { + final entry = corpus.singleWhere((entry) => entry.name == 'twoframes'); + final first = 5000; + + test('${entry.command}: decodes to both or refuses, never to the first ' + 'alone', () { + Uint8List? out; + try { + out = ZstdDecoder().decode(entry.frame); + } on ZstdException { + return; // Refusing is the other acceptable answer. + } + expect( + out.length, + isNot(first), + reason: 'returned the first frame and stopped, raising nothing', + ); + expect(out, orderedEquals(entry.plain)); + }); + }); + + group('a frame declaring no content size', () { + final entry = corpus.singleWhere((entry) => entry.name == 'nofcs_windowed'); + final size = entry.expectedSize!; + + test('decodes when given the size it really has', () { + final out = ZstdDecoder().decode(entry.frame, expectedSize: size); + expect(out, orderedEquals(entry.plain)); + }); + + for (final wrong in [size - 1, size + 1, 0]) { + test('is refused when given $wrong instead of $size', () { + expect( + () => ZstdDecoder().decode(entry.frame, expectedSize: wrong), + throwsA(isA()), + ); + }); + } + }); +} From c4bc910c0559cc79d66f10f8c502c2678ebab473 Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:20:53 -0600 Subject: [PATCH 13/18] Run one decoder over mixed frames in both orders A ZstdDecoder keeps its Huffman table, its three FSE tables, the repeat flags and the three recent offsets between calls, and every one of those belongs to a single frame. A leak is invisible while every frame in the list looks alike, which is what the .hgtz corpus is. Reversing the order gives each frame a different predecessor, so a leak shows up as a difference between the passes. --- test/shared_decoder_test.dart | 74 +++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 test/shared_decoder_test.dart diff --git a/test/shared_decoder_test.dart b/test/shared_decoder_test.dart new file mode 100644 index 0000000..8b43605 --- /dev/null +++ b/test/shared_decoder_test.dart @@ -0,0 +1,74 @@ +import 'dart:typed_data'; + +import 'package:pure_zstd/pure_zstd.dart'; +import 'package:test/test.dart'; + +import 'corpus.dart'; + +/// One `ZstdDecoder` over a mixed list of frames, forward and then reversed. +/// +/// A decoder holds its Huffman table, its three FSE tables, the three "table +/// is ready to repeat" flags and the three recent offsets between calls, and +/// every one of those is per frame. A frame that leaves a table behind is +/// invisible while every frame in the list looks alike, which is exactly what +/// the real `.hgtz` corpus is. Reversing the order moves each frame to a +/// different predecessor, so a leak shows up as a difference between the two +/// passes or against what pyzstd gave. +void main() { + final mixed = loadCorpus() + .where((entry) => entry.legal && entry.trailing == null) + .toList(); + + test('decodes ${mixed.length} mixed frames the same in either order', () { + expect(mixed.length, greaterThan(5), reason: 'the list must be mixed'); + + final decoder = ZstdDecoder(); + final forward = { + for (final entry in mixed) + entry.name: decoder.decode( + entry.frame, + expectedSize: entry.expectedSize, + ), + }; + + final backward = {}; + for (final entry in mixed.reversed) { + backward[entry.name] = decoder.decode( + entry.frame, + expectedSize: entry.expectedSize, + ); + } + + for (final entry in mixed) { + expect( + forward[entry.name], + orderedEquals(entry.plain), + reason: '${entry.name} decoded wrongly in order', + ); + expect( + backward[entry.name], + orderedEquals(entry.plain), + reason: '${entry.name} decoded wrongly reversed', + ); + } + }); + + test('a refused frame does not poison the next one', () { + final decoder = ZstdDecoder(); + final bad = loadCorpus().where((entry) => !entry.legal).toList(); + for (final good in mixed) { + for (final entry in bad) { + try { + decoder.decode(entry.frame); + } on ZstdException { + // The point of the case. Some are open defects and decode fine. + } + } + expect( + decoder.decode(good.frame, expectedSize: good.expectedSize), + orderedEquals(good.plain), + reason: '${good.name} after a run of malformed frames', + ); + } + }); +} From d1159ca695d61f3cfde91690ef9b2e2c6a9e5300 Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:20:55 -0600 Subject: [PATCH 14/18] Check fseDecodeInterleaved has room for the third symbol It has no symbol count to stop on and runs until the bitstream is spent, so its own room check is the only thing keeping it inside the caller's buffer. Both exhausted branches write a third symbol in the same pass. This is the 53-byte reproducer with the frame stripped away, so a regression points at the function rather than at a block somewhere. --- test/fse_interleaved_test.dart | 55 ++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 test/fse_interleaved_test.dart diff --git a/test/fse_interleaved_test.dart b/test/fse_interleaved_test.dart new file mode 100644 index 0000000..8396929 --- /dev/null +++ b/test/fse_interleaved_test.dart @@ -0,0 +1,55 @@ +import 'dart:typed_data'; + +import 'package:pure_zstd/src/bit_reader.dart'; +import 'package:pure_zstd/src/exception.dart'; +import 'package:pure_zstd/src/fse.dart'; +import 'package:test/test.dart'; + +/// `fseDecodeInterleaved` has no symbol count to stop on: it runs until the +/// bitstream is spent, so the only thing keeping it inside the caller's +/// buffer is its own room check. Both of its exhausted branches write a third +/// symbol in the same pass, so a check for two let `at` reach `outLimit`, +/// which is one past the end. +/// +/// `HuffmanTable` calls it with a 256-byte buffer, so overrunning it indexes +/// `_weights[256]`. This is the same reproducer as the 53-byte frame in the +/// corpus, with the frame stripped away: the weight table description and the +/// bitstream the block carried, handed straight to the function. +void main() { + group('fseDecodeInterleaved', () { + // The two bytes are the whole FSE table description: accuracy log 5 and a + // distribution that gives the stream below two symbols per state pair. + final description = Uint8List.fromList([0x10, 0x3F]); + + // 33 bytes of 0xAA and then the end marker, which is the stream the + // crafted block put after the description. + final stream = Uint8List.fromList([...List.filled(33, 0xAA), 0x02]); + + test('refuses a stream longer than the caller has room for', () { + final table = FseTable(6); + expect(readFseTable(table, description, 0, description.length, 255), 2); + + final bits = ReverseBitReader()..reset(stream, 0, stream.length); + final weights = Uint8List(256); + + expect( + () => fseDecodeInterleaved(table, bits, weights, 0, weights.length), + throwsA(isA()), + ); + }); + + test('fills the buffer exactly when there is room for the last pass', () { + final table = FseTable(6); + readFseTable(table, description, 0, description.length, 255); + + // Same stream, a buffer with slack. The count it returns must be the + // symbols it actually wrote, so the last pass cannot have been longer + // than the room the check left. + final bits = ReverseBitReader()..reset(stream, 0, stream.length); + final room = Uint8List(1024); + final written = fseDecodeInterleaved(table, bits, room, 0, room.length); + expect(written, greaterThan(256)); + expect(written, lessThanOrEqualTo(room.length)); + }); + }); +} From ca57654babc10936cccd6681fccfe10b2c515429 Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:20:56 -0600 Subject: [PATCH 15/18] Compress odd shapes at test time rather than commit them Three of the shapes worth covering produce frames of a few hundred kilobytes, which is too much to commit for what they prove. The input is built from a seeded generator and handed to the zstd CLI instead, so the frame is libzstd's own output and the expectation is the exact bytes that went in. The pool size decides which sequence tables libzstd picks. Two hundred tokens gets RLE and repeat tables for the literal lengths, two thousand gets an RLE table for the match lengths, and both put more than 32,512 sequences in one block, which is the only thing that writes the three-byte sequence count. --- test/libzstd_shapes_test.dart | 82 +++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 test/libzstd_shapes_test.dart diff --git a/test/libzstd_shapes_test.dart b/test/libzstd_shapes_test.dart new file mode 100644 index 0000000..ce61cdb --- /dev/null +++ b/test/libzstd_shapes_test.dart @@ -0,0 +1,82 @@ +import 'dart:typed_data'; + +import 'package:pure_zstd/pure_zstd.dart'; +import 'package:test/test.dart'; + +import 'corpus.dart'; + +/// The shapes whose frames are too large to commit. +/// +/// Each input is built here from a seeded generator and handed to the `zstd` +/// CLI, so the frame is libzstd's own output and the expectation is the exact +/// bytes that went in. The round trip is the oracle; nothing is asserted +/// against what this decoder happens to produce. +/// +/// The shapes matter because the real `.hgtz` corpus is 16,490 blocks from +/// one producer at one level, and none of this appears in it: +/// +/// * short tokens from a small pool put more than 32,512 sequences in one +/// block, which is the only thing that writes the three-byte sequence +/// count. The pool size decides which sequence tables libzstd picks: 200 +/// tokens gets RLE and repeat tables for the literal lengths, 2,000 gets an +/// RLE table for the match lengths; +/// * incompressible noise at a low level makes libzstd give up and emit raw +/// blocks and raw literals; +/// * a lopsided alphabet with no matches makes it build one Huffman table and +/// have the blocks that follow reuse it, which is treeless literals. +void main() { + // 600 KB each, as 200,000 three-byte tokens drawn from a pool. + final Uint8List narrowTokens = tokenStream( + tokens: 200, + tokenLength: 3, + count: 200000, + ); + final Uint8List wideTokens = tokenStream( + tokens: 2000, + tokenLength: 3, + count: 200000, + ); + + // 200 KB of noise, which no compressor can do anything with. + final Uint8List random = noise(200000); + + final shapes = arguments, Uint8List input})>{ + 'zstd --ultra -22 --zstd=minMatch=3 over 600 KB of 3-byte tokens, ' + 'pool 200': ( + arguments: ['--ultra', '-22', '--zstd=minMatch=3'], + input: narrowTokens, + ), + 'zstd -19 --zstd=minMatch=3 over 600 KB of 3-byte tokens, pool 200': ( + arguments: ['-19', '--zstd=minMatch=3'], + input: narrowTokens, + ), + 'zstd -19 --zstd=minMatch=3 over 600 KB of 3-byte tokens, pool 2000': ( + arguments: ['-19', '--zstd=minMatch=3'], + input: wideTokens, + ), + 'zstd -1 over 200 KB of noise': (arguments: ['-1'], input: random), + 'zstd --fast=5 over 200 KB of noise': ( + arguments: ['--fast=5'], + input: random, + ), + 'zstd -3 over 300 KB of a skewed alphabet': ( + arguments: ['-3'], + input: skewedAlphabet(300000), + ), + }; + + group( + 'a frame libzstd wrote decodes back to what went in', + () { + for (final shape in shapes.entries) { + test(shape.key, () { + final frame = zstdCli(shape.value.arguments, shape.value.input); + final out = ZstdDecoder().decode(frame); + expect(out.length, shape.value.input.length); + expect(out, orderedEquals(shape.value.input)); + }, timeout: const Timeout(Duration(minutes: 2))); + } + }, + skip: hasZstdCli ? null : 'the zstd CLI is not on the path', + ); +} From 7bdad488ee0fbd0aa44bc42b11dbaf4df5c60ada Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:41:31 -0600 Subject: [PATCH 16/18] Licence under MIT, and name what this was built from MIT rather than a dual with BSD-3. Dual licensing solves a compatibility problem this does not have, since both are permissive and compatible with everything, so the choice would cost a downstream reader a decision and buy them nothing. The debt to zstd is real and a licence is the wrong instrument for it. No code came from libzstd; the format came from RFC 8878 and every answer is checked against libzstd through pyzstd. That belongs in prose where people read it. Also drops the line calling this a feasibility spike. It has 103 tests, an audit against the RFC, and five defects found and closed. --- LICENSE | 21 +++++++++++++++++++++ README.md | 38 ++++++++++++++++++++++++++++++++++---- 2 files changed, 55 insertions(+), 4 deletions(-) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..291790b --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Tag Spilman + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index a4c8a4d..a8d4ef2 100644 --- a/README.md +++ b/README.md @@ -7,9 +7,9 @@ an app that reads block-compressed elevation tiles on a phone would rather not carry a native library per platform to do it. Decompression only. There is no compressor and there is no plan for one. -This is a feasibility spike, not a finished library. It decodes the real data -it was written for, correctly and completely, and it is roughly four times -slower than native zstd. See **Speed** below before depending on it. +It decodes the real data it was written for, correctly and completely, and it +is roughly four times slower than native zstd. See **Speed** below before +depending on it. ## Using it @@ -142,7 +142,37 @@ Run it yourself: ## Licence -Not chosen yet. +MIT. See [LICENSE](LICENSE). + +## Acknowledgements + +This is an independent implementation. It contains no code from libzstd, and +nothing about its licence reaches this package. It would not exist without the +following, and they are named here because that is where the debt actually is. + +**[RFC 8878](https://www.rfc-editor.org/rfc/rfc8878.html), "Zstandard +Compression and the application/zstd Media Type"**, by Yann Collet and Murray +Kucherawy, is the specification this was written from. Every table in +`lib/src/tables.dart` was checked against section 3.1.1.3.2 entry by entry, and +the FSE and Huffman decoders follow the reference algorithms it describes. A +format specification written clearly enough to implement from is a gift, and +this one is. + +**[Zstandard](https://github.com/facebook/zstd)**, by Yann Collet and +contributors at Meta, is the format and the reference implementation. Its +licence is BSD-3-Clause, and it obliges nothing here; it is named because +without it there would be nothing to decode. + +**[pyzstd](https://pypi.org/project/pyzstd/)** wraps libzstd and is the oracle +every test in this package is measured against. Correctness here means +"agrees with libzstd, byte for byte", and pyzstd is how that is asked. The +committed fixtures under `test/fixtures/corpus/` are output of the `zstd` CLI +and of pyzstd, which is tool output rather than anything derived from their +source. + +**The Dart team**, for making `int` an exact 64-bit integer on the VM and on +AOT. The bit reader carries 56 bits in one and XXH64's multiplies wrap exactly, +neither of which would be true on a platform with doubles underneath. ## Measured on the target device, 2026-09-21 From b221ae0e9842cf6db09aee80e28ca0fed7512587 Mon Sep 17 00:00:00 2001 From: Tag Date: Mon, 21 Sep 2026 23:45:05 -0600 Subject: [PATCH 17/18] Add the metadata pub.dev scores a package on description, repository, issue_tracker and topics, plus a changelog, which between them are most of a package's discoverability. The repository URLs are a guess at the new home and want checking before the first publish. publish_to stays none so an accidental publish from inside this checkout cannot succeed. Remove it in the new repository. Version 0.1.0 rather than 0.0.1: a published version can be retracted but never deleted, so the first one should mean something. --- CHANGELOG.md | 30 ++++++++++++++++++++++++++++++ pubspec.yaml | 22 ++++++++++++++++++++-- 2 files changed, 50 insertions(+), 2 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..4b43bed --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,30 @@ +# Changelog + +## 0.1.0 + +First release. + +A zstd decompressor in plain Dart. Decompression only, native platforms only, +one frame per call. + +- Decodes RFC 8878 frames: raw, RLE and compressed blocks; raw, RLE, compressed + and treeless literals in one-stream and four-stream form; Huffman weights in + both the direct nibble and FSE-coded forms; predefined, RLE, FSE and repeat + sequence tables for all three symbols; repeated offsets including the + no-literals shift; multi-block frames reusing an earlier block's tables. +- Verifies the content checksum when a frame carries one, with XXH64 in + `lib/src/xxhash.dart`. +- Refuses dictionaries, several frames in one buffer, and any input left over + after a frame. +- Bounds what a frame header can make it allocate. `ZstdDecoder` takes a + `maxOutputSize`, 64 MiB by default, and a caller passing `expectedSize` gets + an exact check against the frame's own declared size. +- Every failure is a `ZstdException`. + +Correctness is measured against libzstd through pyzstd rather than against +expectations written by hand: 2,615 blocks of real elevation data across 32 +containers, plus a committed corpus of 24 fixtures covering the format branches +that real data never reaches. 103 tests. + +Roughly four times slower than native zstd. On a Pixel 10 Pro XL, profile AOT, +a 128 KiB block decodes in 0.414 ms, which is 302 MB/s. diff --git a/pubspec.yaml b/pubspec.yaml index 952cf4c..62bdd56 100644 --- a/pubspec.yaml +++ b/pubspec.yaml @@ -1,8 +1,26 @@ name: pure_zstd -description: A pure Dart zstd decompressor, with no native dependency. -version: 0.0.1 +description: >- + A zstd decompressor written in plain Dart. Decodes RFC 8878 frames with no + native library and no platform channel, verified against libzstd. +version: 0.1.0 + +# Fill these in with the new repository before the first publish. pub.dev +# scores a package on having them and the score is most of its discoverability; +# a published version can be retracted but never deleted, so they are worth +# getting right the first time. +repository: https://github.com/tagadvance/pure_zstd +issue_tracker: https://github.com/tagadvance/pure_zstd/issues + +# Remove this line to publish. It is here so an accidental `dart pub publish` +# from inside the original checkout cannot succeed. publish_to: none +topics: + - zstd + - zstandard + - compression + - decompression + environment: sdk: ^3.8.0 From 45cab0620f6254f554382dba40c60c85037cf893 Mon Sep 17 00:00:00 2001 From: Tag Date: Tue, 22 Sep 2026 00:06:35 -0600 Subject: [PATCH 18/18] Test the guards that had no test A sweep found no defect in the decoder and several in its tests. Three of the five fixes could be reverted with all 103 tests still green. The content size against expectedSize check is the worst of those, because the .hgtz reader was simplified to rest on it: deleting it left everything passing. Eleven cases cover it now. The RLE block ceiling and the requirement that a frame be the whole input had none either, so there are fixtures for a 13-byte frame that returns 2 MB, for the same shape at exactly the maximum, for four bytes of junk after a frame, and for a trailing skippable frame. Two of the new fixtures close coverage gaps rather than guards. No legal frame in the suite decoded a one-stream Huffman literals section, since the only one present had to be refused. And the only test of hashing each block as it lands rode on the zstd CLI being installed, so a multi-block frame carrying a checksum is committed now. Also replaces an assertion that constrained nothing. "Fills the buffer exactly" checked that the count was over 256 and under 1024, which passes with the guard at two, at three, or absent. The stream encodes 257 symbols, a buffer of 257 takes it and one of 256 does not. And two comments that were wrong. The weight buffer is one larger than the maximum on purpose, and shrinking it to 255 breaks 124 tests. A caller passing expectedSize gets the exact check in addition to the ceiling, not instead of it. --- lib/src/decoder.dart | 7 +- lib/src/huffman.dart | 7 + test/corpus_test.dart | 33 ++ test/fixtures/corpus/lit_one_stream.raw.gz | Bin 0 -> 39 bytes test/fixtures/corpus/lit_one_stream.zst | Bin 0 -> 25 bytes test/fixtures/corpus/manifest.json | 361 +++++++++++-------- test/fixtures/corpus/multiblock_check.raw.gz | Bin 0 -> 2368 bytes test/fixtures/corpus/multiblock_check.zst | Bin 0 -> 2028 bytes test/fixtures/corpus/rle_at_max.raw.gz | Bin 0 -> 162 bytes test/fixtures/corpus/rle_at_max.zst | Bin 0 -> 13 bytes test/fixtures/corpus/rle_over_max.zst | Bin 0 -> 13 bytes test/fixtures/corpus/trailing_junk.zst | Bin 0 -> 29 bytes test/fixtures/corpus/trailing_skippable.zst | Bin 0 -> 37 bytes test/fse_interleaved_test.dart | 32 +- 14 files changed, 271 insertions(+), 169 deletions(-) create mode 100644 test/fixtures/corpus/lit_one_stream.raw.gz create mode 100644 test/fixtures/corpus/lit_one_stream.zst create mode 100644 test/fixtures/corpus/multiblock_check.raw.gz create mode 100644 test/fixtures/corpus/multiblock_check.zst create mode 100644 test/fixtures/corpus/rle_at_max.raw.gz create mode 100644 test/fixtures/corpus/rle_at_max.zst create mode 100644 test/fixtures/corpus/rle_over_max.zst create mode 100644 test/fixtures/corpus/trailing_junk.zst create mode 100644 test/fixtures/corpus/trailing_skippable.zst diff --git a/lib/src/decoder.dart b/lib/src/decoder.dart index 15b61bf..f9c2baf 100644 --- a/lib/src/decoder.dart +++ b/lib/src/decoder.dart @@ -35,8 +35,11 @@ class ZstdDecoder { /// skip it, and this is the bound that replaces it. /// /// A caller that knows the exact size should pass [decode]'s `expectedSize` - /// instead and get an exact check rather than a ceiling. The `.hgtz` reader - /// knows it: a block is `height * width * 2`. + /// as well, and gets an exact check *in addition to* this ceiling rather + /// than instead of it: a frame declaring 70 MiB is refused here before the + /// two are ever compared. Raising it takes a `ZstdDecoder` of one's own, + /// since the top-level [zstdDecode] does not offer it. The `.hgtz` reader + /// knows its size: a block is `height * width * 2`. static const int defaultMaxOutputSize = 64 * 1024 * 1024; /// The ceiling this instance enforces. See [defaultMaxOutputSize]. diff --git a/lib/src/huffman.dart b/lib/src/huffman.dart index 2d66b4c..f26aa53 100644 --- a/lib/src/huffman.dart +++ b/lib/src/huffman.dart @@ -14,6 +14,13 @@ class HuffmanTable { /// The largest table log the format allows for literals. static const int maxLog = 12; + /// One larger than the 255 weights a table may carry, deliberately. + /// + /// `fseDecodeInterleaved` guards each pass for the three symbols its + /// exhausted branches can write, so a 255-weight stream whose final pass + /// writes two has to be able to start that pass at 253. Shrinking this to + /// 255 looks like a tightening and would refuse every 255-weight table, + /// which the real elevation containers are full of. static const int _maxSymbols = 256; /// One entry per table slot: the symbol in the low byte, the code length diff --git a/test/corpus_test.dart b/test/corpus_test.dart index 84847fb..d025dc9 100644 --- a/test/corpus_test.dart +++ b/test/corpus_test.dart @@ -71,6 +71,39 @@ void main() { }); }); + group('a frame whose declared size is not the one asked for', () { + // A guard callers rest on, and it had no test: + // deleting the whole check left every other case green. A caller that + // knows the size, as the .hgtz reader does, gets an exact comparison + // against the frame's own figure rather than only the ceiling. Left to + // the end, a frame declaring less decoded its own content happily and a + // frame declaring more allocated whatever it asked for first. + for (final entry in corpus.where( + // Not a buffer holding more than one frame: this decoder refuses + // those whatever size it is told to expect. + (entry) => + entry.legal && + entry.expectedSize == null && + entry.trailing == null && + entry.size! > 1, + )) { + test('${entry.name} is refused when asked for one byte more', () { + expect( + () => + ZstdDecoder().decode(entry.frame, expectedSize: entry.size! + 1), + throwsA(isA()), + ); + }); + + test('${entry.name} still decodes when asked for the size it has', () { + expect( + ZstdDecoder().decode(entry.frame, expectedSize: entry.size), + orderedEquals(entry.plain), + ); + }); + } + }); + group('a frame declaring no content size', () { final entry = corpus.singleWhere((entry) => entry.name == 'nofcs_windowed'); final size = entry.expectedSize!; diff --git a/test/fixtures/corpus/lit_one_stream.raw.gz b/test/fixtures/corpus/lit_one_stream.raw.gz new file mode 100644 index 0000000000000000000000000000000000000000..8cf50af36922ab60f64b162731c7a4f1d7b8db46 GIT binary patch literal 39 ucmb2|=3q#b+LXog-`mH>$J58hizP_YQ$fp8&||3z6N6mkDo+Up1_l7xH3_-^ literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/lit_one_stream.zst b/test/fixtures/corpus/lit_one_stream.zst new file mode 100644 index 0000000000000000000000000000000000000000..6b21a780061bf5c481ba26dd5c4825b8d614e6ae GIT binary patch literal 25 hcmdPcs{dC(qLqQ6iIJJ@!OPu@R|Gddn!4vR0|0mN3YGu> literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/manifest.json b/test/fixtures/corpus/manifest.json index a9c8195..43a7a70 100644 --- a/test/fixtures/corpus/manifest.json +++ b/test/fixtures/corpus/manifest.json @@ -1,157 +1,208 @@ [ - { - "name": "twoframes", - "frame": "twoframes.zst", - "command": "python3: pyzstd.compress(A, 3) + pyzstd.compress(B, 3), A = 5000 xorshift64 bytes, B = b'the quick brown fox ' * 500", - "verdict": "legal", - "plain": [ - "A.raw.gz", - "B.raw.gz" - ], - "size": 15000, - "trailing": "a second complete frame" - }, - { - "name": "checksum_good", - "frame": "checksum_good.zst", - "command": "zstd -q -c -3 --check A.bin", - "verdict": "legal", - "plain": [ - "A.raw.gz" - ], - "size": 5000 - }, - { - "name": "checksum_flipped", - "frame": "checksum_good.zst", - "command": "zstd -q -c -3 --check A.bin, then the last checksum byte flipped", - "verdict": "refused", - "why": "pyzstd refuses it: ZstdError", - "derive": "the last checksum byte flipped" - }, - { - "name": "checksum_zeroed", - "frame": "checksum_good.zst", - "command": "zstd -q -c -3 --check A.bin, then all four checksum bytes zeroed", - "verdict": "refused", - "why": "pyzstd refuses it: ZstdError", - "derive": "all four checksum bytes zeroed" - }, - { - "name": "checksum_truncated", - "frame": "checksum_good.zst", - "command": "zstd -q -c -3 --check A.bin, then the last two bytes cut", - "verdict": "refused", - "why": "pyzstd refuses it: ZstdError", - "derive": "the last two bytes cut" - }, - { - "name": "fcs_max_u64", - "frame": "fcs_max_u64.zst", - "command": "python3: pyzstd.compress(b'hello world', 3) with the descriptor rewritten to Frame_Content_Size_flag=3 declaring 18446744073709551615", - "verdict": "refused", - "why": "a 27-byte frame asking for a size no allocator will give; pyzstd refuses it: ZstdError" - }, - { - "name": "fcs_2p40", - "frame": "fcs_2p40.zst", - "command": "python3: pyzstd.compress(b'hello world', 3) with the descriptor rewritten to Frame_Content_Size_flag=3 declaring 1099511627776", - "verdict": "refused", - "why": "a 27-byte frame asking for a size no allocator will give; pyzstd refuses it: MemoryError" - }, - { - "name": "fcs8", - "frame": "fcs8.zst", - "command": "python3: pyzstd.compress(B, 3) with the descriptor rewritten to Frame_Content_Size_flag=3 and an 8-byte size", - "verdict": "legal", - "plain": [ - "B.raw.gz" - ], - "size": 10000 - }, - { - "name": "huf_weight_overrun", - "frame": "huf_weight_overrun.zst", - "command": "hand-built: a compressed block whose Huffman weight description is [0x10, 0x3F] and whose FSE weight stream is 33 x 0xAA then 0x02", - "verdict": "refused", - "why": "the weight stream decodes past the 256th weight; pyzstd refuses it: ZstdError" - }, - { - "name": "rlelit_fmt0_R31", - "frame": "rlelit_fmt0_R31.zst", - "command": "hand-built: one compressed block, RLE literals Size_Format=0, Regenerated_Size=31, byte 0x5A, Number_of_Sequences=0", - "verdict": "legal", - "plain": [ - "rlelit_fmt0_R31.raw.gz" - ], - "size": 31 - }, - { - "name": "rlelit_fmt1_R4000", - "frame": "rlelit_fmt1_R4000.zst", - "command": "hand-built: one compressed block, RLE literals Size_Format=1, Regenerated_Size=4000, byte 0x5A, Number_of_Sequences=0", - "verdict": "legal", - "plain": [ - "rlelit_fmt1_R4000.raw.gz" - ], - "size": 4000 - }, - { - "name": "rlelit_fmt2_R70000", - "frame": "rlelit_fmt2_R70000.zst", - "command": "hand-built: one compressed block, RLE literals Size_Format=2, Regenerated_Size=70000, byte 0x5A, Number_of_Sequences=0", - "verdict": "legal", - "plain": [ - "rlelit_fmt2_R70000.raw.gz" - ], - "size": 70000 - }, - { - "name": "rlelit_fmt2_R131072", - "frame": "rlelit_fmt2_R131072.zst", - "command": "hand-built: one compressed block, RLE literals Size_Format=2, Regenerated_Size=131072, byte 0x5A, Number_of_Sequences=0", - "verdict": "legal", - "plain": [ - "rlelit_fmt2_R131072.raw.gz" - ], - "size": 131072 - }, - { - "name": "mixed_rle_raw", - "frame": "mixed_rle_raw.zst", - "command": "hand-built: [RLE 5000 x 0xAB][raw 33 bytes][RLE 40000 x 0x07] in one frame", - "verdict": "legal", - "plain": [ - "mixed_rle_raw.raw.gz" - ], - "size": 45033 - }, - { - "name": "nofcs_windowed", - "frame": "nofcs_windowed.zst", - "command": "zstd -3 --no-check from stdin over b'the quick brown fox jumps over the lazy dog. ' * 24000", - "verdict": "legal", - "plain": [ - "text.raw.gz" - ], - "size": 1080000, - "expectedSize": 1080000 - }, - { - "name": "direct_nibble", - "frame": "direct_nibble.zst", - "command": "block 183 of path/to/srtm1z/N00E042.hgtz, lifted out as its own frame; its literals carry the direct-nibble Huffman weight form", - "verdict": "legal", - "plain": [ - "direct_nibble.raw.gz" - ], - "size": 131072 - }, - { - "name": "dictflag", - "frame": "dictflag.zst", - "command": "python3: pyzstd.compress(A, 3) with Dictionary_ID_flag forced to 1 in the frame header descriptor", - "verdict": "refused", - "why": "a dictionary this decoder does not have; pyzstd refuses it: ZstdError" - } + { + "name": "twoframes", + "frame": "twoframes.zst", + "command": "python3: pyzstd.compress(A, 3) + pyzstd.compress(B, 3), A = 5000 xorshift64 bytes, B = b'the quick brown fox ' * 500", + "verdict": "legal", + "plain": [ + "A.raw.gz", + "B.raw.gz" + ], + "size": 15000, + "trailing": "a second complete frame" + }, + { + "name": "checksum_good", + "frame": "checksum_good.zst", + "command": "zstd -q -c -3 --check A.bin", + "verdict": "legal", + "plain": [ + "A.raw.gz" + ], + "size": 5000 + }, + { + "name": "checksum_flipped", + "frame": "checksum_good.zst", + "command": "zstd -q -c -3 --check A.bin, then the last checksum byte flipped", + "verdict": "refused", + "why": "pyzstd refuses it: ZstdError", + "derive": "the last checksum byte flipped" + }, + { + "name": "checksum_zeroed", + "frame": "checksum_good.zst", + "command": "zstd -q -c -3 --check A.bin, then all four checksum bytes zeroed", + "verdict": "refused", + "why": "pyzstd refuses it: ZstdError", + "derive": "all four checksum bytes zeroed" + }, + { + "name": "checksum_truncated", + "frame": "checksum_good.zst", + "command": "zstd -q -c -3 --check A.bin, then the last two bytes cut", + "verdict": "refused", + "why": "pyzstd refuses it: ZstdError", + "derive": "the last two bytes cut" + }, + { + "name": "fcs_max_u64", + "frame": "fcs_max_u64.zst", + "command": "python3: pyzstd.compress(b'hello world', 3) with the descriptor rewritten to Frame_Content_Size_flag=3 declaring 18446744073709551615", + "verdict": "refused", + "why": "a 27-byte frame asking for a size no allocator will give; pyzstd refuses it: ZstdError" + }, + { + "name": "fcs_2p40", + "frame": "fcs_2p40.zst", + "command": "python3: pyzstd.compress(b'hello world', 3) with the descriptor rewritten to Frame_Content_Size_flag=3 declaring 1099511627776", + "verdict": "refused", + "why": "a 27-byte frame asking for a size no allocator will give; pyzstd refuses it: MemoryError" + }, + { + "name": "fcs8", + "frame": "fcs8.zst", + "command": "python3: pyzstd.compress(B, 3) with the descriptor rewritten to Frame_Content_Size_flag=3 and an 8-byte size", + "verdict": "legal", + "plain": [ + "B.raw.gz" + ], + "size": 10000 + }, + { + "name": "huf_weight_overrun", + "frame": "huf_weight_overrun.zst", + "command": "hand-built: a compressed block whose Huffman weight description is [0x10, 0x3F] and whose FSE weight stream is 33 x 0xAA then 0x02", + "verdict": "refused", + "why": "the weight stream decodes past the 256th weight; pyzstd refuses it: ZstdError" + }, + { + "name": "rlelit_fmt0_R31", + "frame": "rlelit_fmt0_R31.zst", + "command": "hand-built: one compressed block, RLE literals Size_Format=0, Regenerated_Size=31, byte 0x5A, Number_of_Sequences=0", + "verdict": "legal", + "plain": [ + "rlelit_fmt0_R31.raw.gz" + ], + "size": 31 + }, + { + "name": "rlelit_fmt1_R4000", + "frame": "rlelit_fmt1_R4000.zst", + "command": "hand-built: one compressed block, RLE literals Size_Format=1, Regenerated_Size=4000, byte 0x5A, Number_of_Sequences=0", + "verdict": "legal", + "plain": [ + "rlelit_fmt1_R4000.raw.gz" + ], + "size": 4000 + }, + { + "name": "rlelit_fmt2_R70000", + "frame": "rlelit_fmt2_R70000.zst", + "command": "hand-built: one compressed block, RLE literals Size_Format=2, Regenerated_Size=70000, byte 0x5A, Number_of_Sequences=0", + "verdict": "legal", + "plain": [ + "rlelit_fmt2_R70000.raw.gz" + ], + "size": 70000 + }, + { + "name": "rlelit_fmt2_R131072", + "frame": "rlelit_fmt2_R131072.zst", + "command": "hand-built: one compressed block, RLE literals Size_Format=2, Regenerated_Size=131072, byte 0x5A, Number_of_Sequences=0", + "verdict": "legal", + "plain": [ + "rlelit_fmt2_R131072.raw.gz" + ], + "size": 131072 + }, + { + "name": "mixed_rle_raw", + "frame": "mixed_rle_raw.zst", + "command": "hand-built: [RLE 5000 x 0xAB][raw 33 bytes][RLE 40000 x 0x07] in one frame", + "verdict": "legal", + "plain": [ + "mixed_rle_raw.raw.gz" + ], + "size": 45033 + }, + { + "name": "nofcs_windowed", + "frame": "nofcs_windowed.zst", + "command": "zstd -3 --no-check from stdin over b'the quick brown fox jumps over the lazy dog. ' * 24000", + "verdict": "legal", + "plain": [ + "text.raw.gz" + ], + "size": 1080000, + "expectedSize": 1080000 + }, + { + "name": "direct_nibble", + "frame": "direct_nibble.zst", + "command": "block 183 of path/to/srtm1z/N00E042.hgtz, lifted out as its own frame; its literals carry the direct-nibble Huffman weight form", + "verdict": "legal", + "plain": [ + "direct_nibble.raw.gz" + ], + "size": 131072 + }, + { + "name": "dictflag", + "frame": "dictflag.zst", + "command": "python3: pyzstd.compress(A, 3) with Dictionary_ID_flag forced to 1 in the frame header descriptor", + "verdict": "refused", + "why": "a dictionary this decoder does not have; pyzstd refuses it: ZstdError" + }, + { + "name": "rle_over_max", + "frame": "rle_over_max.zst", + "command": "hand-built: one last RLE block declaring Block_Size 2,000,000", + "verdict": "refused", + "why": "RFC 8878 bounds Block_Size by Block_Maximum_Size, never over 128 KB; libzstd's one-shot decoder accepts it and returns 2 MB" + }, + { + "name": "rle_at_max", + "frame": "rle_at_max.zst", + "command": "hand-built: one last RLE block at exactly Block_Maximum_Size, 131072", + "verdict": "legal", + "plain": [ + "rle_at_max.raw.gz" + ], + "size": 131072 + }, + { + "name": "lit_one_stream", + "frame": "lit_one_stream.zst", + "command": "zstd -19 -c --no-check over a file holding b'aaadaabaaaddbbaaacabbada'; from a file rather than stdin so the frame declares its own content size", + "verdict": "legal", + "plain": [ + "lit_one_stream.raw.gz" + ], + "size": 24 + }, + { + "name": "multiblock_check", + "frame": "multiblock_check.zst", + "command": "python3: pyzstd.compress(body, {level: 9, checksumFlag: 1, targetCBlockSize: 8192}), body = 165,450 bytes of runs", + "verdict": "legal", + "plain": [ + "multiblock_check.raw.gz" + ], + "size": 165450 + }, + { + "name": "trailing_junk", + "frame": "trailing_junk.zst", + "command": "hand-built: the lit_one_stream frame with four bytes of junk appended", + "verdict": "refused", + "why": "reading one frame is the whole API, so input left over after it is input this decoder does not understand; it used to be ignored" + }, + { + "name": "trailing_skippable", + "frame": "trailing_skippable.zst", + "command": "hand-built: the lit_one_stream frame followed by a skippable frame", + "verdict": "refused", + "why": "a trailing skippable frame was accepted and silently ignored; only a leading one was refused, and only because the magic check fired" + } ] diff --git a/test/fixtures/corpus/multiblock_check.raw.gz b/test/fixtures/corpus/multiblock_check.raw.gz new file mode 100644 index 0000000000000000000000000000000000000000..32aff9fbc464e44aaeaf9a16fc2da03df35fd5e0 GIT binary patch literal 2368 zcmV-G3BUFqiwFQi8nS8v|LxslgO+Ik2GC~WWZQO4wr$(CZQHw>n{3;*ZQFLAwk8c; zJh**d=Z~Bq9v+^ao&7tUy#c(2~GqIeC?OYre;<^&grTDHTa3i5RiQG!;UJ?V68j;M9a1uP^~cNGM7~F*1r1Qi7C{#FQeZG(lxZDoa#3vdR-yfwYRmRU)r41yyLM zN<}p~s#8*fmYUSmqNg@Rb!e(fRXw`uQ`UgChSW8puQ3BnSZK;bGd7wt(t?$i%(P;s zHA8J!YRgnRw%Rk+fwhjzbz-kG2VHpR%0)Lmx^vQlm!918;-@!9eR%52RX@J^b2fmt zf!qz^Z?FJ{h+wD?hKXUgAV!E{q%cN_W3)iVh-9o##))OTU?zxWqHrdO=N|$6E295| zG+9hj1T|Gu(}Xo$Tr&hVQ)IJ*<|DS*f}10{xx$+#zWEYZAcKWcSR{wVl2{^(rP5d? zkL40sA(NF-StXa%l3630wbEH9pY;;jAft^^+9apVlG-AxtJdin^kztIE2juImcBp|YDwyQQ|6!0r5_!ABM ziw*(AfPi8`Ah95@*x*1Ma3n4`6b~GW4^AWiXA*)_iNLwU;6f5`B`LU+3|vbNZlnNr zQi5Blz`fL9KpHS2Ef|sxj7bj$WdNfxf??iZTqZCfGnkPDOvwu7WCN43gIPJiw47jG zF0ddsSdj-T$qUxx1B>#5RRzGZf?!=Cu%R&6Q3Px$3icEOn~H;7CBU|lU|%U9pfr$B z28bvNWRwF!$^$7CfS8IvP9-3yGLTdSh^h)?RRhAR18FsYxSBv-Euf$_&`<}cs0(z| z14`-xEe(L0hCojvpr|p>)C8z%3UoCC%9;ahEr7a~Kwm3hpf#}22AF6IY_tPL+5;;c zfSHcKPA6cfGqBVJnCc2_bpyt_18Y5ixt_pYFW{gz@X!ai=nH)G15Ww_F9U#^fxyoo z;Ak-LGz7RB3VaO%&V~bTBY?Y+z~3kkz-SP`7!bl(5W_eS#CQtk#62eO2?bASct*u@ zI$lunl9pH0yr$<3MQ>?(N7Z|}K2Y|NwolZ3rtb>_Us?FZL=ZMS81ZB!C^Ny>3C>Um zmO?TWiY+h3LbDczxv=bo;~+c_5x9uRMH+a*?DEN=mV$5=?5*q!CV9@uU+_dJ$z1QbsX( z3o4VSG7BqYEr-~03NDxEatkky`0`31pA7O#p@19;N}`Y~3QMDiJc>%B zm`sXGrG#8cN~V--N=v7Ve9B6woQ%p#se+s;N~)5qDod-1ysAp9n#`(8t%lrcO0Jgd zYD=$<{OT&8o(k$Kp@AA2Dx#4p8Y`oTI+`k^nM#@~rG;8rDyEfcS}UiGdfFBk zse_t2DyoyJIxDM-y1FW?o65Q?t%urrDz2C6dMmGw`ub|1pAPzKVSpY6YGRNs25V!8 zK89*!m`;XkWrSWvYG#yfMr&t`e#UBOoQ_b=V|PcSju!ejX2L9(WEoxy|6h7I0-NxU>yi+YWB*0C#qRTf4x$ z-C)2TFk&wlvJZ^e4+b3oqYi>$hrqbQV8RhF<0zPN49qzWCY=DYPJ(Htz`WC7!5OgP zELd_5tT_)BT>z^tf@PP$y31h06|mzf*m4c*xehkn0K0C2ZMVR_+d#k_AmJ_$aSzD2 z4}?4bQXT>^kAR%VK+qE)=_wHP49I#8guMXLUIKBifV|g0!5g6AEl}|e=y(s5d;nTL z0yUq2p3gwh7oh1YQ1uPy3IddQ0BxQ?T~MGe7%&hVSO@`3gakH10V7_(N@!pv46qXx z7zzh0g$Jf009z4(u}HvLWMD1|uoo3Lhz2}F2QFd&A2ETGSinnc;3f|66Bjs&2Ry|G zt`Y!W34yajz*}PAE(!3L6au6sm z0y>k>g@~?XbR(oYDLshkNlq_+NBdkAZ1Be?)-XIDF(=ddJp>zzRWH>D& ms2NGmD2hhYG={3Nbd95IJZ%%Gn@Haz2L1;SRn+fFh5`V4kvNn9 literal 0 HcmV?d00001 diff --git a/test/fixtures/corpus/multiblock_check.zst b/test/fixtures/corpus/multiblock_check.zst new file mode 100644 index 0000000000000000000000000000000000000000..0f137e634c8ac6e9d541f4ddf11cfcb71ea85c0d GIT binary patch literal 2028 zcmdtj=`$M$7zXf&`#$SF+s1vw-G)VM6>&zID2h@-i!N5(N8&zOEfq)IHYq}@l&quf zGbyU5meM#%6;)^5x|yBXnfhNhOb zj_!qv5Iy~$3=EB+#y?*&xom1?4zsYdvbM2>+aazXQTA6I9G$K?ySQF=bHCx?iT3io zdCSMw@AfbL0fD~;-3bl}4GWKmjEauIU}NLr6B3h>Q&Q9J-b=s#AS3f(7A`v{H}6q? zL19sGNom<{c*5iIib`Tt^^>PH&uZ(Q*Ef)UZ)|FAdC}U|-qG3B-P7CGKR~9u92^>c zHS&6NZ2Zl{+jo;w@26)zP-o}n7d|d7eOms!vby$V{p-f&*7nZsx4r#?LjWD{zgRu{ zf86cIS!0_jjF*#sq95hnUF9>9=3!xom1g;;y1ly3zb|4vPk&a zVi}~2Iskyf)po<;lf&(WCa?1JQ~j{b*$+)VFN2+DZoV$gSFu6g4GjfTUjzkEA6w~E z`%DlC5EX639k24CJY~4GBw78ONly|hJJ)hfO_6@cou#`+6KFqCO^+Mhp^Ie+yGWuP z^r9#6JQYHWNxLhfOS|M;C1E0h0f2`y$w}Nzs^E=Hc#%0|n?$m_nv`E2FHRs#pKmtV>K1}KBg7}b>Cz%qx(48G(U7X&B9r>sJ za_Dt#_73p(SrO76_hNk7cQ`N>;F#^EQZi(%$T%gkp*Nyq34w|g$Z||UmM?lEmN-hk zSZA>MF{~-pn;;I1gi&-4Z~NU??-wzEt!s?rZB#w$O)=cS!r@PnmkEjZ}Yx z)1-^Vm-@1(LcG%mEqfG1gRKasRTY zS9~qIGEu&l-&~xxhx@{~ZK9Cf1(|rMk&;+G{RbU%bjSTP^r-Ny4XRI#JXz|M77 zQby04XGpgPjuuicgC;jsMJug!;buXN?e9WLsb`BBTXR&M>aJC4fMCz-YM%{1Qn4Um z=LUVj8;K_LkJjg`Vj3}dN8Rq=LBmk~_eV*5v-2@oiPD&C?OMSRG!}*pMjvz4-%}_o z;f!UCBi#yE90`Vc7>lX5u=xczw(;}{A+EV6OSxZlW0kTT!P1>3xkfEUsI@L8 z>`4T_!L2uaOlM!$ogPP#-rDIs@h)~lIglkS)}D2oc2zwMY{gAHPu)F(LgH2`@#$ run(256), throwsA(isA())); + }, + ); }); }