halkyon-days/pure-zstd
Watch
1
0
Fork
You've already forked pure-zstd
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add a pure-Dart zstd decoder, as a benchmark

Browse source 
A feasibility spike, standalone so it can move to its own repository. Not
wired into the app and not a dependency of it.

Decompression only, and only what our containers need: no dictionaries, no
multi-frame, no streaming, and the content checksum is parsed but never
verified. Native only, since the bit reader carries 56 bits in an int.

It is correct on real data. The committed fixture decodes to the samples
the generator wrote, and tool/oracle.py drives a byte-for-byte comparison
against pyzstd over 32 containers and 2,615 blocks from both tiers, every
thinned column count from 3601 to 361, and 88 S to 80 N. All identical.

The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median
against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles
the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The
cost is the sequence loop and nothing else: a terrain block at level 19
carries around 1,700 literals and 14,400 sequences, so optimising the
literal path bought nothing.
This commit is contained in:
Tag 2026-09-21 21:37:56 -06:00
commit d970a0cca8
16 changed files with 1717 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

5
.gitignore vendored Normal file
View file

@ -0,0 +1,5 @@
# Destined for its own repository, so it carries its own rules rather than
# relying on the enclosing one.
.dart_tool/
# A library, not an application: the lockfile is the consumer's business.
pubspec.lock

101
README.md Normal file
View file

@ -0,0 +1,101 @@
# pure_zstd
A zstd decompressor in plain Dart, with no native dependency.
It exists because every zstd binding on pub.dev is a wrapper over libzstd, and
an app that reads block-compressed elevation tiles on a phone would rather not
carry a native library per platform to do it. Decompression only. There is no
compressor and there is no plan for one.
This is a feasibility spike, not a finished library. It decodes the real data
it was written for, correctly and completely, and it is roughly four times
slower than native zstd. See **Speed** below before depending on it.
## Using it
```dart
import 'package:pure_zstd/pure_zstd.dart';
final plain = zstdDecode(frame);
// Decoding many frames: hold the decoder, which keeps its entropy tables and
// scratch buffers between calls and then allocates only the output.
final decoder = ZstdDecoder();
for (final frame in frames) {
use(decoder.decode(frame));
}
```
A frame that declares no content size needs `expectedSize`, because the
output is allocated whole rather than grown.
## What it implements
RFC 8878 is the specification. Of it:
| Part of the format | State |
| --- | --- |
| Frame header, single segment and windowed | Decoded. The window descriptor is skipped, since the output is allocated from the content size. |
| Raw, RLE and compressed blocks | All three. |
| Multi-block frames | Yes, including a later block reusing an earlier one's tables and recent offsets. |
| Raw, RLE, compressed and treeless literals | All four, in one-stream and four-stream form. |
| Huffman weights | Both the direct nibble form and the FSE-coded form. |
| Sequence tables | Predefined, RLE, FSE-compressed and repeat, for all three of literal length, match length and offset. |
| Repeated offsets | Yes, including the no-literals shift. |
| Content checksum | Parsed and **skipped, never verified**. |
| Dictionaries | Refused. |
| Skippable frames, several frames in one buffer | Refused. Reading one frame is the whole API. |
| Streaming, or decoding part of a frame | Not offered. |
It targets native Dart and not the web: the literals header reads a five-byte
field with a 32-bit shift, and the bit reader carries 56 bits in an `int`.
Both are exact on the VM and on AOT, and neither is on JavaScript.
## Correctness
The reference is pyzstd, which wraps libzstd. `tool/oracle.py` extracts every
block of a `.hgtz` container as its own frame, decompresses it with pyzstd,
hands it to `tool/verify.dart`, and compares byte for byte.
./tool/oracle.py path/to/srtm1z/N39W105.hgtz
./tool/oracle.py --every 1700 path/to/srtm1z path/to/srtm3z
As of 2026-09-21 that is 32 real containers, 2,615 blocks and 0.3 GB of
output, all identical, spanning both the one and three arc-second sets, every
thinned column count from 3601 down to 361, and latitudes from 88 S to 80 N.
`dart test` needs no Python. It decodes the five blocks of the container in
`test/fixtures/`, whose every sample is a known function of its position, so a
wrong decode cannot pass.
## Speed
Measured 2026-09-21 on this eight-core Linux box, `dart compile exe`, Dart
3.12.2. One 256 by 256 block is 131,072 bytes out. The frames are read into
memory first, so only decoding is timed, and the figure includes allocating
the output buffer because the API allocates it.
| Block set | pure_zstd, AOT | pyzstd (libzstd) |
| --- | --- | --- |
| Colorado, `N39W105`, 196 blocks, median | **0.66 ms** | 0.18 ms |
| Alaska, `N60W160`, 98 blocks, median | 0.94 ms | not measured |
| One block, repeated 1000 times | 0.85 ms | 0.22 ms |
| 113 blocks, a 360 degree sweep | 67 ms | 19 ms |
That is 140 to 200 MB/s of output. The JIT is about 25% slower than AOT, so
measure with `dart compile exe`.
The cost is almost all in the sequence loop rather than in Huffman literals.
A block of real terrain carries around 1,700 literals and 14,400 sequences,
so the hot path is three FSE state updates, six bit reads and a nine-byte
match copy, about 46 ns per sequence against native's dozen or so. The
headroom left is bounds checks and the byte-at-a-time match copy; widening
that copy to 64-bit words, the way libzstd does, is the next thing to try.
Run it yourself:
dart compile exe benchmark/bench.dart -o /tmp/bench && /tmp/bench
## Licence
Not chosen yet.

1
analysis_options.yaml Normal file
View file

@ -0,0 +1 @@
include: package:lints/recommended.yaml

122
benchmark/bench.dart Normal file
View file

@ -0,0 +1,122 @@
import 'dart:io';
import 'dart:typed_data';
import 'package:pure_zstd/pure_zstd.dart';
/// Times decoding real `.hgtz` blocks.
///
/// dart run benchmark/bench.dart [path to a .hgtz]
/// dart compile exe benchmark/bench.dart -o /tmp/bench && /tmp/bench
///
/// Reports the cost of one 256 by 256 block, which is 131,072 bytes out, and
/// the cost of the 113 blocks a 360 degree skyline sweep touches. The frames
/// are read into memory first, so nothing but decoding is timed. The output
/// buffer is allocated per call, as the API allocates it, so the figure
/// includes that allocation and the garbage it makes.
const int sweepBlocks = 113;
void main(List<String> arguments) {
final path = arguments.isEmpty
? 'path/to/srtm1z/N39W105.hgtz'
: arguments.single;
final bytes = File(path).readAsBytesSync();
final header = ByteData.sublistView(bytes);
final rows = header.getUint16(6);
final columns = header.getUint16(8);
final block = header.getUint16(10);
final blockColumns = (columns + block - 1) ~/ block;
final blockRows = (rows + block - 1) ~/ block;
// Only the square blocks, so every timing is over the same 131,072 bytes.
final full = <Uint8List>[];
for (var index = 0; index < blockRows * blockColumns; index++) {
final entry = 12 + 8 * index;
final top = (index ~/ blockColumns) * block;
final left = (index % blockColumns) * block;
if (rows - top < block || columns - left < block) {
continue;
}
final offset = header.getUint32(entry);
full.add(
Uint8List.sublistView(
bytes,
offset,
offset + header.getUint32(entry + 4),
),
);
}
if (full.isEmpty) {
stderr.writeln('$path has no full $block by $block block');
exit(2);
}
final decoder = ZstdDecoder();
final size = decoder.decode(full.first).length;
stdout.writeln(path);
stdout.writeln(
'${full.length} full blocks of $size bytes, '
'${(full.map((f) => f.length).reduce((a, b) => a + b) / full.length).round()}'
' compressed bytes on average',
);
// One block, over and over. The same frame every time, so the caches are as
// warm as they will ever be. This is the optimistic figure.
final one = full[full.length ~/ 2];
for (var i = 0; i < 300; i++) {
decoder.decode(one);
}
final repeated = <int>[];
for (var i = 0; i < 1000; i++) {
final watch = Stopwatch()..start();
decoder.decode(one);
watch.stop();
repeated.add(watch.elapsedMicroseconds);
}
_report('one block repeated, warm', repeated, size);
// Every full block once, which is the mix a real sweep meets: different
// entropy tables, different literal counts, different sequence counts.
final each = <int>[];
for (var pass = 0; pass < 8; pass++) {
for (final frame in full) {
final watch = Stopwatch()..start();
decoder.decode(frame);
watch.stop();
if (pass > 0) {
each.add(watch.elapsedMicroseconds);
}
}
}
_report('every full block, warm', each, size);
// A sweep as one unit, since that is the latency the user feels.
final sweeps = <int>[];
for (var pass = 0; pass < 30; pass++) {
final watch = Stopwatch()..start();
for (var i = 0; i < sweepBlocks; i++) {
decoder.decode(full[i % full.length]);
}
watch.stop();
sweeps.add(watch.elapsedMicroseconds);
}
sweeps.sort();
final sweep = sweeps[sweeps.length ~/ 2] / 1000;
stdout.writeln(
'$sweepBlocks block sweep, median ${sweep.toStringAsFixed(2)} ms '
'(${(sweep / sweepBlocks * 1000).toStringAsFixed(0)} us per block)',
);
}
void _report(String label, List<int> microseconds, int size) {
microseconds.sort();
final median = microseconds[microseconds.length ~/ 2];
final low = microseconds[microseconds.length ~/ 10];
final high = microseconds[microseconds.length * 9 ~/ 10];
final mbPerSecond = size / median; // bytes per microsecond is MB/s.
stdout.writeln(
'${label.padRight(28)} ${(median / 1000).toStringAsFixed(3)} ms '
'median (p10 ${(low / 1000).toStringAsFixed(3)}, '
'p90 ${(high / 1000).toStringAsFixed(3)}, '
'n=${microseconds.length}) ${mbPerSecond.toStringAsFixed(0)} MB/s',
);
}

8
lib/pure_zstd.dart Normal file
View file

@ -0,0 +1,8 @@
/// A pure Dart zstd decompressor.
///
/// Decompression only, and only the subset of RFC 8878 that the elevation
/// containers this was written for use. See the README for what is left out.
library;
export 'src/decoder.dart' show ZstdDecoder, zstdDecode;
export 'src/exception.dart' show ZstdException;

140
lib/src/bit_reader.dart Normal file
View file

@ -0,0 +1,140 @@
import 'dart:typed_data';
import 'exception.dart';
/// Reads a zstd entropy bitstream, which runs backwards.
///
/// Every FSE and Huffman stream in a zstd frame is written from its end: the
/// last byte holds the first bits, and within a byte the most significant bit
/// comes first. Read as a little-endian integer the whole stream is one big
/// number consumed from the top down, which is what this does.
///
/// The final byte carries a set bit marking where the data stops; the zeroes
/// above it are padding and the marker itself is not data.
class ReverseBitReader {
ReverseBitReader();
Uint8List _bytes = Uint8List(0);
/// A view over [_bytes], so a refill can take four bytes at a time. Kept
/// between resets, since every stream in one block shares a buffer.
ByteData _view = ByteData(0);
/// Index of the first byte of the stream. Reading stops when it is reached.
int _start = 0;
/// Index of the next byte to pull into [_acc], counting down to [_start].
int _next = 0;
/// The unread bits, in the low [_available] bits, next bit at the top.
int _acc = 0;
int _available = 0;
/// Zero bits shifted in past the front of the stream. They are only ever
/// at the bottom of [_acc], so any unread count below this means the
/// stream was over-read.
int _padding = 0;
void reset(Uint8List bytes, int start, int end) {
if (end <= start) {
throw const ZstdException('empty entropy stream');
}
final last = bytes[end - 1];
if (last == 0) {
throw const ZstdException('entropy stream has no end marker');
}
if (!identical(bytes, _bytes)) {
_bytes = bytes;
_view = ByteData.sublistView(bytes);
}
_start = start;
_next = end - 1;
_acc = last;
// bitLength - 1 is the marker's position; the data is everything below.
_available = last.bitLength - 1;
_padding = 0;
}
/// True once more bits have been taken than the stream held.
bool get exhausted => _available < _padding;
/// Tops [_acc] up to at least 49 bits, so a [peek] of up to 32 is safe and
/// several small reads can follow one call.
void fill() {
if (_available > 48) {
return;
}
var acc = _acc & ((1 << _available) - 1);
var available = _available;
var next = _next;
// Four bytes at a time while there is room for them. The stream is one
// little-endian number read from the top down, so a little-endian word
// lands the four bytes in the order the byte-at-a-time loop would.
while (available <= 24 && next - 4 >= _start) {
next -= 4;
acc = (acc << 32) | _view.getUint32(next, Endian.little);
available += 32;
}
while (available <= 48) {
if (next > _start) {
next--;
acc = (acc << 8) | _bytes[next];
} else {
acc <<= 8;
_padding += 8;
}
available += 8;
}
_acc = acc;
_available = available;
_next = next;
}
/// The next [count] bits without consuming them. Caller has called [fill].
int peek(int count) => (_acc >> (_available - count)) & ((1 << count) - 1);
void skip(int count) {
_available -= count;
}
/// Reads [count] bits, refilling as needed. `count == 0` yields 0.
int read(int count) {
if (_available < count) {
fill();
}
_available -= count;
return (_acc >> _available) & ((1 << count) - 1);
}
}
/// Reads the forward, least-significant-bit-first bitstream that an FSE table
/// description uses. Table descriptions are tiny and read once per block, so
/// this favours being obviously correct over being quick.
class ForwardBitReader {
ForwardBitReader(this._bytes, this._end, int startByte)
: _position = startByte * 8;
final Uint8List _bytes;
final int _end;
int _position;
/// The next 32 bits, unconsumed. Reads past the end come back as zeroes.
int peek32() {
final index = _position >> 3;
var value = 0;
for (var k = 0; k < 5; k++) {
final at = index + k;
if (at < _end) {
value |= _bytes[at] << (8 * k);
}
}
return value >> (_position & 7);
}
void skip(int count) {
_position += count;
}
/// The number of bytes consumed, counting a partial byte as whole.
int bytesUsed(int startByte) => ((_position + 7) >> 3) - startByte;
}

604
lib/src/decoder.dart Normal file
View file

@ -0,0 +1,604 @@
import 'dart:typed_data';
import 'bit_reader.dart';
import 'exception.dart';
import 'fse.dart';
import 'huffman.dart';
import 'tables.dart';
/// Decodes zstd frames, holding its entropy tables and scratch between calls
/// so that decoding a second frame allocates nothing but the output.
///
/// One instance is not safe to share between isolates, and a frame must be
/// decoded to completion before the next one starts.
class ZstdDecoder {
static const int magic = 0xFD2FB528;
/// The largest block a frame may contain, and so the most literals one
/// block can hold.
static const int maxBlockSize = 128 * 1024;
final ReverseBitReader _bits = ReverseBitReader();
final HuffmanTable _huffman = HuffmanTable();
final FseTable _literalLengths = FseTable(literalLengthMaxLog);
final FseTable _matchLengths = FseTable(matchLengthMaxLog);
final FseTable _offsets = FseTable(offsetCodeMaxLog);
bool _literalLengthsReady = false;
bool _matchLengthsReady = false;
bool _offsetsReady = false;
/// The three recent offsets. They live for the whole frame, so a later
/// block can refer back to one a previous block established.
int _repeat1 = 1;
int _repeat2 = 4;
int _repeat3 = 8;
Uint8List _literals = Uint8List(maxBlockSize);
/// Where the current block's literals live: either [_literals] or, for raw
/// literals, the frame itself.
Uint8List _literalSource = Uint8List(0);
int _literalAt = 0;
int _literalEnd = 0;
/// Decodes one frame.
///
/// [expectedSize] is only needed when the frame header omits its content
/// size, which the containers this was written for never do.
Uint8List decode(Uint8List src, {int? expectedSize}) {
if (src.length < 6) {
throw const ZstdException('too short to be a zstd frame');
}
if (src[0] | (src[1] << 8) | (src[2] << 16) | (src[3] << 24) != magic) {
throw const ZstdException('not a zstd frame: bad magic');
}
final descriptor = src[4];
var at = 5;
final contentSizeFlag = descriptor >> 6;
final singleSegment = (descriptor >> 5) & 1;
if ((descriptor >> 3) & 1 != 0) {
throw const ZstdException('frame header reserved bit set');
}
final checksum = (descriptor >> 2) & 1;
final dictionaryFlag = descriptor & 3;
if (singleSegment == 0) {
at++; // Window_Descriptor. The output is allocated whole, so unused.
}
if (dictionaryFlag != 0) {
throw const ZstdException('dictionaries are not supported');
}
final contentSizeBytes = switch (contentSizeFlag) {
0 => singleSegment == 1 ? 1 : 0,
1 => 2,
2 => 4,
_ => 8,
};
if (at + contentSizeBytes > src.length) {
throw const ZstdException('truncated frame header');
}
int? contentSize;
if (contentSizeBytes > 0) {
var value = 0;
for (var k = 0; k < contentSizeBytes; k++) {
value |= src[at + k] << (8 * k);
}
// The two-byte form is biased, since a frame that small would use the
// one-byte form.
contentSize = contentSizeBytes == 2 ? value + 256 : value;
at += contentSizeBytes;
}
final size = contentSize ?? expectedSize;
if (size == null) {
throw const ZstdException(
'frame declares no content size and none was given',
);
}
final out = Uint8List(size);
_huffman.log = 0;
_literalLengthsReady = false;
_matchLengthsReady = false;
_offsetsReady = false;
_repeat1 = 1;
_repeat2 = 4;
_repeat3 = 8;
var written = 0;
while (true) {
if (at + 3 > src.length) {
throw const ZstdException('truncated block header');
}
final header = src[at] | (src[at + 1] << 8) | (src[at + 2] << 16);
at += 3;
final last = header & 1;
final type = (header >> 1) & 3;
final blockSize = header >> 3;
if (blockSize > maxBlockSize && type != 1) {
throw ZstdException('block of $blockSize bytes is over the maximum');
}
switch (type) {
case 0:
if (at + blockSize > src.length || written + blockSize > out.length) {
throw const ZstdException('raw block does not fit');
}
out.setRange(written, written + blockSize, src, at);
written += blockSize;
at += blockSize;
case 1:
if (at >= src.length || written + blockSize > out.length) {
throw const ZstdException('RLE block does not fit');
}
out.fillRange(written, written + blockSize, src[at]);
written += blockSize;
at += 1;
case 2:
if (at + blockSize > src.length) {
throw const ZstdException('truncated compressed block');
}
written = _decodeBlock(src, at, at + blockSize, out, written);
at += blockSize;
default:
throw const ZstdException('reserved block type');
}
if (last == 1) {
break;
}
}
if (checksum == 1) {
// Parsed so the frame is consumed whole, but not verified. Every block
// in a .hgtz is its own frame and pyzstd writes none.
at += 4;
}
if (written != out.length) {
throw ZstdException('frame decoded to $written bytes, not ${out.length}');
}
return out;
}
/// Decodes one compressed block, returning the new output position.
int _decodeBlock(
Uint8List src,
int start,
int end,
Uint8List out,
int written,
) {
var at = _decodeLiterals(src, start, end);
if (at == end) {
throw const ZstdException('block has no sequences section');
}
// Number_of_Sequences, one to three bytes.
final first = src[at++];
int sequences;
if (first == 0) {
sequences = 0;
} else if (first < 128) {
sequences = first;
} else if (first < 255) {
if (at >= end) {
throw const ZstdException('truncated sequence count');
}
sequences = ((first - 128) << 8) + src[at++];
} else {
if (at + 2 > end) {
throw const ZstdException('truncated sequence count');
}
sequences = src[at] + (src[at + 1] << 8) + 0x7F00;
at += 2;
}
if (sequences == 0) {
return _copyRemainingLiterals(out, written);
}
if (at >= end) {
throw const ZstdException('truncated compression modes');
}
final modes = src[at++];
if (modes & 3 != 0) {
throw const ZstdException('sequence modes reserved bits set');
}
at = _readSequenceTable(
_literalLengths,
modes >> 6,
literalLengthDefault,
6,
literalLengthMaxSymbol,
src,
at,
end,
_literalLengthsReady,
);
_literalLengthsReady = true;
at = _readSequenceTable(
_offsets,
(modes >> 4) & 3,
offsetCodeDefault,
5,
offsetCodeMaxSymbol,
src,
at,
end,
_offsetsReady,
);
_offsetsReady = true;
at = _readSequenceTable(
_matchLengths,
(modes >> 2) & 3,
matchLengthDefault,
6,
matchLengthMaxSymbol,
src,
at,
end,
_matchLengthsReady,
);
_matchLengthsReady = true;
return _executeSequences(src, at, end, out, written, sequences);
}
int _readSequenceTable(
FseTable table,
int mode,
List<int> predefined,
int predefinedLog,
int maxSymbol,
Uint8List src,
int at,
int end,
bool ready,
) {
switch (mode) {
case 0:
buildPredefined(table, predefined, predefinedLog);
return at;
case 1:
if (at >= end) {
throw const ZstdException('truncated RLE sequence table');
}
if (src[at] > maxSymbol) {
throw ZstdException('RLE sequence symbol ${src[at]} out of range');
}
table.setRle(src[at]);
return at + 1;
case 2:
return at + readFseTable(table, src, at, end, maxSymbol);
default:
if (!ready) {
throw const ZstdException('repeat mode with no previous table');
}
return at;
}
}
/// Reads the literals section, leaving the literals addressable through
/// [_literalSource]. Returns the offset just past the section.
int _decodeLiterals(Uint8List src, int start, int end) {
if (start >= end) {
throw const ZstdException('empty compressed block');
}
var at = start;
final header = src[at];
final type = header & 3;
final sizeFormat = (header >> 2) & 3;
if (type < 2) {
int regenerated;
if (sizeFormat & 1 == 0) {
regenerated = header >> 3;
at += 1;
} else if (sizeFormat == 1) {
if (at + 2 > end) {
throw const ZstdException('truncated literals header');
}
regenerated = (header >> 4) | (src[at + 1] << 4);
at += 2;
} else {
if (at + 3 > end) {
throw const ZstdException('truncated literals header');
}
regenerated = (header >> 4) | (src[at + 1] << 4) | (src[at + 2] << 12);
at += 3;
}
if (regenerated > maxBlockSize) {
throw ZstdException('$regenerated literals is over the maximum');
}
if (type == 0) {
if (at + regenerated > end) {
throw const ZstdException('truncated raw literals');
}
// No copy: the literals are already contiguous in the frame.
_literalSource = src;
_literalAt = at;
_literalEnd = at + regenerated;
return at + regenerated;
}
if (at >= end) {
throw const ZstdException('truncated RLE literals');
}
_literals.fillRange(0, regenerated, src[at]);
_literalSource = _literals;
_literalAt = 0;
_literalEnd = regenerated;
return at + 1;
}
final int regenerated;
final int compressed;
final int streams;
switch (sizeFormat) {
case 0:
case 1:
if (at + 3 > end) {
throw const ZstdException('truncated literals header');
}
final value = src[at] | (src[at + 1] << 8) | (src[at + 2] << 16);
regenerated = (value >> 4) & 0x3FF;
compressed = (value >> 14) & 0x3FF;
streams = sizeFormat == 0 ? 1 : 4;
at += 3;
case 2:
if (at + 4 > end) {
throw const ZstdException('truncated literals header');
}
final value =
src[at] |
(src[at + 1] << 8) |
(src[at + 2] << 16) |
(src[at + 3] << 24);
regenerated = (value >> 4) & 0x3FFF;
compressed = (value >> 18) & 0x3FFF;
streams = 4;
at += 4;
default:
if (at + 5 > end) {
throw const ZstdException('truncated literals header');
}
final value =
src[at] |
(src[at + 1] << 8) |
(src[at + 2] << 16) |
(src[at + 3] << 24) |
(src[at + 4] << 32);
regenerated = (value >> 4) & 0x3FFFF;
compressed = (value >> 22) & 0x3FFFF;
streams = 4;
at += 5;
}
if (regenerated > maxBlockSize) {
throw ZstdException('$regenerated literals is over the maximum');
}
if (at + compressed > end) {
throw const ZstdException('truncated compressed literals');
}
if (_literals.length < regenerated) {
_literals = Uint8List(regenerated);
}
var streamStart = at;
final streamEnd = at + compressed;
if (type == 2) {
streamStart += _huffman.read(src, at, streamEnd, _bits);
}
if (streams == 1) {
_huffman.decodeStream(
_bits,
src,
streamStart,
streamEnd,
_literals,
0,
regenerated,
);
} else {
if (streamStart + 6 > streamEnd) {
throw const ZstdException('truncated literals jump table');
}
final one = src[streamStart] | (src[streamStart + 1] << 8);
final two = src[streamStart + 2] | (src[streamStart + 3] << 8);
final three = src[streamStart + 4] | (src[streamStart + 5] << 8);
var from = streamStart + 6;
final four = streamEnd - from - one - two - three;
if (four < 0) {
throw const ZstdException('literals jump table does not add up');
}
// The first three streams carry a quarter each, rounded up.
final segment = (regenerated + 3) >> 2;
final sizes = <int>[one, two, three, four];
var outAt = 0;
for (var i = 0; i < 4; i++) {
final count = i == 3 ? regenerated - 3 * segment : segment;
if (count < 0) {
throw const ZstdException('literals do not split into four');
}
if (count > 0) {
_huffman.decodeStream(
_bits,
src,
from,
from + sizes[i],
_literals,
outAt,
count,
);
}
from += sizes[i];
outAt += count;
}
}
_literalSource = _literals;
_literalAt = 0;
_literalEnd = regenerated;
return streamEnd;
}
int _copyRemainingLiterals(Uint8List out, int written) {
final count = _literalEnd - _literalAt;
if (written + count > out.length) {
throw const ZstdException('literals overflow the frame content size');
}
out.setRange(written, written + count, _literalSource, _literalAt);
_literalAt = _literalEnd;
return written + count;
}
int _executeSequences(
Uint8List src,
int start,
int end,
Uint8List out,
int written,
int sequences,
) {
final bits = _bits;
bits.reset(src, start, end);
final llSymbol = _literalLengths.symbol;
final llBits = _literalLengths.nbBits;
final llNext = _literalLengths.newState;
final mlSymbol = _matchLengths.symbol;
final mlBits = _matchLengths.nbBits;
final mlNext = _matchLengths.newState;
final ofSymbol = _offsets.symbol;
final ofBits = _offsets.nbBits;
final ofNext = _offsets.newState;
var llState = bits.read(_literalLengths.log);
var ofState = bits.read(_offsets.log);
var mlState = bits.read(_matchLengths.log);
final literals = _literalSource;
var literalAt = _literalAt;
final literalEnd = _literalEnd;
var at = written;
var repeat1 = _repeat1;
var repeat2 = _repeat2;
var repeat3 = _repeat3;
for (var n = 0; n < sequences; n++) {
// Every table's symbols were bounded when it was built, so a code out
// of range is impossible here and is not re-checked per sequence.
final llCode = llSymbol[llState];
final mlCode = mlSymbol[mlState];
final ofCode = ofSymbol[ofState];
// Additional bits come off the stream offset first, then match
// length, then literals length.
bits.fill();
final offsetValue = (1 << ofCode) + bits.read(ofCode);
final matchLength =
matchLengthBaseline[mlCode] + bits.read(matchLengthExtra[mlCode]);
final literalLength =
literalLengthBaseline[llCode] + bits.read(literalLengthExtra[llCode]);
if (n + 1 < sequences) {
// The three updates together need at most 26 bits, which one fill
// covers, so they peek rather than each testing for a refill.
bits.fill();
final llTake = llBits[llState];
llState = llNext[llState] + bits.peek(llTake);
bits.skip(llTake);
final mlTake = mlBits[mlState];
mlState = mlNext[mlState] + bits.peek(mlTake);
bits.skip(mlTake);
final ofTake = ofBits[ofState];
ofState = ofNext[ofState] + bits.peek(ofTake);
bits.skip(ofTake);
}
int offset;
if (offsetValue > 3) {
offset = offsetValue - 3;
repeat3 = repeat2;
repeat2 = repeat1;
repeat1 = offset;
} else {
// One to three is a reference to a recent offset, and a sequence
// with no literals shifts which one it means.
final which = offsetValue - 1 + (literalLength == 0 ? 1 : 0);
if (which == 0) {
offset = repeat1;
} else {
offset = which == 3 ? repeat1 - 1 : (which == 1 ? repeat2 : repeat3);
if (offset == 0) {
throw const ZstdException('repeated offset of zero');
}
if (which != 1) {
repeat3 = repeat2;
}
repeat2 = repeat1;
repeat1 = offset;
}
}
if (literalAt + literalLength > literalEnd) {
throw const ZstdException('sequence wants more literals than exist');
}
if (at + literalLength + matchLength > out.length) {
throw const ZstdException('sequences overflow the content size');
}
final from = at + literalLength - offset;
if (from < 0) {
throw const ZstdException('match reaches before the output');
}
if (literalLength > 0) {
if (literalLength < 16) {
for (var i = 0; i < literalLength; i++) {
out[at++] = literals[literalAt++];
}
} else {
out.setRange(at, at + literalLength, literals, literalAt);
at += literalLength;
literalAt += literalLength;
}
}
if (matchLength < 16 || offset < matchLength) {
// A short or overlapping match is quicker byte by byte than through
// a range copy, and the median match here is nine bytes.
var source = from;
for (var i = 0; i < matchLength; i++) {
out[at++] = out[source++];
}
} else {
out.setRange(at, at + matchLength, out, from);
at += matchLength;
}
}
if (bits.exhausted) {
throw const ZstdException('sequence stream ran past its end');
}
_literalAt = literalAt;
_repeat1 = repeat1;
_repeat2 = repeat2;
_repeat3 = repeat3;
return _copyRemainingLiterals(out, at);
}
}
/// Decodes one zstd frame into a new list.
///
/// Allocates a decoder per call. Decoding many frames, which is what reading
/// a tile does, should hold a [ZstdDecoder] instead.
Uint8List zstdDecode(Uint8List src, {int? expectedSize}) =>
ZstdDecoder().decode(src, expectedSize: expectedSize);

9
lib/src/exception.dart Normal file
View file

@ -0,0 +1,9 @@
/// Thrown when a frame is malformed, or uses a feature this decoder omits.
class ZstdException implements Exception {
const ZstdException(this.message);
final String message;
@override
String toString() => 'ZstdException: $message';
}

230
lib/src/fse.dart Normal file
View file

@ -0,0 +1,230 @@
import 'dart:typed_data';
import 'bit_reader.dart';
import 'exception.dart';
/// An FSE decoding table: one entry per state, giving the symbol that state
/// emits, how many bits to read next, and the base the read is added to.
///
/// The three arrays are parallel and sized for the largest accuracy log the
/// format allows, so a table is built in place and never reallocated.
class FseTable {
FseTable(this.maxLog)
: symbol = Uint8List(1 << maxLog),
nbBits = Uint8List(1 << maxLog),
newState = Uint16List(1 << maxLog),
_spread = Uint8List(1 << maxLog);
final int maxLog;
final Uint8List symbol;
final Uint8List nbBits;
final Uint16List newState;
/// Accuracy log. Zero for an RLE table, which has a single state.
int log = 0;
/// The normalised distribution this table was last built from. Written by
/// [readFseTable] and [buildPredefined], read by [build].
final Int16List distribution = Int16List(256);
final Uint8List _spread;
final Int32List _next = Int32List(256);
/// Makes this a table with one state emitting [value] and reading no bits.
void setRle(int value) {
log = 0;
symbol[0] = value;
nbBits[0] = 0;
newState[0] = 0;
}
/// Builds the table from [distribution], where a count of -1 is the "less
/// than one in the table" symbol the format reserves.
void build(int maxSymbol, int accuracyLog) {
if (accuracyLog > maxLog) {
throw ZstdException('FSE accuracy log $accuracyLog above $maxLog');
}
final size = 1 << accuracyLog;
log = accuracyLog;
// Low-probability symbols go at the top of the table, working down, and
// then count as one when the states are numbered.
var highThreshold = size - 1;
for (var s = 0; s <= maxSymbol; s++) {
final count = distribution[s];
if (count == -1) {
_spread[highThreshold--] = s;
_next[s] = 1;
} else {
_next[s] = count;
}
}
// The step is coprime with the table size, so walking it visits every
// slot exactly once. Skipping the low-probability slots at the top is
// what keeps those symbols where they were just put.
final mask = size - 1;
final step = (size >> 1) + (size >> 3) + 3;
var position = 0;
for (var s = 0; s <= maxSymbol; s++) {
final count = distribution[s];
for (var i = 0; i < count; i++) {
_spread[position] = s;
position = (position + step) & mask;
while (position > highThreshold) {
position = (position + step) & mask;
}
}
}
if (position != 0) {
throw const ZstdException('FSE distribution does not fill its table');
}
for (var state = 0; state < size; state++) {
final s = _spread[state];
final nth = _next[s]++;
final bits = accuracyLog - (nth.bitLength - 1);
symbol[state] = s;
nbBits[state] = bits;
newState[state] = (nth << bits) - size;
}
}
}
/// Reads an FSE table description into [table.distribution] and builds it.
///
/// Returns the number of bytes the description occupied. The description is a
/// forward, least-significant-bit-first bitstream: an accuracy log, then one
/// value per symbol coded in as few bits as the remaining probability mass
/// allows, with a repeat flag after any zero.
int readFseTable(
FseTable table,
Uint8List src,
int start,
int end,
int maxSymbol,
) {
final bits = ForwardBitReader(src, end, start);
final counts = table.distribution;
for (var s = 0; s <= maxSymbol; s++) {
counts[s] = 0;
}
final accuracyLog = (bits.peek32() & 0xF) + 5;
bits.skip(4);
if (accuracyLog > table.maxLog) {
throw ZstdException('FSE accuracy log $accuracyLog above ${table.maxLog}');
}
var remaining = (1 << accuracyLog) + 1;
var threshold = 1 << accuracyLog;
var nbBits = accuracyLog + 1;
var symbol = 0;
var previousZero = false;
while (remaining > 1 && symbol <= maxSymbol) {
if (previousZero) {
var repeat = 0;
while (true) {
final flag = bits.peek32() & 3;
bits.skip(2);
repeat += flag;
if (flag != 3) {
break;
}
}
while (repeat-- > 0) {
if (symbol > maxSymbol) {
throw const ZstdException('FSE zero run past the last symbol');
}
counts[symbol++] = 0;
}
previousZero = false;
continue;
}
final value = bits.peek32();
final small = (2 * threshold - 1) - remaining;
int count;
if ((value & (threshold - 1)) < small) {
count = value & (threshold - 1);
bits.skip(nbBits - 1);
} else {
count = value & (2 * threshold - 1);
if (count >= threshold) {
count -= small;
}
bits.skip(nbBits);
}
count -= 1;
remaining -= count < 0 ? -count : count;
counts[symbol++] = count;
previousZero = count == 0;
while (remaining < threshold) {
nbBits--;
threshold >>= 1;
}
}
if (remaining != 1) {
throw const ZstdException('FSE distribution is short');
}
table.build(symbol - 1, accuracyLog);
return bits.bytesUsed(start);
}
/// Builds [table] from one of the format's fixed distributions.
void buildPredefined(FseTable table, List<int> values, int accuracyLog) {
final counts = table.distribution;
for (var s = 0; s < values.length; s++) {
counts[s] = values[s];
}
table.build(values.length - 1, accuracyLog);
}
/// Decodes a whole FSE bitstream with the two interleaved states the format
/// uses, appending symbols to [out] from [outStart] and returning the count.
///
/// The symbol count is not recorded anywhere, so decoding runs until the
/// stream is spent. This is how the Huffman weights are coded.
int fseDecodeInterleaved(
FseTable table,
ReverseBitReader bits,
Uint8List out,
int outStart,
int outLimit,
) {
final symbol = table.symbol;
final nbBits = table.nbBits;
final newState = table.newState;
final log = table.log;
var state1 = bits.read(log);
var state2 = bits.read(log);
var at = outStart;
while (true) {
if (at + 2 > outLimit) {
throw const ZstdException('FSE stream longer than its output allows');
}
out[at++] = symbol[state1];
bits.fill();
state1 = newState[state1] + bits.read(nbBits[state1]);
if (bits.exhausted) {
out[at++] = symbol[state2];
break;
}
out[at++] = symbol[state2];
bits.fill();
state2 = newState[state2] + bits.read(nbBits[state2]);
if (bits.exhausted) {
out[at++] = symbol[state1];
break;
}
}
return at - outStart;
}

201
lib/src/huffman.dart Normal file
View file

@ -0,0 +1,201 @@
import 'dart:typed_data';
import 'bit_reader.dart';
import 'exception.dart';
import 'fse.dart';
/// The Huffman table for a block's literals, flattened so that decoding one
/// literal is a peek of [log] bits and one array index.
///
/// Weights, not code lengths, are what the format stores: a symbol of weight
/// `w` occupies `1 << (w - 1)` of the table's slots, and the last symbol's
/// weight is left out and recovered from the slack.
class HuffmanTable {
/// The largest table log the format allows for literals.
static const int maxLog = 12;
static const int _maxSymbols = 256;
/// One entry per table slot: the symbol in the low byte, the code length
/// in the next. One load per literal rather than two.
final Uint16List entries = Uint16List(1 << maxLog);
/// Bits to peek per literal. Zero until a table has been read.
int log = 0;
final Uint8List _weights = Uint8List(_maxSymbols);
final Int32List _rankCount = Int32List(maxLog + 2);
final Int32List _rankStart = Int32List(maxLog + 2);
/// The FSE table the weights are coded with, when they are coded at all.
final FseTable _weightTable = FseTable(6);
/// Reads a Huffman tree description starting at [start], returning the
/// number of bytes it occupied.
int read(Uint8List src, int start, int end, ReverseBitReader bitReader) {
if (start >= end) {
throw const ZstdException('literals section has no tree');
}
final header = src[start];
int nbWeights;
int used;
if (header >= 128) {
// Directly encoded: one nibble per weight, high nibble first.
nbWeights = header - 127;
final packed = (nbWeights + 1) >> 1;
if (start + 1 + packed > end) {
throw const ZstdException('truncated Huffman weights');
}
for (var i = 0; i < nbWeights; i += 2) {
final byte = src[start + 1 + (i >> 1)];
_weights[i] = byte >> 4;
if (i + 1 < nbWeights) {
_weights[i + 1] = byte & 0xF;
}
}
used = 1 + packed;
} else {
final size = header;
if (start + 1 + size > end) {
throw const ZstdException('truncated Huffman weight stream');
}
final read = readFseTable(
_weightTable,
src,
start + 1,
start + 1 + size,
255,
);
bitReader.reset(src, start + 1 + read, start + 1 + size);
nbWeights = fseDecodeInterleaved(
_weightTable,
bitReader,
_weights,
0,
_maxSymbols,
);
used = 1 + size;
}
_buildFromWeights(nbWeights);
return used;
}
void _buildFromWeights(int nbWeights) {
if (nbWeights < 1 || nbWeights >= _maxSymbols) {
throw ZstdException('Huffman table with $nbWeights weights');
}
var total = 0;
for (var i = 0; i < nbWeights; i++) {
final weight = _weights[i];
if (weight > maxLog) {
throw ZstdException('Huffman weight $weight out of range');
}
if (weight > 0) {
total += 1 << (weight - 1);
}
}
if (total == 0) {
throw const ZstdException('Huffman table with no used symbol');
}
final tableLog = total.bitLength;
if (tableLog > maxLog) {
throw ZstdException('Huffman table log $tableLog above $maxLog');
}
final size = 1 << tableLog;
final rest = size - total;
if (rest & (rest - 1) != 0) {
throw const ZstdException('Huffman weights do not complete a tree');
}
// The weight left out of the description is whatever fills the slack.
_weights[nbWeights] = rest.bitLength;
final nbSymbols = nbWeights + 1;
log = tableLog;
for (var n = 0; n < _rankCount.length; n++) {
_rankCount[n] = 0;
}
for (var s = 0; s < nbSymbols; s++) {
final weight = _weights[s];
if (weight > 0) {
_rankCount[tableLog + 1 - weight]++;
}
}
// Codes are handed out from the lowest weight up, so the longest codes
// take the lowest table indices and the shortest take the highest.
var start = 0;
for (var n = tableLog; n >= 1; n--) {
_rankStart[n] = start;
start += _rankCount[n] << (tableLog - n);
}
if (start != size) {
throw const ZstdException('Huffman ranks do not fill the table');
}
// Within one code length, symbols keep their natural order.
for (var s = 0; s < nbSymbols; s++) {
final weight = _weights[s];
if (weight == 0) {
continue;
}
final length = tableLog + 1 - weight;
final span = 1 << (weight - 1);
var at = _rankStart[length];
_rankStart[length] = at + span;
final entry = s | (length << 8);
for (var i = 0; i < span; i++) {
entries[at++] = entry;
}
}
}
/// Decodes [count] literals from the reverse bitstream in [src] between
/// [start] and [end] into [out] at [outStart].
void decodeStream(
ReverseBitReader reader,
Uint8List src,
int start,
int end,
Uint8List out,
int outStart,
int count,
) {
if (log == 0) {
throw const ZstdException('literals coded with no Huffman table');
}
reader.reset(src, start, end);
final table = entries;
final peekBits = log;
var at = outStart;
final stop = outStart + count;
// One fill leaves at least 49 bits, which covers four literals at the
// format's twelve-bit maximum, so the inner steps are a peek and an
// index with nothing between them.
final groups = stop - 4;
while (at <= groups) {
reader.fill();
var entry = table[reader.peek(peekBits)];
out[at++] = entry & 0xFF;
reader.skip(entry >> 8);
entry = table[reader.peek(peekBits)];
out[at++] = entry & 0xFF;
reader.skip(entry >> 8);
entry = table[reader.peek(peekBits)];
out[at++] = entry & 0xFF;
reader.skip(entry >> 8);
entry = table[reader.peek(peekBits)];
out[at++] = entry & 0xFF;
reader.skip(entry >> 8);
}
while (at < stop) {
reader.fill();
final entry = table[reader.peek(peekBits)];
out[at++] = entry & 0xFF;
reader.skip(entry >> 8);
}
if (reader.exhausted) {
throw const ZstdException('Huffman stream ran past its end');
}
}
}

67
lib/src/tables.dart Normal file
View file

@ -0,0 +1,67 @@
/// The fixed tables RFC 8878 defines for sequence decoding.
library;
import 'dart:typed_data';
/// Literals length code to the smallest length it can mean.
final Int32List literalLengthBaseline = Int32List.fromList(const <int>[
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, //
16, 18, 20, 22, 24, 28, 32, 40, //
48, 64, 128, 256, 512, 1024, 2048, 4096, //
8192, 16384, 32768, 65536,
]);
/// Extra bits read after a literals length code.
final Uint8List literalLengthExtra = Uint8List.fromList(const <int>[
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, //
1, 1, 1, 1, 2, 2, 3, 3, //
4, 6, 7, 8, 9, 10, 11, 12, //
13, 14, 15, 16,
]);
/// Match length code to the smallest length it can mean.
final Int32List matchLengthBaseline = Int32List.fromList(const <int>[
3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, //
19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, //
35, 37, 39, 41, 43, 47, 51, 59, //
67, 83, 99, 131, 259, 515, 1027, 2051, //
4099, 8195, 16387, 32771, 65539,
]);
/// Extra bits read after a match length code.
final Uint8List matchLengthExtra = Uint8List.fromList(const <int>[
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, //
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, //
1, 1, 1, 1, 2, 2, 3, 3, //
4, 4, 5, 7, 8, 9, 10, 11, //
12, 13, 14, 15, 16,
]);
/// The default literals length distribution, accuracy log 6.
const List<int> literalLengthDefault = <int>[
4, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, //
2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 2, 1, 1, 1, 1, 1, //
-1, -1, -1, -1,
];
/// The default match length distribution, accuracy log 6.
const List<int> matchLengthDefault = <int>[
1, 4, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, //
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, //
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -1, -1, //
-1, -1, -1, -1, -1,
];
/// The default offset code distribution, accuracy log 5.
const List<int> offsetCodeDefault = <int>[
1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, //
1, 1, 1, 1, 1, 1, 1, 1, -1, -1, -1, -1, -1,
];
const int literalLengthMaxLog = 9;
const int matchLengthMaxLog = 9;
const int offsetCodeMaxLog = 8;
const int literalLengthMaxSymbol = 35;
const int matchLengthMaxSymbol = 52;
const int offsetCodeMaxSymbol = 31;

11
pubspec.yaml Normal file
View file

@ -0,0 +1,11 @@
name: pure_zstd
description: A pure Dart zstd decompressor, with no native dependency.
version: 0.0.1
publish_to: none
environment:
sdk: ^3.8.0
dev_dependencies:
lints: ^6.0.0
test: ^1.25.0

BIN
test/fixtures/N51E000.hgtz vendored Normal file
View file

Binary file not shown.

63
test/hgtz_block_test.dart Normal file
View file

@ -0,0 +1,63 @@
import 'dart:io';
import 'dart:typed_data';
import 'package:pure_zstd/pure_zstd.dart';
import 'package:test/test.dart';
/// The committed container was written by `hgtz.py`, so it is a
/// real level-19 pyzstd frame rather than anything this package produced.
///
/// Every sample in it is `(northRow * 1000 + column) % 30000`, which makes the
/// expected bytes of each block computable without a reference decompressor.
void main() {
group('a real .hgtz container', () {
final bytes = File('test/fixtures/N51E000.hgtz').readAsBytesSync();
final header = ByteData.sublistView(bytes);
final rows = header.getUint16(6);
final columns = header.getUint16(8);
final block = header.getUint16(10);
final blockColumns = (columns + block - 1) ~/ block;
final blockRows = (rows + block - 1) ~/ block;
test('has the shape its README documents', () {
expect(rows, 1201);
expect(columns, 121);
expect(block, 256);
expect(blockRows * blockColumns, 5);
});
test('decodes every block to the samples the generator wrote', () {
final decoder = ZstdDecoder();
for (var index = 0; index < blockRows * blockColumns; index++) {
final entry = 12 + 8 * index;
final offset = header.getUint32(entry);
final length = header.getUint32(entry + 4);
final blockRow = index ~/ blockColumns;
final blockColumn = index % blockColumns;
final top = blockRow * block;
final left = blockColumn * block;
final height = block < rows - top ? block : rows - top;
final width = block < columns - left ? block : columns - left;
final frame = Uint8List.sublistView(bytes, offset, offset + length);
final out = decoder.decode(frame);
expect(out.length, height * width * 2, reason: 'block $index size');
final samples = ByteData.sublistView(out);
for (var row = 0; row < height; row++) {
for (var column = 0; column < width; column++) {
// Row 0 of the container is the north edge, which is what the
// generator counted from.
final northRow = top + row;
final expected = (northRow * 1000 + left + column) % 30000;
expect(
samples.getInt16((row * width + column) * 2),
expected,
reason: 'block $index at $row,$column',
);
}
}
}
});
});
}

103
tool/oracle.py Executable file
View file

@ -0,0 +1,103 @@
#!/usr/bin/env python3
"""Check the Dart decoder against pyzstd over real .hgtz containers.
./tool/oracle.py path/to/srtm1z/N39W105.hgtz
./tool/oracle.py --every 1500 --limit 8 path/to/srtm1z
pyzstd is the reference. Every block of every container named is extracted as
its own frame, decompressed by pyzstd, and handed to tool/verify.dart, which
decodes it again and compares byte for byte. One tile at a time, through a
temporary directory that is deleted as it goes, because a whole tile's blocks
decompress to around 29 MB.
"""
from __future__ import annotations
import argparse
import struct
import subprocess
import sys
import tempfile
from pathlib import Path
import pyzstd
HEADER = ">4sBbHHH"
ENTRY = ">II"
def frames(path: Path):
"""Yields (index, frame bytes, expected decompressed size) per block."""
data = path.read_bytes()
magic, version, _level, rows, cols, block = struct.unpack_from(HEADER, data)
if magic != b"HGTZ" or version != 1:
raise SystemExit(f"{path}: not a version 1 .hgtz")
block_cols = (cols + block - 1) // block
block_rows = (rows + block - 1) // block
for index in range(block_rows * block_cols):
offset, length = struct.unpack_from(ENTRY, data, 12 + 8 * index)
top = (index // block_cols) * block
left = (index % block_cols) * block
height = min(block, rows - top)
width = min(block, cols - left)
yield index, data[offset : offset + length], height * width * 2
def check(path: Path, root: Path) -> bool:
with tempfile.TemporaryDirectory() as name:
directory = Path(name)
count = 0
for index, frame, expected in frames(path):
plain = pyzstd.decompress(frame)
if len(plain) != expected:
raise SystemExit(
f"{path} block {index}: pyzstd gave {len(plain)},"
f" the index implies {expected}"
)
(directory / f"{index:05d}.zst").write_bytes(frame)
(directory / f"{index:05d}.raw").write_bytes(plain)
count += 1
result = subprocess.run(
["dart", "run", "tool/verify.dart", str(directory)],
cwd=root,
capture_output=True,
text=True,
)
line = result.stdout.strip().splitlines()[-1] if result.stdout else ""
print(f"{path.name}: {count} blocks, {line or result.stderr.strip()}")
return result.returncode == 0
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("paths", nargs="+", type=Path)
parser.add_argument(
"--every",
type=int,
default=1,
help="when a directory is named, take every Nth container",
)
parser.add_argument("--limit", type=int, default=0)
arguments = parser.parse_args()
containers: list[Path] = []
for path in arguments.paths:
if path.is_dir():
found = sorted(path.glob("*.hgtz"))
containers += found[:: arguments.every]
else:
containers.append(path)
if arguments.limit:
containers = containers[: arguments.limit]
root = Path(__file__).resolve().parent.parent
failures = [c for c in containers if not check(c, root)]
if failures:
print(f"FAILED on {len(failures)} of {len(containers)} containers")
return 1
print(f"all {len(containers)} containers match pyzstd byte for byte")
return 0
if __name__ == "__main__":
sys.exit(main())

52
tool/verify.dart Normal file
View file

@ -0,0 +1,52 @@
import 'dart:io';
import 'dart:typed_data';
import 'package:pure_zstd/pure_zstd.dart';
/// Decodes every `*.zst` in a directory and compares it to the `*.raw`
/// beside it. Driven by tool/oracle.py, which writes the pair.
void main(List<String> arguments) {
if (arguments.length != 1) {
stderr.writeln('usage: verify.dart <directory>');
exit(2);
}
final directory = Directory(arguments.single);
final frames =
directory
.listSync()
.whereType<File>()
.where((file) => file.path.endsWith('.zst'))
.toList()
..sort((a, b) => a.path.compareTo(b.path));
final decoder = ZstdDecoder();
var checked = 0;
var bytes = 0;
for (final frame in frames) {
final expected = File(
frame.path.replaceRange(frame.path.length - 4, null, '.raw'),
).readAsBytesSync();
final Uint8List actual;
try {
actual = decoder.decode(frame.readAsBytesSync());
} on ZstdException catch (error) {
stderr.writeln('${frame.path}: $error');
exit(1);
}
if (actual.length != expected.length) {
stderr.writeln(
'${frame.path}: ${actual.length} bytes, expected ${expected.length}',
);
exit(1);
}
for (var i = 0; i < actual.length; i++) {
if (actual[i] != expected[i]) {
stderr.writeln('${frame.path}: byte $i differs');
exit(1);
}
}
checked++;
bytes += actual.length;
}
stdout.writeln('$checked frames, $bytes bytes, identical');
}