halkyon-days/pure-zstd
Watch
1
0
Fork
You've already forked pure-zstd
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
pure-zstd/README.md

172 lines
8.1 KiB
Markdown
Raw Normal View History

Add a pure-Dart zstd decoder, as a benchmark A feasibility spike, standalone so it can move to its own repository. Not wired into the app and not a dependency of it. Decompression only, and only what our containers need: no dictionaries, no multi-frame, no streaming, and the content checksum is parsed but never verified. Native only, since the bit reader carries 56 bits in an int. It is correct on real data. The committed fixture decodes to the samples the generator wrote, and tool/oracle.py drives a byte-for-byte comparison against pyzstd over 32 containers and 2,615 blocks from both tiers, every thinned column count from 3601 to 361, and 88 S to 80 N. All identical. The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The cost is the sequence loop and nothing else: a terrain block at level 19 carries around 1,700 literals and 14,400 sequences, so optimising the literal path bought nothing.
2026-09-21 21:37:56 -06:00
# pure_zstd
Add README badges
2026-09-22 19:48:27 -06:00
[![CI](https://github.com/tagadvance/pure-zstd/actions/workflows/ci.yml/badge.svg)](https://github.com/tagadvance/pure-zstd/actions/workflows/ci.yml)
[![pub package](https://img.shields.io/pub/v/pure_zstd.svg)](https://pub.dev/packages/pure_zstd)
[![pub points](https://img.shields.io/pub/points/pure_zstd)](https://pub.dev/packages/pure_zstd/score)
[![Dart](https://img.shields.io/badge/dart-%3E%3D3.8-blue)](https://dart.dev)
[![License: MIT](https://img.shields.io/github/license/tagadvance/pure-zstd)](LICENSE)
Add a pure-Dart zstd decoder, as a benchmark A feasibility spike, standalone so it can move to its own repository. Not wired into the app and not a dependency of it. Decompression only, and only what our containers need: no dictionaries, no multi-frame, no streaming, and the content checksum is parsed but never verified. Native only, since the bit reader carries 56 bits in an int. It is correct on real data. The committed fixture decodes to the samples the generator wrote, and tool/oracle.py drives a byte-for-byte comparison against pyzstd over 32 containers and 2,615 blocks from both tiers, every thinned column count from 3601 to 361, and 88 S to 80 N. All identical. The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The cost is the sequence loop and nothing else: a terrain block at level 19 carries around 1,700 literals and 14,400 sequences, so optimising the literal path bought nothing.
2026-09-21 21:37:56 -06:00
A zstd decompressor in plain Dart, with no native dependency.
It exists because every zstd binding on pub.dev is a wrapper over libzstd, and
an app that reads block-compressed elevation tiles on a phone would rather not
carry a native library per platform to do it. Decompression only. There is no
compressor and there is no plan for one.
Licence under MIT, and name what this was built from MIT rather than a dual with BSD-3. Dual licensing solves a compatibility problem this does not have, since both are permissive and compatible with everything, so the choice would cost a downstream reader a decision and buy them nothing. The debt to zstd is real and a licence is the wrong instrument for it. No code came from libzstd; the format came from RFC 8878 and every answer is checked against libzstd through pyzstd. That belongs in prose where people read it. Also drops the line calling this a feasibility spike. It has 103 tests, an audit against the RFC, and five defects found and closed.
2026-09-21 23:41:31 -06:00
It decodes the real data it was written for, correctly and completely, and it
is roughly four times slower than native zstd. See **Speed** below before
depending on it.
Add a pure-Dart zstd decoder, as a benchmark A feasibility spike, standalone so it can move to its own repository. Not wired into the app and not a dependency of it. Decompression only, and only what our containers need: no dictionaries, no multi-frame, no streaming, and the content checksum is parsed but never verified. Native only, since the bit reader carries 56 bits in an int. It is correct on real data. The committed fixture decodes to the samples the generator wrote, and tool/oracle.py drives a byte-for-byte comparison against pyzstd over 32 containers and 2,615 blocks from both tiers, every thinned column count from 3601 to 361, and 88 S to 80 N. All identical. The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The cost is the sequence loop and nothing else: a terrain block at level 19 carries around 1,700 literals and 14,400 sequences, so optimising the literal path bought nothing.
2026-09-21 21:37:56 -06:00
## Using it
```dart
import 'package:pure_zstd/pure_zstd.dart';
final plain = zstdDecode(frame);
// Decoding many frames: hold the decoder, which keeps its entropy tables and
// scratch buffers between calls and then allocates only the output.
final decoder = ZstdDecoder();
for (final frame in frames) {
use(decoder.decode(frame));
}
```
Decode frames that declare no content size Streaming encoders leave Frame_Content_Size out unless told the size up front, and their readers often can't know it either: every zstd frame in an Anki package (its collection, its media list, each media file) is written that way. Such a frame needed the exact size passed as expectedSize, and anything else was refused. Now it gets the most its blocks can hold, Block_Maximum_Size (128 KiB) each, found by walking the block headers alone, and the output is trimmed to what the blocks held. The output is still allocated once. expectedSize keeps its exact check when given, and maxOutputSize bounds the allocation as before. Checked against libzstd (python-zstandard) on Anki 26.09.3's exports: a small package's four frames and a 30 MB collection decode byte for byte the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 04:28:06 +02:00
The output is allocated whole rather than grown. A frame that declares no
content size, as streaming encoders write them, gets the most its blocks can
hold (128 KiB each, found from their headers alone) and is trimmed to what
they held; `expectedSize` turns that into an exact check when the caller
knows the size.
Add a pure-Dart zstd decoder, as a benchmark A feasibility spike, standalone so it can move to its own repository. Not wired into the app and not a dependency of it. Decompression only, and only what our containers need: no dictionaries, no multi-frame, no streaming, and the content checksum is parsed but never verified. Native only, since the bit reader carries 56 bits in an int. It is correct on real data. The committed fixture decodes to the samples the generator wrote, and tool/oracle.py drives a byte-for-byte comparison against pyzstd over 32 containers and 2,615 blocks from both tiers, every thinned column count from 3601 to 361, and 88 S to 80 N. All identical. The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The cost is the sequence loop and nothing else: a terrain block at level 19 carries around 1,700 literals and 14,400 sequences, so optimising the literal path bought nothing.
2026-09-21 21:37:56 -06:00
## What it implements
RFC 8878 is the specification. Of it:
| Part of the format | State |
| --- | --- |
| Frame header, single segment and windowed | Decoded. The window descriptor is skipped, since the output is allocated from the content size. |
| Raw, RLE and compressed blocks | All three. |
| Multi-block frames | Yes, including a later block reusing an earlier one's tables and recent offsets. |
| Raw, RLE, compressed and treeless literals | All four, in one-stream and four-stream form. |
| Huffman weights | Both the direct nibble form and the FSE-coded form. |
| Sequence tables | Predefined, RLE, FSE-compressed and repeat, for all three of literal length, match length and offset. |
| Repeated offsets | Yes, including the no-literals shift. |
Close the last of the five defects
2026-09-21 23:12:46 -06:00
| Content checksum | Verified. XXH64 over the output, compared with the stored low 32 bits. Hashed only when the frame carries one. |
Add a pure-Dart zstd decoder, as a benchmark A feasibility spike, standalone so it can move to its own repository. Not wired into the app and not a dependency of it. Decompression only, and only what our containers need: no dictionaries, no multi-frame, no streaming, and the content checksum is parsed but never verified. Native only, since the bit reader carries 56 bits in an int. It is correct on real data. The committed fixture decodes to the samples the generator wrote, and tool/oracle.py drives a byte-for-byte comparison against pyzstd over 32 containers and 2,615 blocks from both tiers, every thinned column count from 3601 to 361, and 88 S to 80 N. All identical. The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The cost is the sequence loop and nothing else: a terrain block at level 19 carries around 1,700 literals and 14,400 sequences, so optimising the literal path bought nothing.
2026-09-21 21:37:56 -06:00
| Dictionaries | Refused. |
Mark four of the five defects closed Only the unverified checksum is left, and it is narrower here than it looks: the downloader verifies a SHA-256 over the whole container before it renames the .part, so a tile damaged in transit is already caught. What remains is corruption at rest on the phone.
2026-09-21 23:09:56 -06:00
| Skippable frames, several frames in one buffer | Refused. Reading one frame is the whole API, and input left over after it is an error. |
Add a pure-Dart zstd decoder, as a benchmark A feasibility spike, standalone so it can move to its own repository. Not wired into the app and not a dependency of it. Decompression only, and only what our containers need: no dictionaries, no multi-frame, no streaming, and the content checksum is parsed but never verified. Native only, since the bit reader carries 56 bits in an int. It is correct on real data. The committed fixture decodes to the samples the generator wrote, and tool/oracle.py drives a byte-for-byte comparison against pyzstd over 32 containers and 2,615 blocks from both tiers, every thinned column count from 3601 to 361, and 88 S to 80 N. All identical. The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The cost is the sequence loop and nothing else: a terrain block at level 19 carries around 1,700 literals and 14,400 sequences, so optimising the literal path bought nothing.
2026-09-21 21:37:56 -06:00
| Streaming, or decoding part of a frame | Not offered. |
It targets native Dart and not the web: the literals header reads a five-byte
field with a 32-bit shift, and the bit reader carries 56 bits in an `int`.
Both are exact on the VM and on AOT, and neither is on JavaScript.
## Correctness
The reference is pyzstd, which wraps libzstd. `tool/oracle.py` extracts every
block of a `.hgtz` container as its own frame, decompresses it with pyzstd,
hands it to `tool/verify.dart`, and compares byte for byte.
Bring the decoder over from the project it grew in Eighteen commits of history, grafted onto the scaffold rather than over it. It leaves behind everything that only meant something inside that project: paths under one machine's mount, references to a transcoder that is not here, and a benchmark defaulting to a tile nobody else has. One fixture is real terrain and stays, because it is the only small input that exercises the direct-nibble Huffman weight form. Its licence requires attribution to travel with it, so the README's acknowledgements and the corpus manifest both carry the Copernicus notice now. Everything else committed here is synthetic or was written by the zstd CLI, including the .hgtz container, whose name looks like a tile and whose samples are a function of their position.
2026-09-22 19:25:08 -06:00
./tool/oracle.py <a .hgtz container>
./tool/oracle.py --every 1700 <a directory of them>
Add a pure-Dart zstd decoder, as a benchmark A feasibility spike, standalone so it can move to its own repository. Not wired into the app and not a dependency of it. Decompression only, and only what our containers need: no dictionaries, no multi-frame, no streaming, and the content checksum is parsed but never verified. Native only, since the bit reader carries 56 bits in an int. It is correct on real data. The committed fixture decodes to the samples the generator wrote, and tool/oracle.py drives a byte-for-byte comparison against pyzstd over 32 containers and 2,615 blocks from both tiers, every thinned column count from 3601 to 361, and 88 S to 80 N. All identical. The numbers, AOT on this box, 131,072 bytes a block: 0.66 ms median against pyzstd's 0.175, so about 200 MB/s against 750. Across three tiles the range is 0.66 to 0.94 ms, and a 113 block sweep is 67 to 105 ms. The cost is the sequence loop and nothing else: a terrain block at level 19 carries around 1,700 literals and 14,400 sequences, so optimising the literal path bought nothing.
2026-09-21 21:37:56 -06:00
As of 2026-09-21 that is 32 real containers, 2,615 blocks and 0.3 GB of
output, all identical, spanning both the one and three arc-second sets, every
thinned column count from 3601 down to 361, and latitudes from 88 S to 80 N.
`dart test` needs no Python. It decodes the five blocks of the container in
`test/fixtures/`, whose every sample is a known function of its position, so a
wrong decode cannot pass.
## Speed
Measured 2026-09-21 on this eight-core Linux box, `dart compile exe`, Dart
3.12.2. One 256 by 256 block is 131,072 bytes out. The frames are read into
memory first, so only decoding is timed, and the figure includes allocating
the output buffer because the API allocates it.
| Block set | pure_zstd, AOT | pyzstd (libzstd) |
| --- | --- | --- |
| Colorado, `N39W105`, 196 blocks, median | **0.66 ms** | 0.18 ms |
| Alaska, `N60W160`, 98 blocks, median | 0.94 ms | not measured |
| One block, repeated 1000 times | 0.85 ms | 0.22 ms |
| 113 blocks, a 360 degree sweep | 67 ms | 19 ms |
That is 140 to 200 MB/s of output. The JIT is about 25% slower than AOT, so
measure with `dart compile exe`.
The cost is almost all in the sequence loop rather than in Huffman literals.
A block of real terrain carries around 1,700 literals and 14,400 sequences,
so the hot path is three FSE state updates, six bit reads and a nine-byte
match copy, about 46 ns per sequence against native's dozen or so. The
headroom left is bounds checks and the byte-at-a-time match copy; widening
that copy to 64-bit words, the way libzstd does, is the next thing to try.
Run it yourself:
dart compile exe benchmark/bench.dart -o /tmp/bench && /tmp/bench
Licence under MIT, and name what this was built from MIT rather than a dual with BSD-3. Dual licensing solves a compatibility problem this does not have, since both are permissive and compatible with everything, so the choice would cost a downstream reader a decision and buy them nothing. The debt to zstd is real and a licence is the wrong instrument for it. No code came from libzstd; the format came from RFC 8878 and every answer is checked against libzstd through pyzstd. That belongs in prose where people read it. Also drops the line calling this a feasibility spike. It has 103 tests, an audit against the RFC, and five defects found and closed.
2026-09-21 23:41:31 -06:00
## Acknowledgements
This is an independent implementation. It contains no code from libzstd, and
nothing about its licence reaches this package. It would not exist without the
following, and they are named here because that is where the debt actually is.
**[RFC 8878](https://www.rfc-editor.org/rfc/rfc8878.html), "Zstandard
Compression and the application/zstd Media Type"**, by Yann Collet and Murray
Kucherawy, is the specification this was written from. Every table in
`lib/src/tables.dart` was checked against section 3.1.1.3.2 entry by entry, and
the FSE and Huffman decoders follow the reference algorithms it describes. A
format specification written clearly enough to implement from is a gift, and
this one is.
**[Zstandard](https://github.com/facebook/zstd)**, by Yann Collet and
contributors at Meta, is the format and the reference implementation. Its
licence is BSD-3-Clause, and it obliges nothing here; it is named because
without it there would be nothing to decode.
**[pyzstd](https://pypi.org/project/pyzstd/)** wraps libzstd and is the oracle
every test in this package is measured against. Correctness here means
"agrees with libzstd, byte for byte", and pyzstd is how that is asked. The
committed fixtures under `test/fixtures/corpus/` are output of the `zstd` CLI
and of pyzstd, which is tool output rather than anything derived from their
source.
Bring the decoder over from the project it grew in Eighteen commits of history, grafted onto the scaffold rather than over it. It leaves behind everything that only meant something inside that project: paths under one machine's mount, references to a transcoder that is not here, and a benchmark defaulting to a tile nobody else has. One fixture is real terrain and stays, because it is the only small input that exercises the direct-nibble Huffman weight form. Its licence requires attribution to travel with it, so the README's acknowledgements and the corpus manifest both carry the Copernicus notice now. Everything else committed here is synthetic or was written by the zstd CLI, including the .hgtz container, whose name looks like a tile and whose samples are a function of their position.
2026-09-22 19:25:08 -06:00
**The Copernicus Programme.** One committed fixture,
`test/fixtures/corpus/direct_nibble.zst`, is a block of real terrain, kept
because it is the only small input that exercises the direct-nibble Huffman
weight form. Its licence requires this to travel with it:
> Produced using Copernicus WorldDEM-30 © DLR e.V. 2010-2014 and © Airbus Defence and Space GmbH 2014-2018 provided under COPERNICUS by the European Union and ESA; all rights reserved.
Licence under MIT, and name what this was built from MIT rather than a dual with BSD-3. Dual licensing solves a compatibility problem this does not have, since both are permissive and compatible with everything, so the choice would cost a downstream reader a decision and buy them nothing. The debt to zstd is real and a licence is the wrong instrument for it. No code came from libzstd; the format came from RFC 8878 and every answer is checked against libzstd through pyzstd. That belongs in prose where people read it. Also drops the line calling this a feasibility spike. It has 103 tests, an audit against the RFC, and five defects found and closed.
2026-09-21 23:41:31 -06:00
**The Dart team**, for making `int` an exact 64-bit integer on the VM and on
AOT. The bit reader carries 56 bits in one and XXH64's multiplies wrap exactly,
neither of which would be true on a platform with doubles underneath.
Measure the decoder on the phone, which is the number that decides it 0.414 ms a block in profile AOT on the Pixel, so a 113-block sweep is 47 ms. The phone is about twice as quick at this as the laptop the first figures came from, which read 0.917 ms and 104 ms for the same sixteen blocks. Debug on device is 0.540 ms, so the shipped configuration is the fast one rather than the slow one. The blocks are real, out of N39W105 in srtm1z. The committed .hgtz fixture is no use for timing: its samples are a synthetic pattern that compresses to almost no sequences, and decode cost tracks sequence count, so it reads about six times too fast. It was measured that way once before this was noticed.
2026-09-21 21:53:21 -06:00
## Measured on the target device, 2026-09-21
Bring the decoder over from the project it grew in Eighteen commits of history, grafted onto the scaffold rather than over it. It leaves behind everything that only meant something inside that project: paths under one machine's mount, references to a transcoder that is not here, and a benchmark defaulting to a tile nobody else has. One fixture is real terrain and stays, because it is the only small input that exercises the direct-nibble Huffman weight form. Its licence requires attribution to travel with it, so the README's acknowledgements and the corpus manifest both carry the Copernicus notice now. Everything else committed here is synthetic or was written by the zstd CLI, including the .hgtz container, whose name looks like a tile and whose samples are a function of their position.
2026-09-22 19:25:08 -06:00
Sixteen real 256x256 blocks lifted out of one elevation container,
Measure the decoder on the phone, which is the number that decides it 0.414 ms a block in profile AOT on the Pixel, so a 113-block sweep is 47 ms. The phone is about twice as quick at this as the laptop the first figures came from, which read 0.917 ms and 104 ms for the same sixteen blocks. Debug on device is 0.540 ms, so the shipped configuration is the fast one rather than the slow one. The blocks are real, out of N39W105 in srtm1z. The committed .hgtz fixture is no use for timing: its samples are a synthetic pattern that compresses to almost no sequences, and decode cost tracks sequence count, so it reads about six times too fast. It was measured that way once before this was noticed.
2026-09-21 21:53:21 -06:00
131,072 bytes of output each, the same blocks in every row so the numbers are
comparable. Median of 960 timed calls after 320 warm-up calls.
| where | mode | median | MB/s | 113-block sweep |
| --- | --- | --- | --- | --- |
| Pixel 10 Pro XL | **profile, AOT** | **0.414 ms** | 302 | **47 ms** |
| Pixel 10 Pro XL | debug, JIT | 0.540 ms | 231 | 61 ms |
| this laptop | AOT | 0.917 ms | 136 | 104 ms |
| this laptop | JIT | 1.125 ms | 111 | 127 ms |
The phone is about twice as fast as the laptop at this, which is why the
desktop figures in the commit message read worse than reality.
Native, for scale: pyzstd on the laptop is 0.175 ms on comparable blocks. That
is a Python C extension rather than Dart FFI, so take it as an indication. No
native-on-phone figure was taken.
Do not benchmark against `test/fixtures/N51E000.hgtz`. Its samples are a
synthetic pattern that compresses to almost no sequences, and decode cost
tracks sequence count, so it reads about six times too fast. That mistake was
made once already.
Add funding links
2026-09-22 19:48:27 -06:00
---
If this saved you some time, please consider [sponsoring the work](https://github.com/sponsors/tagadvance).
Reference in a new issue Copy permalink