halkyon-curator

CLI tool for turning curated dictionary data into Halkyon Learn archives (.halkyon) — the portable studyset + flashcard + tag bundle the app can import.

Full reference: docs/curator-data-catalog.md documents every tool — server-side seeders, .halkyon archive builders, and config assets — with sources, commands, and flags. docs/seeding-doc-symbols.md is the deep-dive runbook for the doc_symbols "Link to docs" catalog. docs/architecture.md explains how the curator fits the wider Halkyon system and the provider/seeder design.

The first source is jmdict-simplified: Japanese vocabulary emitted as vocabulary flashcards (expression / kana reading / meanings) tagged with their part of speech (noun, godan verb, ichidan verb, i-adjective, na-adjective, …) and a common tag.

Usage

dart pub get

# A focused JLPT N5 deck (POS + common + jlpt-n5 tags):
dart run halkyon_curator:curator \
  --input jmdict-eng.json \
  --output jlpt-n5.halkyon \
  --title "JLPT N5 vocabulary" \
  --jlpt-dir ./yomitan-jlpt-vocab/original_data \
  --jlpt-level n5

# Or the common-word slice, capped:
dart run halkyon_curator:curator \
  --input jmdict-eng.json \
  --output common-vocab.halkyon \
  --title "Common Japanese vocabulary" \
  --common-only --limit 1000

--jlpt-level accepts one or more levels (n5, or n5,n4) and requires --jlpt-dir; it keeps only words listed at those levels.

Then import japanese-vocab.halkyon in Halkyon Learn (Library → Import archive). Imported studysets become a fresh local copy owned by the importing user.

Flutter documentation

A hand-authored JSON file of Flutter-docs lessons (each a note plus its derived flashcards) becomes one studyset, stamped with its Flutter source_version and curated_at date:

dart run halkyon_curator:flutter_docs \
  --input flutter_docs.json \
  --output flutter.halkyon \
  --title "Flutter"

See example/flutter_docs.sample.json for the input format. The optional announcements array is a license-safe "What's new" section: each entry is an original summary plus a link to the source — never the source's own text (see licensing below).

Seeding a server-side dictionary

Besides emitting archives, the curator can seed a server's dictionary_entries catalog — the on-demand lookup dictionary the app queries (and the browser extension mines from). The catalog is reference data shared read-only across a server's users, written only by the service role, and scoped by a language code (a studyset's reference_language selects which dictionary to query). This is how a private, self-hosted server gets its own dictionaries.

Pass the project URL and read the service-role key from the environment (never an argv flag — it would land in shell history):

export SUPABASE_SERVICE_ROLE_KEY=ey...        # service_role, not anon

# JMdict (Japanese), language code `ja`:
dart run halkyon_curator:seed_dictionary \
  --input jmdict-eng.json \
  --supabase-url https://<project>.supabase.co

# A bilingual dictionary JSONL — one file per direction, e.g. de-en:
dart run halkyon_curator:seed_bilingual \
  --input de-en.jsonl \
  --supabase-url https://<project>.supabase.co

Both stream the input and upsert in batches on the (id, language) primary key, so re-running — after a newer dump or to resume a partial seed — is idempotent. Use --dry-run to validate the built rows without a server. The target project must already have the dictionary_entries schema (apply the halkyon-infra migrations there first).

Bring your own data. seed_bilingual ships no dictionary data; it imports a JSONL you supply. Seed only sources you are entitled to use, and prefer a private, self-hosted server for anything not openly licensed.

Data sources & licensing

Nothing here is authored by the curator — it repackages openly-licensed dictionary data, and every emitted archive is stamped accordingly:

Data	Source	License
Expressions, readings, meanings, part-of-speech, common flag	JMdict via jmdict-simplified	© EDRDG, CC BY-SA 4.0
Flutter documentation prose	docs.flutter.dev (the flutter/website repo)	© Google LLC, CC BY 4.0 (code samples BSD 3-Clause)

Emitted archives carry license: CC-BY-SA-4.0 and an EDRDG attribution line in the studyset description. Because the data is share-alike, studysets imported from these archives remain CC BY-SA 4.0.

The Flutter-docs archive carries license: CC-BY-4.0 and an attribution line. Release announcements are different: Flutter release news is typically posted on Medium, whose articles are all-rights-reserved by default. The announcements input therefore carries only original summaries (written in our own words) plus a source link — never the article's text — so the studyset stays license-clean.

Planned: JLPT N5–N1 tags via yomitan-jlpt-vocab (joined to JMdict by entry id; CC BY-SA; JLPT levels are unofficial).

Development

dart analyze
dart test