halkyon-days/halkyon
Watch
1
0
Fork
You've already forked halkyon
 0
Code Issues Pull requests Projects Releases 37 Packages Wiki Activity Actions 1

feat(curator): Tatoeba's sentences as examples in every new language

Browse source 
JMdict brings Tatoeba's sentences for Japanese; the other languages'
dictionaries had none (CC-CEDICT) or Wiktionary's own, often
quotations. Each file now takes Tatoeba's sentences of the week (its
per-language exports, CC BY 2.0 FR) as its entries' first examples,
which become a card's hints: each sentence with a translation into the
dictionary's meaning language goes to the entries its words find, cut
as the app cuts a line of subtitles (at spaces, Korean's particles off,
Chinese by its longest words), up to three for each, those using the
word as its headword writes it first (Haus rather than zu Hause; 学习
rather than 學習 in the simplified dictionary), then those nearest seven
words long. German's common words nearly all get some (8,849 of
9,292), in ten seconds and under 300 MB; a dictionary whose sentences
can't be had keeps its words. `curate tatoeba` does the same for a file
made by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-14 07:49:34 +02:00
commit 222175584b
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
1 changed files with 811 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

24
halkyon_learn_curator/README.md
View file

@ -102,6 +102,24 @@ forms):
apart, each given to the entry read as the syllabus reads the word (的 *de*,
not 的 *dì*), and rank them, HSK 1 first; wordfreq ranks the rest.
Example sentences from Tatoeba for those dictionaries (JMdict brings its own),
as hints on the cards made from them:
```sh
dart run halkyon_learn_curator:curate tatoeba \
--input wiktionary-de-en.jsonl.gz --sentences deu_sentences.tsv.bz2 \
--translations eng_sentences.tsv.bz2 --links deu-eng_links.tsv.bz2 \
--output wiktionary-de-en.jsonl.gz
```
from [Tatoeba's weekly exports](https://downloads.tatoeba.org/exports/per_language/)
of the dictionary's language, of its meaning language and of the links between
them (CC BY 2.0 FR). Each sentence with a translation goes to the entries its
words find by their text keys, as the app finds words in a line of subtitles,
up to three for each: those using the word as its headword writes it first, then
those nearest seven words long, the shortest then; they come first in an entry's
first sense, the word as it stands there beside. Some ten seconds for German.
Put the files into the server's `dictionaries` folder. The server loads each on
its next start, and again whenever it changes (see the server's README).
@ -130,8 +148,10 @@ so in a container of its own (the server's README). Beside the files,
from the same download (a label, a field), and an instance converts again as
soon as the new curator runs, not a week later.
With `--languages de,el,en,ko,zh` it also keeps those languages current: Chinese
from MDBG's CC-CEDICT of the day as often as EDRDG's files (below), the rest from
With `--languages de,el,en,ko,zh` it also keeps those languages current, each
file with Tatoeba's sentences of the week as examples (a dictionary whose
sentences can't be had keeps its words without them): Chinese from MDBG's
CC-CEDICT of the day as often as EDRDG's files (below), the rest from
Wiktionary: each time it wakes it asks kaikki.org when its dump last changed
(`Last-Modified`, without downloading it), and converts it again only when it is
newer than the files' (about monthly), when the languages named changed, or when