feat: a dictionary's corrections reach the cards made from it, by offer
All checks were successful
CI / check (push) Successful in 12m33s
CI / deploy (push) Successful in 1m26s

Every dictionary entry now has a version: 16 hex digits of the SHA-256 of
what cards take of it (headword, reading, entry), made by the loader and
carried on every hit. A card keeps the version it took (card schema 8).
DictionaryEndpoint.changed returns, of the entries named with the
versions cards keep (1,000 at most a call; the app asks 500 at a time),
those whose version is another now.

The card browser says how many cards are behind their dictionary and takes
the corrections at a tap (decided 2026-10-09: by offer, never by
themselves). Taking one writes the entry as it is now over the revision
in which the card last took it, beside whatever came after, so the
revisions merge per side as two devices' edits do (ADR 0003): a side only
the dictionary changed takes the correction, a side the person changed
too is a conflict settled in the composer, whose words now name that
case. Cards made before versions are never offered one.

The server rereads its dictionary folder hourly now, not daily: the
dictionaries service fills its volume after an instance's first start,
and an hour with nothing changed costs a hash of each file.

Migration 20261009145434785-dictionary-entry-version: adds
dictionary_entry.version, text NOT NULL DEFAULT '' (rows loaded before
have none and are never reported as corrected), and by hand empties
dictionary_source."fileHash", so every source loads once more and its
entries get their versions; the schema stays as definition.sql says. The
old server goes on working until the switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-09 17:32:00 +02:00
commit 44cee0d6c6
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
20 changed files with 12710 additions and 28 deletions

View file

@ -14,8 +14,8 @@ When you are finished, you can shut down the running server with `Q`.
The dictionaries people look words up in (JMdict, …) are reference data the
server holds for everyone. They come as dictionary files, which
`halkyon_learn_curator` makes from the sources' downloads (its README says how).
At every start, and daily at 04:07, the server loads the files in its
`dictionaries` folder that changed since they were last loaded (by their
At every start, and every hour at seven past, the server loads the files in
its `dictionaries` folder that changed since they were last loaded (by their
SHA-256): in the background, a source at a time, and all at once, so searches
meanwhile still find the old entries. A file it can't read is logged and left
out. `DICTIONARIES_DIR` names another folder.
@ -27,12 +27,12 @@ once a month:
`deploy/compose.yaml` (the curator, in a container of its own) downloads
EDRDG's files of the day once its files are a week old, converts them, and
replaces them in a volume the server reads (`curate update`, the curator's
README); the server takes them up the next morning. To supply files yourself
README); the server takes them up within the hour. To supply files yourself
instead, remove the service and mount `deploy/dictionaries/` into the server
(`compose.yaml` says how), and update them monthly.
- **Development:** `dart run halkyon_learn_curator:curate update --into
../halkyon_learn_server/dictionaries` in the curator's folder (about a
minute), then restart `serverpod start`, or wait for the morning.
minute); the server takes them up within the hour.
JMdict in English, with the JLPT levels and Tatoeba's example sentences, is
about 20 MB gzipped and takes the server half a minute to load (219,000

View file

@ -43,13 +43,14 @@ Future<void> run(List<String> args) async {
.pruneFiles();
// The dictionaries in the server's folder, those that changed since; in
// the background, as loading takes a minute or two. Then again daily, as
// the dictionaries service replaces them weekly.
// the background, as loading takes a minute or two. Then again hourly, as
// the dictionaries service replaces them weekly (and fills its volume
// first after an instance's first start).
unawaited(_loadDictionaries(pod));
await pod.futureCalls.cancel(_dictionaryReload);
await pod.futureCalls
.callRecurring(identifier: _dictionaryReload)
.cron('7 4 * * *')
.cron('7 * * * *')
.dictionaryReload
.reloadDictionaries();
}
@ -73,5 +74,5 @@ const _pruning = 'prune-abandoned-anonymous-people';
/// The identifier of the daily pruning of files.
const _filePruning = 'prune-unreferenced-files';
/// The identifier of the daily rereading of the dictionaries.
/// The identifier of the hourly rereading of the dictionaries.
const _dictionaryReload = 'reload-dictionaries';

View file

@ -132,6 +132,45 @@ class DictionaryEndpoint extends Endpoint {
_hit(entry),
];
/// The most entries [changed] compares at once.
static const maxCompared = 1000;
/// The entries of the dictionary of [source] with meanings in
/// [glossLanguage] that changed since cards were made from them: of
/// [versions] (an entry's id and the version a card keeps of it, at most
/// [maxCompared]), those whose version is another now. An entry gone
/// from the source isn't among them; nor is one loaded before versions,
/// until its source is loaded again.
Future<List<DictionaryHit>> changed(
Session session,
String source,
String glossLanguage,
Map<String, String> versions,
) async {
if (versions.length > maxCompared) {
throw ArgumentError.value(
versions.length,
'versions',
'At most $maxCompared at once',
);
}
if (versions.isEmpty) {
return const [];
}
return [
for (final entry in await DictionaryEntry.db.find(
session,
where: (t) =>
t.source.equals(source) &
t.glossLanguage.equals(glossLanguage) &
t.sourceId.inSet(versions.keys.toSet()),
))
if (entry.version.isNotEmpty &&
entry.version != versions[entry.sourceId])
_hit(entry),
];
}
DictionaryHit _hit(DictionaryEntry entry) => DictionaryHit(
source: entry.source,
sourceId: entry.sourceId,
@ -142,5 +181,6 @@ class DictionaryEndpoint extends Endpoint {
if (label.isNotEmpty) label,
],
entry: entry.entry,
version: entry.version.isEmpty ? null : entry.version,
);
}

View file

@ -30,6 +30,10 @@ fields:
looseKeys: String
### The entry as cards take it: its JSON, as the file holds it.
entry: String
### A digest of what cards take of it (headword, reading, entry): a card
### keeps the one it was made from, and a different one means the source
### corrected the entry since. Empty until the source is loaded again.
version: String, default=''
indexes:
# An entry by its source and id; loading replaces one source at a time.
dictionary_entry_source_idx:

View file

@ -13,3 +13,6 @@ fields:
labels: List<String>
### The entry as cards take it, as JSON (the curator's README).
entry: String
### A digest of what cards take of it; a card keeps it, to know when the
### entry was corrected since. Null from a server older than it.
version: String?

View file

@ -142,21 +142,25 @@ Future<bool> loadDictionaryFile(Session session, File file) async {
continue;
}
final json = jsonDecode(line) as Map<String, Object?>;
final headword = json['headword']! as String;
final reading = json['reading'] as String?;
final entry = jsonEncode(json['entry']);
batch.add(
DictionaryEntry(
source: source,
sourceId: json['id']! as String,
language: language,
glossLanguage: glossLanguage,
headword: json['headword']! as String,
reading: json['reading'] as String?,
headword: headword,
reading: reading,
rank: json['rank']! as int,
labels: barred((json['labels']! as List).cast<String>()),
searchKeys: barred((json['keys']! as List).cast<String>()),
looseKeys: barred(
(json['looseKeys'] as List? ?? const []).cast<String>(),
),
entry: jsonEncode(json['entry']),
entry: entry,
version: entryVersion(headword, reading, entry),
),
);
if (batch.length == _batch) {
@ -205,6 +209,14 @@ Future<bool> loadDictionaryFile(Session session, File file) async {
return true;
}
/// A digest of what a card takes of an entry: its [headword], [reading]
/// and [entry] (JSON). Sixteen hexadecimal digits of their SHA-256: enough
/// to tell a corrected entry from the one a card was made from.
String entryVersion(String headword, String? reading, String entry) => sha256
.convert(utf8.encode([headword, reading ?? '', entry].join('\u0000')))
.toString()
.substring(0, 16);
/// The lines of [file], unzipped when its name ends in `.gz`.
Stream<String> _lines(File file) {
final bytes = file.openRead();

View file

@ -3,7 +3,8 @@ import 'package:serverpod/serverpod.dart';
/// Rereads the dictionary folder and loads the files that changed: the
/// dictionaries service replaces them weekly (`deploy/compose.yaml`); the
/// server schedules it daily when it starts (`server.dart`).
/// server schedules it hourly when it starts (`server.dart`): an hour
/// with nothing changed costs it a hash of each file.
class DictionaryReloadFutureCall extends FutureCall {
/// Loads the dictionary files that changed since they were loaded.
Future<void> reloadDictionaries(Session session) =>

View file

@ -32,7 +32,8 @@ abstract class DictionaryEntry
required this.searchKeys,
required this.looseKeys,
required this.entry,
});
String? version,
}) : version = version ?? '';
factory DictionaryEntry({
int? id,
@ -47,6 +48,7 @@ abstract class DictionaryEntry
required String searchKeys,
required String looseKeys,
required String entry,
String? version,
}) = _DictionaryEntryImpl;
factory DictionaryEntry.fromJson(Map<String, dynamic> jsonSerialization) {
@ -63,6 +65,7 @@ abstract class DictionaryEntry
searchKeys: jsonSerialization['searchKeys'] as String,
looseKeys: jsonSerialization['looseKeys'] as String,
entry: jsonSerialization['entry'] as String,
version: jsonSerialization['version'] as String?,
);
}
@ -108,6 +111,11 @@ abstract class DictionaryEntry
/// The entry as cards take it: its JSON, as the file holds it.
String entry;
/// A digest of what cards take of it (headword, reading, entry): a card
/// keeps the one it was made from, and a different one means the source
/// corrected the entry since. Empty until the source is loaded again.
String version;
@override
_is.Table<int?> get table => t;
@ -127,6 +135,7 @@ abstract class DictionaryEntry
String? searchKeys,
String? looseKeys,
String? entry,
String? version,
});
@override
Map<String, dynamic> toJson() {
@ -144,6 +153,7 @@ abstract class DictionaryEntry
'searchKeys': searchKeys,
'looseKeys': looseKeys,
'entry': entry,
'version': version,
};
}
@ -163,6 +173,7 @@ abstract class DictionaryEntry
'searchKeys': searchKeys,
'looseKeys': looseKeys,
'entry': entry,
'version': version,
};
}
@ -210,6 +221,7 @@ class _DictionaryEntryImpl extends DictionaryEntry {
required String searchKeys,
required String looseKeys,
required String entry,
String? version,
}) : super._(
id: id,
source: source,
@ -223,6 +235,7 @@ class _DictionaryEntryImpl extends DictionaryEntry {
searchKeys: searchKeys,
looseKeys: looseKeys,
entry: entry,
version: version,
);
/// Returns a shallow copy of this [DictionaryEntry]
@ -242,6 +255,7 @@ class _DictionaryEntryImpl extends DictionaryEntry {
String? searchKeys,
String? looseKeys,
String? entry,
String? version,
}) {
return DictionaryEntry(
id: id is int? ? id : this.id,
@ -256,6 +270,7 @@ class _DictionaryEntryImpl extends DictionaryEntry {
searchKeys: searchKeys ?? this.searchKeys,
looseKeys: looseKeys ?? this.looseKeys,
entry: entry ?? this.entry,
version: version ?? this.version,
);
}
}
@ -318,6 +333,11 @@ class DictionaryEntryUpdateTable extends _is.UpdateTable<DictionaryEntryTable> {
table.entry,
value,
);
_is.ColumnValue<String, String> version(String value) => _is.ColumnValue(
table.version,
value,
);
}
class DictionaryEntryTable extends _is.Table<int?> {
@ -368,6 +388,11 @@ class DictionaryEntryTable extends _is.Table<int?> {
'entry',
this,
);
version = _is.ColumnString(
'version',
this,
hasDefault: true,
);
}
late final DictionaryEntryUpdateTable updateTable;
@ -407,6 +432,11 @@ class DictionaryEntryTable extends _is.Table<int?> {
/// The entry as cards take it: its JSON, as the file holds it.
late final _is.ColumnString entry;
/// A digest of what cards take of it (headword, reading, entry): a card
/// keeps the one it was made from, and a different one means the source
/// corrected the entry since. Empty until the source is loaded again.
late final _is.ColumnString version;
@override
List<_is.Column> get columns => [
id,
@ -421,6 +451,7 @@ class DictionaryEntryTable extends _is.Table<int?> {
searchKeys,
looseKeys,
entry,
version,
];
}

View file

@ -24,6 +24,7 @@ abstract class DictionaryHit
this.reading,
required this.labels,
required this.entry,
this.version,
});
factory DictionaryHit({
@ -33,6 +34,7 @@ abstract class DictionaryHit
String? reading,
required List<String> labels,
required String entry,
String? version,
}) = _DictionaryHitImpl;
factory DictionaryHit.fromJson(Map<String, dynamic> jsonSerialization) {
@ -45,6 +47,7 @@ abstract class DictionaryHit
jsonSerialization['labels'],
),
entry: jsonSerialization['entry'] as String,
version: jsonSerialization['version'] as String?,
);
}
@ -65,6 +68,10 @@ abstract class DictionaryHit
/// The entry as cards take it, as JSON (the curator's README).
String entry;
/// A digest of what cards take of it; a card keeps it, to know when the
/// entry was corrected since. Null from a server older than it.
String? version;
/// Returns a shallow copy of this [DictionaryHit]
/// with some or all fields replaced by the given arguments.
@_is.useResult
@ -75,6 +82,7 @@ abstract class DictionaryHit
String? reading,
List<String>? labels,
String? entry,
String? version,
});
@override
Map<String, dynamic> toJson() {
@ -86,6 +94,7 @@ abstract class DictionaryHit
if (reading != null) 'reading': reading,
'labels': labels.toJson(),
'entry': entry,
if (version != null) 'version': version,
};
}
@ -99,6 +108,7 @@ abstract class DictionaryHit
if (reading != null) 'reading': reading,
'labels': labels.toJson(),
'entry': entry,
if (version != null) 'version': version,
};
}
@ -118,6 +128,7 @@ class _DictionaryHitImpl extends DictionaryHit {
String? reading,
required List<String> labels,
required String entry,
String? version,
}) : super._(
source: source,
sourceId: sourceId,
@ -125,6 +136,7 @@ class _DictionaryHitImpl extends DictionaryHit {
reading: reading,
labels: labels,
entry: entry,
version: version,
);
/// Returns a shallow copy of this [DictionaryHit]
@ -138,6 +150,7 @@ class _DictionaryHitImpl extends DictionaryHit {
Object? reading = _Undefined,
List<String>? labels,
String? entry,
Object? version = _Undefined,
}) {
return DictionaryHit(
source: source ?? this.source,
@ -146,6 +159,7 @@ class _DictionaryHitImpl extends DictionaryHit {
reading: reading is String? ? reading : this.reading,
labels: labels ?? this.labels.map((e0) => e0).toList(),
entry: entry ?? this.entry,
version: version is String? ? version : this.version,
);
}
}

View file

@ -445,6 +445,38 @@ class Endpoints extends _is.EndpointDispatch {
limit: params['limit'],
),
),
'changed': _is.MethodConnector(
name: 'changed',
params: {
'source': _is.ParameterDescription(
name: 'source',
type: _is.getType<String>(),
nullable: false,
),
'glossLanguage': _is.ParameterDescription(
name: 'glossLanguage',
type: _is.getType<String>(),
nullable: false,
),
'versions': _is.ParameterDescription(
name: 'versions',
type: _is.getType<Map<String, String>>(),
nullable: false,
),
},
call:
(
_is.Session session,
Map<String, dynamic> params,
) async =>
(endpoints['dictionary'] as _ilgtwlto.DictionaryEndpoint)
.changed(
session,
params['source'],
params['glossLanguage'],
params['versions'],
),
),
},
);
connectors['instance'] = _is.EndpointConnector(

View file

@ -595,6 +595,13 @@ class Protocol extends _is.DatabaseSerializationManager {
isNullable: false,
dartType: 'String',
),
_isp.ColumnDefinition(
name: 'version',
columnType: _isp.ColumnType.text,
isNullable: false,
dartType: 'String',
columnDefault: '\'\'',
),
],
foreignKeys: [],
indexes: [
@ -2479,6 +2486,12 @@ class Protocol extends _is.DatabaseSerializationManager {
.toList()
as T;
}
if (t == Map<String, String>) {
return (data as Map).map(
(k, v) => MapEntry(deserialize<String>(k), deserialize<String>(v)),
)
as T;
}
try {
return _iais.Protocol().deserialize<T>(data, t);
} on _is.DeserializationTypeNotFoundException catch (_) {}

View file

@ -19,5 +19,6 @@ dictionary:
- search:
- countLabelled:
- labelled:
- changed:
instance:
- info:

View file

@ -0,0 +1,34 @@
{
"__className__": "serverpod.DatabaseMigration",
"actions": [
{
"__className__": "serverpod.DatabaseMigrationAction",
"type": "alterTable",
"alterTable": {
"__className__": "serverpod.TableMigration",
"name": "dictionary_entry",
"schema": "public",
"addColumns": [
{
"__className__": "serverpod.ColumnDefinition",
"name": "version",
"fieldName": "version",
"columnType": 0,
"isNullable": false,
"columnDefault": "''",
"dartType": "String"
}
],
"deleteColumns": [],
"modifyColumns": [],
"addIndexes": [],
"deleteIndexes": [],
"addForeignKeys": [],
"deleteForeignKeys": [],
"warnings": []
}
}
],
"warnings": [],
"migrationApiVersion": 1
}

View file

@ -0,0 +1,56 @@
BEGIN;
--
-- ACTION ALTER TABLE
--
ALTER TABLE "dictionary_entry" ADD COLUMN "version" text NOT NULL DEFAULT ''::text;
--
-- DATA (by hand): every dictionary loads again at the next start, so its
-- entries get their versions; the loader reloads a source whose file hash
-- differs from the one stored.
--
UPDATE "dictionary_source" SET "fileHash" = '';
--
-- MIGRATION VERSION FOR halkyon_learn
--
INSERT INTO "serverpod_migrations" ("module", "version", "timestamp")
VALUES ('halkyon_learn', '20261009145434785-dictionary-entry-version', now())
ON CONFLICT ("module")
DO UPDATE SET "version" = '20261009145434785-dictionary-entry-version', "timestamp" = now();
--
-- MIGRATION VERSION FOR serverpod
--
INSERT INTO "serverpod_migrations" ("module", "version", "timestamp")
VALUES ('serverpod', '20260824182259319', now())
ON CONFLICT ("module")
DO UPDATE SET "version" = '20260824182259319', "timestamp" = now();
--
-- MIGRATION VERSION FOR serverpod_auth_idp
--
INSERT INTO "serverpod_migrations" ("module", "version", "timestamp")
VALUES ('serverpod_auth_idp', '20260924105404509', now())
ON CONFLICT ("module")
DO UPDATE SET "version" = '20260924105404509', "timestamp" = now();
--
-- MIGRATION VERSION FOR serverpod_offline_sync
--
INSERT INTO "serverpod_migrations" ("module", "version", "timestamp")
VALUES ('serverpod_offline_sync', '20260914143806119', now())
ON CONFLICT ("module")
DO UPDATE SET "version" = '20260914143806119', "timestamp" = now();
--
-- MIGRATION VERSION FOR serverpod_auth_core
--
INSERT INTO "serverpod_migrations" ("module", "version", "timestamp")
VALUES ('serverpod_auth_core', '20260924105232991', now())
ON CONFLICT ("module")
DO UPDATE SET "version" = '20260924105232991', "timestamp" = now();
COMMIT;

View file

@ -17,3 +17,4 @@
20261008165051407-pdf-position
20261008223835206-dictionaries
20261009123850448-dictionary-kind
20261009145434785-dictionary-entry-version

View file

@ -1,6 +1,7 @@
import 'dart:convert';
import 'dart:io';
import 'package:halkyon_learn_server/src/dictionary/dictionary_endpoint.dart';
import 'package:halkyon_learn_server/src/dictionary/dictionary_loader.dart';
import 'package:halkyon_learn_server/src/dictionary/dictionary_search.dart';
import 'package:halkyon_learn_server/src/generated/protocol.dart';
@ -512,6 +513,75 @@ void main() {
expect(jsonDecode(hit.entry), isA<Map<String, Object?>>());
});
test("a hit carries its entry's version, which changes with what a "
'card takes of it', () async {
Future<String?> version() async => (await endpoints.dictionary.search(
signedIn,
'jmdict',
'en',
'taberu',
)).single.version;
await loadDictionaryFile(session, write([taberu]));
final before = await version();
final corrected = write([
{
...taberu,
'entry': {
'senses': [
{
'glosses': ['to eat', 'to consume'],
},
],
},
},
], head: header(version: '3.6.3 (2026-11-01)'));
await loadDictionaryFile(session, corrected);
final after = await version();
expect(before, hasLength(16));
expect(after, isNot(before));
expect(entryVersion('食べる', 'たべる', '{}'), hasLength(16));
});
test('changed: of the entries named, those whose version is another '
'now; none gone from the source', () async {
await loadDictionaryFile(session, write([taberu, terebi]));
final versions = {
for (final hit in await endpoints.dictionary.labelled(
signedIn,
'jmdict',
'en',
'jlpt:n5',
))
hit.sourceId: hit.version!,
};
Future<List<String>> changed(Map<String, String> asked) async => [
for (final hit in await endpoints.dictionary.changed(
signedIn,
'jmdict',
'en',
asked,
))
hit.sourceId,
];
expect(await changed(versions), isEmpty);
expect(
await changed({...versions, '1358280': 'an older one'}),
['1358280'],
);
expect(await changed({'404': 'gone'}), isEmpty);
expect(await changed({}), isEmpty);
expect(
() => changed({
for (var i = 0; i <= DictionaryEndpoint.maxCompared; i++) '$i': 'v',
}),
throwsA(anything),
);
});
test('the dictionaries are for those signed in', () {
expect(
() => endpoints.dictionary.search(

View file

@ -775,6 +775,43 @@ class _DictionaryEndpoint {
}
});
}
_ida.Future<List<_ism30i9n.DictionaryHit>> changed(
_ist.TestSessionBuilder sessionBuilder,
String source,
String glossLanguage,
Map<String, String> versions,
) async {
return _ist.callAwaitableFunctionAndHandleExceptions(() async {
var _localUniqueSession =
(sessionBuilder as _ist.InternalTestSessionBuilder).internalBuild(
endpoint: 'dictionary',
method: 'changed',
);
try {
var _localCallContext = await _endpointDispatch.getMethodCallContext(
createSessionCallback: (_) => _localUniqueSession,
endpointPath: 'dictionary',
methodName: 'changed',
parameters: _ist.testObjectToJson({
'source': source,
'glossLanguage': glossLanguage,
'versions': versions,
}),
serializationManager: _serializationManager,
);
var _localReturnValue =
await (_localCallContext.method.call(
_localUniqueSession,
_localCallContext.arguments,
)
as _ida.Future<List<_ism30i9n.DictionaryHit>>);
return _localReturnValue;
} finally {
await _localUniqueSession.close();
}
});
}
}
class _InstanceEndpoint {