feat(server): dictionaries, loaded from files and searched
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.
DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).
The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:51:27 +02:00
|
|
|
import 'dart:async';
|
|
|
|
|
import 'dart:io';
|
|
|
|
|
|
2026-09-30 09:03:52 +02:00
|
|
|
import 'package:halkyon_learn_server/src/auth/auth_config.dart';
|
feat(server): dictionaries, loaded from files and searched
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.
DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).
The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:51:27 +02:00
|
|
|
import 'package:halkyon_learn_server/src/dictionary/dictionary_loader.dart';
|
chore(lint): LeanCode's strict rules and a pre-push check
The workspace root's analysis_options.yaml includes leancode_lint 27 and enables its analyzer plugin (rules for blocs, Equatable and Flutter layout); the server and the app include that file, while the generated client keeps its own options. Plugins can only be enabled at the root of a pub workspace and only reach packages whose options include the root file, which was verified before settling on this layout. Most rules are warnings, casts, inference and raw types are strict, and the check treats infos as fatal too. lints and flutter_lints are no longer needed.
tool/check.sh checks formatting, runs dart analyze --fatal-infos and the server tests; .git/hooks/pre-push links to it, so every push has to pass it. AGENTS.md describes the checklist and how to install the hook in a fresh clone.
Fixes for the new rules: package imports, run() returns Future<void>, the server setup as one cascade, a primary constructor for the app widget, a const and a redundant argument. The workspace root keeps its conventional name `_` with a documented ignore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-29 23:06:29 +02:00
|
|
|
import 'package:halkyon_learn_server/src/generated/serverpod.dart';
|
2026-09-16 19:24:59 +02:00
|
|
|
import 'package:serverpod_auth_idp_server/core.dart';
|
|
|
|
|
|
|
|
|
|
/// The starting point of the Serverpod server.
|
chore(lint): LeanCode's strict rules and a pre-push check
The workspace root's analysis_options.yaml includes leancode_lint 27 and enables its analyzer plugin (rules for blocs, Equatable and Flutter layout); the server and the app include that file, while the generated client keeps its own options. Plugins can only be enabled at the root of a pub workspace and only reach packages whose options include the root file, which was verified before settling on this layout. Most rules are warnings, casts, inference and raw types are strict, and the check treats infos as fatal too. lints and flutter_lints are no longer needed.
tool/check.sh checks formatting, runs dart analyze --fatal-infos and the server tests; .git/hooks/pre-push links to it, so every push has to pass it. AGENTS.md describes the checklist and how to install the hook in a fresh clone.
Fixes for the new rules: package imports, run() returns Future<void>, the server setup as one cascade, a primary constructor for the app widget, a const and a redundant argument. The workspace root keeps its conventional name `_` with a documented ignore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-29 23:06:29 +02:00
|
|
|
Future<void> run(List<String> args) async {
|
2026-09-16 19:24:59 +02:00
|
|
|
// Initialize Serverpod. The generated Serverpod class is already connected
|
|
|
|
|
// with your project's generated code.
|
feat(server): GitHub sign-in that keeps only the GitHub ID
As decided, a GitHub sign-in keeps nothing but the numeric GitHub ID, so the login is our own, on Serverpod's GitHub helpers (lib/src/auth/github.dart): Serverpod's stock login also stores the name and email in a profile and downloads the avatar, and its endpoint is never exposed. The server exchanges the authorization code with the client secret, reads the ID, and drops the access token.
GitHubUpgradeEndpoint links the GitHub ID to the signed-in anonymous person, grants identified (to their existing sessions too) and removes the anonymous identity, in one transaction: the person, their spaces and their studysets stay the same. GitHubSignInEndpoint gives a session for a person who has the GitHub account already; it's called without a session, since Serverpod issues no session for another person inside a signed-in call, and it never creates people. InstanceEndpoint.info() tells the app the instance's GitHub client ID. Refusals are GitHubSignInRefusedException with a reason: GitHub sign-in off here, the account taken by another person, another GitHub account linked already, an unknown account.
GitHub sign-in is on only where the passwords hold githubClientId and githubClientSecret (empty counts as missing, as Compose passes unset variables on as empty strings); production takes them from GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET in deploy/.env. grantIdentified can now join a caller's transaction. No migration: the GitHub accounts table comes with the auth module.
Tests, refusals first: upgrading when not signed in (checked by switching the guard off: the test fails), both ways without a GitHub app, an account taken by someone else with nothing changed, a second GitHub account, an unknown account with nobody created; then linking keeps the person and reaches their sessions and stores no email, linking twice changes nothing, signing in reaches the right person.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-30 13:35:51 +02:00
|
|
|
final pod = Serverpod(args);
|
|
|
|
|
pod
|
2026-09-30 09:03:52 +02:00
|
|
|
// Token managers validate and issue authentication keys; identity
|
|
|
|
|
// providers are the ways people can sign in (see auth_config.dart).
|
chore(lint): LeanCode's strict rules and a pre-push check
The workspace root's analysis_options.yaml includes leancode_lint 27 and enables its analyzer plugin (rules for blocs, Equatable and Flutter layout); the server and the app include that file, while the generated client keeps its own options. Plugins can only be enabled at the root of a pub workspace and only reach packages whose options include the root file, which was verified before settling on this layout. Most rules are warnings, casts, inference and raw types are strict, and the check treats infos as fatal too. lints and flutter_lints are no longer needed.
tool/check.sh checks formatting, runs dart analyze --fatal-infos and the server tests; .git/hooks/pre-push links to it, so every push has to pass it. AGENTS.md describes the checklist and how to install the hook in a fresh clone.
Fixes for the new rules: package imports, run() returns Future<void>, the server setup as one cascade, a primary constructor for the app widget, a const and a redundant argument. The workspace root keeps its conventional name `_` with a documented ignore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-29 23:06:29 +02:00
|
|
|
..initializeAuthServices(
|
2026-09-30 09:03:52 +02:00
|
|
|
tokenManagerBuilders: tokenManagerBuilders,
|
feat(server): GitHub sign-in that keeps only the GitHub ID
As decided, a GitHub sign-in keeps nothing but the numeric GitHub ID, so the login is our own, on Serverpod's GitHub helpers (lib/src/auth/github.dart): Serverpod's stock login also stores the name and email in a profile and downloads the avatar, and its endpoint is never exposed. The server exchanges the authorization code with the client secret, reads the ID, and drops the access token.
GitHubUpgradeEndpoint links the GitHub ID to the signed-in anonymous person, grants identified (to their existing sessions too) and removes the anonymous identity, in one transaction: the person, their spaces and their studysets stay the same. GitHubSignInEndpoint gives a session for a person who has the GitHub account already; it's called without a session, since Serverpod issues no session for another person inside a signed-in call, and it never creates people. InstanceEndpoint.info() tells the app the instance's GitHub client ID. Refusals are GitHubSignInRefusedException with a reason: GitHub sign-in off here, the account taken by another person, another GitHub account linked already, an unknown account.
GitHub sign-in is on only where the passwords hold githubClientId and githubClientSecret (empty counts as missing, as Compose passes unset variables on as empty strings); production takes them from GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET in deploy/.env. grantIdentified can now join a caller's transaction. No migration: the GitHub accounts table comes with the auth module.
Tests, refusals first: upgrading when not signed in (checked by switching the guard off: the test fails), both ways without a GitHub app, an account taken by someone else with nothing changed, a second GitHub account, an unknown account with nobody created; then linking keeps the person and reaches their sessions and stores no email, linking twice changes nothing, signing in reaches the right person.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-30 13:35:51 +02:00
|
|
|
identityProviderBuilders: identityProviderBuilders(pod),
|
chore(lint): LeanCode's strict rules and a pre-push check
The workspace root's analysis_options.yaml includes leancode_lint 27 and enables its analyzer plugin (rules for blocs, Equatable and Flutter layout); the server and the app include that file, while the generated client keeps its own options. Plugins can only be enabled at the root of a pub workspace and only reach packages whose options include the root file, which was verified before settling on this layout. Most rules are warnings, casts, inference and raw types are strict, and the check treats infos as fatal too. lints and flutter_lints are no longer needed.
tool/check.sh checks formatting, runs dart analyze --fatal-infos and the server tests; .git/hooks/pre-push links to it, so every push has to pass it. AGENTS.md describes the checklist and how to install the hook in a fresh clone.
Fixes for the new rules: package imports, run() returns Future<void>, the server setup as one cascade, a primary constructor for the app widget, a const and a redundant argument. The workspace root keeps its conventional name `_` with a documented ignore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-29 23:06:29 +02:00
|
|
|
)
|
|
|
|
|
// Files live in Postgres until the self-hosted S3-compatible storage arrives later.
|
|
|
|
|
..addCloudStorage(DatabaseCloudStorage('private'))
|
|
|
|
|
..addCloudStorage(DatabaseCloudStorage('public'));
|
2026-09-16 19:24:59 +02:00
|
|
|
|
|
|
|
|
// Start the server.
|
|
|
|
|
await pod.start();
|
feat(server): deleting accounts and pruning abandoned ones
deletePerson deletes a person with everything they have, in one transaction and in ADR 0001 rule 6's order: their sessions first, since a device with a valid session would re-upload everything; then their memberships and personal space, whose foreign keys cascade into every synced table and the sync engine's bookkeeping; then the person, whose identities cascade. AccountEndpoint.delete deletes the caller and tells their devices that their sessions ended.
pruneAbandonedAnonymousPeople deletes, 500 at a time, anonymous people created and unused for 90 days: their sessions have expired and nobody can ever return to them. Identified people are never pruned. PruningFutureCall runs it daily at 03:17; the server schedules it at start, cancelling the previous entry by its identifier first, since scheduling always adds one.
Tests, refusal first: deleting when not signed in; deleting removes the caller, their sessions, their anonymous identity and the content of their space, and nobody else's; pruning takes the unused and the never-signed-in, and spares the recently used, newcomers and identified people.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-30 17:09:47 +02:00
|
|
|
|
|
|
|
|
// Abandoned anonymous people are pruned daily (decided 2026-09-30).
|
|
|
|
|
// Scheduling always adds an entry, so the previous one goes first.
|
|
|
|
|
await pod.futureCalls.cancel(_pruning);
|
|
|
|
|
await pod.futureCalls
|
|
|
|
|
.callRecurring(identifier: _pruning)
|
|
|
|
|
.cron('17 3 * * *')
|
|
|
|
|
.pruning
|
|
|
|
|
.pruneAnonymousPeople();
|
Store people's files on the server
The asset endpoint (login required) takes a file into the caller's own space
by its SHA-256 and serves an asset row's file only to its space's owner and
members; anyone else, and a deleted row, gets NotFoundException as for a
missing file. startUpload hands out Serverpod's upload description, or null
when the file is there already (claiming it anew); finishUpload keeps the
upload only if its content has the announced hash. Files are stored in the
private database storage at spaces/<space>/<sha256>, at most 20 MB.
deletePerson deletes the person's files in its transaction, and a daily
future call deletes the files no live asset row refers to, and uploads never
finished, 7 days after they were last claimed (ADR 0004, rule 5).
The tests run on the sync engine's database, as the server does, without
rollback: its writes take transactions of their own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-02 04:52:02 +02:00
|
|
|
|
|
|
|
|
// So are the files nobody can reach any more (ADR 0004, rule 5).
|
|
|
|
|
await pod.futureCalls.cancel(_filePruning);
|
|
|
|
|
await pod.futureCalls
|
|
|
|
|
.callRecurring(identifier: _filePruning)
|
|
|
|
|
.cron('47 3 * * *')
|
|
|
|
|
.filePruning
|
|
|
|
|
.pruneFiles();
|
feat(server): dictionaries, loaded from files and searched
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.
DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).
The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:51:27 +02:00
|
|
|
|
feat(deploy): instances keep their dictionaries current, weekly, by themselves
EDRDG's licence asks of a server showing its dictionaries that it update
them at least monthly. The instance now does it alone: a `dictionaries`
service in compose, built from the curator (halkyon_learn_curator/
Dockerfile, its own unprivileged user 10001, 1 GB at most), runs `curate
update --every 7`, downloading EDRDG's files of the day once its files are
a week old and replacing them in a volume the server mounts read-only.
The curator never enters the server's image or process (decided
2026-10-09: a container of its own, weekly). The server rereads the
folder daily at 04:07 (DictionaryReloadFutureCall), loading what changed,
and loads now run one at a time. redeploy.sh builds both images.
An operator who'd rather supply the files removes the service and mounts
deploy/dictionaries/ into the server, as compose.yaml says.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 16:52:37 +02:00
|
|
|
// The dictionaries in the server's folder, those that changed since; in
|
|
|
|
|
// the background, as loading takes a minute or two. Then again daily, as
|
|
|
|
|
// the dictionaries service replaces them weekly.
|
fix: dictionary headwords, search order and a loader that can't crash
The curator:
- A word JMdict says is usually written in kana gets a kana headword,
even where the JLPT lists give kanji (その, not 其の), and a rare or
search-only spelling is never the headword while a usual reading
exists (そして, not 而して); 9,611 headwords change, 96 of them N5.
- Keys hold full-width letters as typed (Tシャツ as tシャツ), デュ/テュ/
フュ/イェ romanised, long a, i and e typed once as loose keys (kado for
カード), and a blank JLPT line is skipped.
The server:
- A search is one query ordered by tier, then rank: a loose or prefix
match no longer pushes an exact one out of the list (75-150 ms over
all of JMdict, before 150-450).
- Queries find long vowels with macrons spelt おお or ー and typed once
(ōkii, kōhī), ん typed n' or nn, and full-width letters.
- Pages of a label are ordered by rank and id, so entries that share a
rank are neither repeated nor skipped.
- The loader never throws (an unreadable folder ended the server from
the background), refuses two files of one source rather than loading
both at every start, reads no rows back when replacing a source, and
takes its time from clock.
- Dictionary files stay out of the Docker build context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 12:15:16 +02:00
|
|
|
unawaited(_loadDictionaries(pod));
|
feat(deploy): instances keep their dictionaries current, weekly, by themselves
EDRDG's licence asks of a server showing its dictionaries that it update
them at least monthly. The instance now does it alone: a `dictionaries`
service in compose, built from the curator (halkyon_learn_curator/
Dockerfile, its own unprivileged user 10001, 1 GB at most), runs `curate
update --every 7`, downloading EDRDG's files of the day once its files are
a week old and replacing them in a volume the server mounts read-only.
The curator never enters the server's image or process (decided
2026-10-09: a container of its own, weekly). The server rereads the
folder daily at 04:07 (DictionaryReloadFutureCall), loading what changed,
and loads now run one at a time. redeploy.sh builds both images.
An operator who'd rather supply the files removes the service and mounts
deploy/dictionaries/ into the server, as compose.yaml says.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 16:52:37 +02:00
|
|
|
await pod.futureCalls.cancel(_dictionaryReload);
|
|
|
|
|
await pod.futureCalls
|
|
|
|
|
.callRecurring(identifier: _dictionaryReload)
|
|
|
|
|
.cron('7 4 * * *')
|
|
|
|
|
.dictionaryReload
|
|
|
|
|
.reloadDictionaries();
|
fix: dictionary headwords, search order and a loader that can't crash
The curator:
- A word JMdict says is usually written in kana gets a kana headword,
even where the JLPT lists give kanji (その, not 其の), and a rare or
search-only spelling is never the headword while a usual reading
exists (そして, not 而して); 9,611 headwords change, 96 of them N5.
- Keys hold full-width letters as typed (Tシャツ as tシャツ), デュ/テュ/
フュ/イェ romanised, long a, i and e typed once as loose keys (kado for
カード), and a blank JLPT line is skipped.
The server:
- A search is one query ordered by tier, then rank: a loose or prefix
match no longer pushes an exact one out of the list (75-150 ms over
all of JMdict, before 150-450).
- Queries find long vowels with macrons spelt おお or ー and typed once
(ōkii, kōhī), ん typed n' or nn, and full-width letters.
- Pages of a label are ordered by rank and id, so entries that share a
rank are neither repeated nor skipped.
- The loader never throws (an unreadable folder ended the server from
the background), refuses two files of one source rather than loading
both at every start, reads no rows back when replacing a source, and
takes its time from clock.
- Dictionary files stay out of the Docker build context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 12:15:16 +02:00
|
|
|
}
|
|
|
|
|
|
feat(deploy): instances keep their dictionaries current, weekly, by themselves
EDRDG's licence asks of a server showing its dictionaries that it update
them at least monthly. The instance now does it alone: a `dictionaries`
service in compose, built from the curator (halkyon_learn_curator/
Dockerfile, its own unprivileged user 10001, 1 GB at most), runs `curate
update --every 7`, downloading EDRDG's files of the day once its files are
a week old and replacing them in a volume the server mounts read-only.
The curator never enters the server's image or process (decided
2026-10-09: a container of its own, weekly). The server rereads the
folder daily at 04:07 (DictionaryReloadFutureCall), loading what changed,
and loads now run one at a time. redeploy.sh builds both images.
An operator who'd rather supply the files removes the service and mounts
deploy/dictionaries/ into the server, as compose.yaml says.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 16:52:37 +02:00
|
|
|
/// Loads the dictionaries in the server's folder ([dictionaryFolder]).
|
|
|
|
|
/// [loadDictionaries] throws nothing; a session that can't be had is
|
|
|
|
|
/// reported here, as an error in the background would end the server.
|
fix: dictionary headwords, search order and a loader that can't crash
The curator:
- A word JMdict says is usually written in kana gets a kana headword,
even where the JLPT lists give kanji (その, not 其の), and a rare or
search-only spelling is never the headword while a usual reading
exists (そして, not 而して); 9,611 headwords change, 96 of them N5.
- Keys hold full-width letters as typed (Tシャツ as tシャツ), デュ/テュ/
フュ/イェ romanised, long a, i and e typed once as loose keys (kado for
カード), and a blank JLPT line is skipped.
The server:
- A search is one query ordered by tier, then rank: a loose or prefix
match no longer pushes an exact one out of the list (75-150 ms over
all of JMdict, before 150-450).
- Queries find long vowels with macrons spelt おお or ー and typed once
(ōkii, kōhī), ん typed n' or nn, and full-width letters.
- Pages of a label are ordered by rank and id, so entries that share a
rank are neither repeated nor skipped.
- The loader never throws (an unreadable folder ended the server from
the background), refuses two files of one source rather than loading
both at every start, reads no rows back when replacing a source, and
takes its time from clock.
- Dictionary files stay out of the Docker build context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 12:15:16 +02:00
|
|
|
Future<void> _loadDictionaries(Serverpod pod) async {
|
|
|
|
|
try {
|
|
|
|
|
await pod.withSession(
|
feat(deploy): instances keep their dictionaries current, weekly, by themselves
EDRDG's licence asks of a server showing its dictionaries that it update
them at least monthly. The instance now does it alone: a `dictionaries`
service in compose, built from the curator (halkyon_learn_curator/
Dockerfile, its own unprivileged user 10001, 1 GB at most), runs `curate
update --every 7`, downloading EDRDG's files of the day once its files are
a week old and replacing them in a volume the server mounts read-only.
The curator never enters the server's image or process (decided
2026-10-09: a container of its own, weekly). The server rereads the
folder daily at 04:07 (DictionaryReloadFutureCall), loading what changed,
and loads now run one at a time. redeploy.sh builds both images.
An operator who'd rather supply the files removes the service and mounts
deploy/dictionaries/ into the server, as compose.yaml says.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 16:52:37 +02:00
|
|
|
(session) => loadDictionaries(session, dictionaryFolder()),
|
fix: dictionary headwords, search order and a loader that can't crash
The curator:
- A word JMdict says is usually written in kana gets a kana headword,
even where the JLPT lists give kanji (その, not 其の), and a rare or
search-only spelling is never the headword while a usual reading
exists (そして, not 而して); 9,611 headwords change, 96 of them N5.
- Keys hold full-width letters as typed (Tシャツ as tシャツ), デュ/テュ/
フュ/イェ romanised, long a, i and e typed once as loose keys (kado for
カード), and a blank JLPT line is skipped.
The server:
- A search is one query ordered by tier, then rank: a loose or prefix
match no longer pushes an exact one out of the list (75-150 ms over
all of JMdict, before 150-450).
- Queries find long vowels with macrons spelt おお or ー and typed once
(ōkii, kōhī), ん typed n' or nn, and full-width letters.
- Pages of a label are ordered by rank and id, so entries that share a
rank are neither repeated nor skipped.
- The loader never throws (an unreadable folder ended the server from
the background), refuses two files of one source rather than loading
both at every start, reads no rows back when replacing a source, and
takes its time from clock.
- Dictionary files stay out of the Docker build context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 12:15:16 +02:00
|
|
|
);
|
|
|
|
|
} on Object catch (error) {
|
|
|
|
|
stderr.writeln('Loading the dictionaries failed: $error');
|
|
|
|
|
}
|
2026-09-16 19:24:59 +02:00
|
|
|
}
|
feat(server): deleting accounts and pruning abandoned ones
deletePerson deletes a person with everything they have, in one transaction and in ADR 0001 rule 6's order: their sessions first, since a device with a valid session would re-upload everything; then their memberships and personal space, whose foreign keys cascade into every synced table and the sync engine's bookkeeping; then the person, whose identities cascade. AccountEndpoint.delete deletes the caller and tells their devices that their sessions ended.
pruneAbandonedAnonymousPeople deletes, 500 at a time, anonymous people created and unused for 90 days: their sessions have expired and nobody can ever return to them. Identified people are never pruned. PruningFutureCall runs it daily at 03:17; the server schedules it at start, cancelling the previous entry by its identifier first, since scheduling always adds one.
Tests, refusal first: deleting when not signed in; deleting removes the caller, their sessions, their anonymous identity and the content of their space, and nobody else's; pruning takes the unused and the never-signed-in, and spares the recently used, newcomers and identified people.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-30 17:09:47 +02:00
|
|
|
|
|
|
|
|
/// The identifier of the daily pruning.
|
|
|
|
|
const _pruning = 'prune-abandoned-anonymous-people';
|
Store people's files on the server
The asset endpoint (login required) takes a file into the caller's own space
by its SHA-256 and serves an asset row's file only to its space's owner and
members; anyone else, and a deleted row, gets NotFoundException as for a
missing file. startUpload hands out Serverpod's upload description, or null
when the file is there already (claiming it anew); finishUpload keeps the
upload only if its content has the announced hash. Files are stored in the
private database storage at spaces/<space>/<sha256>, at most 20 MB.
deletePerson deletes the person's files in its transaction, and a daily
future call deletes the files no live asset row refers to, and uploads never
finished, 7 days after they were last claimed (ADR 0004, rule 5).
The tests run on the sync engine's database, as the server does, without
rollback: its writes take transactions of their own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-02 04:52:02 +02:00
|
|
|
|
|
|
|
|
/// The identifier of the daily pruning of files.
|
|
|
|
|
const _filePruning = 'prune-unreferenced-files';
|
feat(deploy): instances keep their dictionaries current, weekly, by themselves
EDRDG's licence asks of a server showing its dictionaries that it update
them at least monthly. The instance now does it alone: a `dictionaries`
service in compose, built from the curator (halkyon_learn_curator/
Dockerfile, its own unprivileged user 10001, 1 GB at most), runs `curate
update --every 7`, downloading EDRDG's files of the day once its files are
a week old and replacing them in a volume the server mounts read-only.
The curator never enters the server's image or process (decided
2026-10-09: a container of its own, weekly). The server rereads the
folder daily at 04:07 (DictionaryReloadFutureCall), loading what changed,
and loads now run one at a time. redeploy.sh builds both images.
An operator who'd rather supply the files removes the service and mounts
deploy/dictionaries/ into the server, as compose.yaml says.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 16:52:37 +02:00
|
|
|
|
|
|
|
|
/// The identifier of the daily rereading of the dictionaries.
|
|
|
|
|
const _dictionaryReload = 'reload-dictionaries';
|