halkyon/packages/halkyon_llm/README.md

# halkyon_llm

Language models' APIs in pure Dart, streamed, over plain HTTP, no SDKs: Anthropic's
Messages API, OpenAI's chat completions (and every service compatible with them), and
OpenAI's Responses API, behind one interface of what they share: a system prompt and
messages, text as the model writes it, images, and tool calls. Halkyon's server forwards
the operator's assistants with it, and its apps call a person's own provider with it
(Halkyon Learn's ADR 0021, points 5 and 10).

## Asking

```dart
final client = LlmClient();
final endpoint = LlmEndpoint(
  protocol: LlmProtocol.anthropic,
  model: 'claude-sonnet-5-5',
  apiKey: key,
);
final request = LlmRequest(
  system: 'You help with learning Japanese.',
  messages: [
    LlmMessage.user([const LlmText('What is "dog" in Japanese?')]),
  ],
  maxOutputTokens: 4096,
);
await for (final event in client.send(endpoint, request)) {
  switch (event) {
    case LlmTextDelta(:final text):
      stdout.write(text);
    case LlmFinished(:final message, :final stop, :final usage):
      // The model's whole turn, to send back in the next request.
  }
}
client.close();
```

`send` streams `LlmTextDelta`s as the model writes, then one `LlmFinished` with the
model's turn, why it stopped (`LlmStop`: `end`, `maxTokens`, `toolCalls`, `refused`,
`other`) and what the provider counted (`LlmUsage`, input with what was read from a
cache, output with the model's thinking). A failure ends the stream with an `LlmFailure`
of a kind a caller can act on (`key`, `rateLimited` with the provider's `retryAfter`,
`unavailable`, `request`, `provider`, `network`, `malformed`), the usage counted so far
and the provider's words, cut to 300 characters with anything that looks like a key
removed. Cancelling the subscription aborts the request.

### The protocols

| `LlmProtocol` | Request | Default base URL |
|---|---|---|
| `anthropic` | `POST {base}/v1/messages` | `https://api.anthropic.com` |
| `openai` | `POST {base}/chat/completions` | `https://api.openai.com/v1` |
| `openaiResponses` (`openai-responses`) | `POST {base}/responses` | `https://api.openai.com/v1` |

A base URL is `https`, but for a model on the same machine or network (`localhost`, a
private or link-local address, a single-label name such as a container's, `.local`,
`.internal`, `.home.arpa`), as Ollama and llama.cpp serve it; anything else is a
`FormatException` when the endpoint is made. `onThisMachine` says whether its requests stay on
this machine (a loopback address, or `localhost` as RFC 6761 reserves it), not merely on
the local network, which may be shared.

- **Chat completions** go to OpenAI, Mistral, Gemini's compatibility endpoint, OpenRouter
  and local servers. OpenAI, Mistral and Gemini refuse fields they don't know and differ
  in which they take, so each of those hosts, and OpenRouter, gets its own fields; every
  other host gets what Ollama, vLLM and llama.cpp all take. Replies are read leniently:
  usage at the end, with the last choice or as a running total; tool calls in pieces or
  whole, with or without an index; errors as a status, in a chunk, or lost.
- **The Responses API** is stateless here (`store: false`): each request carries the
  whole conversation, with the model's encrypted reasoning as it streamed it.
- **What goes back:** a model's turn holds, besides its text and calls, what its provider
  wants returned with the tools' results, kept as it came: Anthropic's thinking blocks
  and the Responses API's reasoning items as `LlmProviderBlock`s (sent to that protocol
  alone), Gemini's thought signatures in `LlmToolCall.extra`, the Responses API's message
  phase in `LlmText.extra`. Send the `LlmFinished.message` back unchanged.
- **The person's opaque ID** (`LlmRequest.personId`) goes where the provider takes one:
  Anthropic's `metadata.user_id`, OpenAI's `safety_identifier`, OpenRouter's `user`.
- **The effort** (`LlmRequest.effort`, `LlmEffort`: `low`, `medium`, `high`, `xhigh`,
  `max`, Anthropic's names; null leaves it to the model) goes as Anthropic's
  `output_config.effort`, the Responses API's and OpenRouter's `reasoning.effort`, and
  `reasoning_effort` to OpenAI's chat completions, Gemini and any other host.
  `endpoint.efforts` says which levels a host takes: Gemini's low to high, Mistral's
  none, every one elsewhere; `send` throws an `ArgumentError` for another. Which levels a
  model has is the model's, and its provider refuses one it lacks. The model's thinking
  counts towards `maxOutputTokens`, so a higher effort wants a higher ceiling:
  `LlmEffort.outputCeiling` (8,192 at low to 65,536 at max; 16,384 without an effort).

## Testing with it

`package:halkyon_llm/testing.dart` has `FakeLlmClient`: it keeps what it was sent, and
answers with what a test queues (`replyText`, `reply` with events and a failure, `fail`),
so code that forwards or runs a tools' loop is tested without HTTP.

## How it is tested

- **Documented replies** (`test/fixtures/documented/<protocol>/`): each provider's stream
  written event by event from its documentation, in our own words: text, tool calls,
  errors mid-stream, the statuses, and the variants each compatible service streams.
  Each is a `.http` file: `HTTP <status>`, the headers that matter, an empty line, the
  body. The tests replay them in pieces of 1, 7 and 4,096 bytes, so a line or a character
  cut in two is read whole.
- **Recorded replies** (`test/fixtures/recorded/<protocol>[@<host>]/`): real ones, which
  `tool/record_llm.dart` writes with a key of one's own (four small requests: a text, a
  tool call, the tool's result sent back with the model's whole turn, a wrong key). Their
  words are the model's, so only what every such reply must hold is checked. Read them
  before committing them.

```sh
cd packages/halkyon_llm
dart run tool/record_llm.dart --protocol anthropic
dart run tool/record_llm.dart --protocol openai-responses --model <a small model>
```

## Licence

BSD-3-Clause (`LICENSE`), as Serverpod's own and `halkyon_oidc`, so that others can use
it; the rest of Halkyon is AGPL. Not on pub.dev yet.