halkyon/packages/halkyon_llm/README.md

halkyon_llm

Language models' APIs in pure Dart, streamed, over plain HTTP, no SDKs: Anthropic's Messages API, OpenAI's chat completions (and every service compatible with them), and OpenAI's Responses API, behind one interface of what they share: a system prompt and messages, text as the model writes it, images, and tool calls. Halkyon's server forwards the operator's assistants with it, and its apps call a person's own provider with it (Halkyon Learn's ADR 0021, points 5 and 10).

Asking

final client = LlmClient();
final endpoint = LlmEndpoint(
  protocol: LlmProtocol.anthropic,
  model: 'claude-sonnet-5-5',
  apiKey: key,
);
final request = LlmRequest(
  system: 'You help with learning Japanese.',
  messages: [
    LlmMessage.user([const LlmText('What is "dog" in Japanese?')]),
  ],
  maxOutputTokens: 4096,
);
await for (final event in client.send(endpoint, request)) {
  switch (event) {
    case LlmTextDelta(:final text):
      stdout.write(text);
    case LlmFinished(:final message, :final stop, :final usage):
      // The model's whole turn, to send back in the next request.
  }
}
client.close();

send streams LlmTextDeltas as the model writes, then one LlmFinished with the model's turn, why it stopped (LlmStop: end, maxTokens, toolCalls, refused, other) and what the provider counted (LlmUsage, input with what was read from a cache, output with the model's thinking). A failure ends the stream with an LlmFailure of a kind a caller can act on (key, rateLimited with the provider's retryAfter, unavailable, request, provider, network, malformed), the usage counted so far and the provider's words, cut to 300 characters with anything that looks like a key removed. Cancelling the subscription aborts the request.

The protocols

LlmProtocol	Request	Default base URL
anthropic	POST {base}/v1/messages	https://api.anthropic.com
openai	POST {base}/chat/completions	https://api.openai.com/v1
openaiResponses (openai-responses)	POST {base}/responses	https://api.openai.com/v1

A base URL is https, but for a model on the same machine or network (localhost, a private or link-local address, a single-label name such as a container's, .local, .internal, .home.arpa), as Ollama and llama.cpp serve it; anything else is a FormatException when the endpoint is made. onThisMachine says whether its requests stay on this machine (a loopback address, or localhost as RFC 6761 reserves it), not merely on the local network, which may be shared.

• Chat completions go to OpenAI, Mistral, Gemini's compatibility endpoint, OpenRouter and local servers. OpenAI, Mistral and Gemini refuse fields they don't know and differ in which they take, so each of those hosts, and OpenRouter, gets its own fields; every other host gets what Ollama, vLLM and llama.cpp all take. Replies are read leniently: usage at the end, with the last choice or as a running total; tool calls in pieces or whole, with or without an index; errors as a status, in a chunk, or lost.
• The Responses API is stateless here (store: false): each request carries the whole conversation, with the model's encrypted reasoning as it streamed it.
• What goes back: a model's turn holds, besides its text and calls, what its provider wants returned with the tools' results, kept as it came: Anthropic's thinking blocks and the Responses API's reasoning items as LlmProviderBlocks (sent to that protocol alone), Gemini's thought signatures in LlmToolCall.extra, the Responses API's message phase in LlmText.extra. Send the LlmFinished.message back unchanged.
• The person's opaque ID (LlmRequest.personId) goes where the provider takes one: Anthropic's metadata.user_id, OpenAI's safety_identifier, OpenRouter's user.
• The effort (LlmRequest.effort, LlmEffort: low, medium, high, xhigh, max, Anthropic's names; null leaves it to the model) goes as Anthropic's output_config.effort, the Responses API's and OpenRouter's reasoning.effort, and reasoning_effort to OpenAI's chat completions, Gemini and any other host. endpoint.efforts says which levels a host takes: Gemini's low to high, Mistral's none, every one elsewhere; send throws an ArgumentError for another. Which levels a model has is the model's, and its provider refuses one it lacks. The model's thinking counts towards maxOutputTokens, so a higher effort wants a higher ceiling: LlmEffort.outputCeiling (8,192 at low to 65,536 at max; 16,384 without an effort).

Testing with it

package:halkyon_llm/testing.dart has FakeLlmClient: it keeps what it was sent, and answers with what a test queues (replyText, reply with events and a failure, fail), so code that forwards or runs a tools' loop is tested without HTTP.

How it is tested

• Documented replies (test/fixtures/documented/<protocol>/): each provider's stream written event by event from its documentation, in our own words: text, tool calls, errors mid-stream, the statuses, and the variants each compatible service streams. Each is a .http file: HTTP <status>, the headers that matter, an empty line, the body. The tests replay them in pieces of 1, 7 and 4,096 bytes, so a line or a character cut in two is read whole.
• Recorded replies (test/fixtures/recorded/<protocol>[@<host>]/): real ones, which tool/record_llm.dart writes with a key of one's own (four small requests: a text, a tool call, the tool's result sent back with the model's whole turn, a wrong key). Their words are the model's, so only what every such reply must hold is checked. Read them before committing them.

cd packages/halkyon_llm
dart run tool/record_llm.dart --protocol anthropic
dart run tool/record_llm.dart --protocol openai-responses --model <a small model>

Licence

BSD-3-Clause (LICENSE), as Serverpod's own and halkyon_oidc, so that others can use it; the rest of Halkyon is AGPL. Not on pub.dev yet.