halkyon-days/halkyon
Watch
1
0
Fork
You've already forked halkyon
 0
Code Issues Pull requests Projects Releases 37 Packages Wiki Activity Actions

feat(llm): halkyon_llm, language models' APIs streamed over plain HTTP (ADR 0021, step 2)

Browse source 
A pure Dart package, no SDKs, one interface (LlmClient) over Anthropic's
Messages API, OpenAI's chat completions, in each known host's dialect and
the local servers' (D45), and OpenAI's Responses API, stateless (D46). A
model's turn keeps what its provider wants back with the tools' results:
Anthropic's thinking blocks, the Responses API's reasoning items and
message phase, Gemini's thought signatures. Failures come by kind, the
provider's words cut and without keys.

Tested on replies written from the providers' documentation, replayed in
pieces of 1, 7 and 4,096 bytes; tool/record_llm.dart records real ones
with a key of one's own, which the tests then run on too (D36).
FakeLlmClient for the code that uses it. BSD-3-Clause, as halkyon_oidc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-25 17:38:53 +01:00
commit 038aca5439
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
53 changed files with 4841 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

6
packages/halkyon_llm/CHANGELOG.md Normal file
View file

@ -0,0 +1,6 @@
## 0.1.0
* One interface (`LlmClient`) over Anthropic's Messages API, OpenAI's chat completions
(with each known host's fields, and the local servers') and OpenAI's Responses API,
streamed over plain HTTP; failures by kind; `FakeLlmClient` for tests; the recording
tool.

28
packages/halkyon_llm/LICENSE Normal file
View file

@ -0,0 +1,28 @@
BSD 3-Clause License
Copyright (c) 2026, the Project Halkyon authors
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

104
packages/halkyon_llm/README.md Normal file
View file

@ -0,0 +1,104 @@
# halkyon_llm
Language models' APIs in pure Dart, streamed, over plain HTTP, no SDKs: Anthropic's
Messages API, OpenAI's chat completions (and every service compatible with them), and
OpenAI's Responses API, behind one interface of what they share: a system prompt and
messages, text as the model writes it, images, and tool calls. Halkyon's server forwards
the operator's assistants with it, and its apps call a person's own provider with it
(Halkyon Learn's ADR 0021, points 5 and 10).
## Asking
```dart
final client = LlmClient();
final endpoint = LlmEndpoint(
protocol: LlmProtocol.anthropic,
model: 'claude-sonnet-5-5',
apiKey: key,
);
final request = LlmRequest(
system: 'You help with learning Japanese.',
messages: [
LlmMessage.user([const LlmText('What is "dog" in Japanese?')]),
],
maxOutputTokens: 4096,
);
await for (final event in client.send(endpoint, request)) {
switch (event) {
case LlmTextDelta(:final text):
stdout.write(text);
case LlmFinished(:final message, :final stop, :final usage):
// The model's whole turn, to send back in the next request.
}
}
client.close();
```
`send` streams `LlmTextDelta`s as the model writes, then one `LlmFinished` with the
model's turn, why it stopped (`LlmStop`: `end`, `maxTokens`, `toolCalls`, `refused`,
`other`) and what the provider counted (`LlmUsage`, input with what was read from a
cache, output with the model's thinking). A failure ends the stream with an `LlmFailure`
of a kind a caller can act on (`key`, `rateLimited` with the provider's `retryAfter`,
`unavailable`, `request`, `provider`, `network`, `malformed`), the usage counted so far
and the provider's words, cut to 300 characters with anything that looks like a key
removed. Cancelling the subscription aborts the request.
### The protocols
| `LlmProtocol` | Request | Default base URL |
|---|---|---|
| `anthropic` | `POST {base}/v1/messages` | `https://api.anthropic.com` |
| `openai` | `POST {base}/chat/completions` | `https://api.openai.com/v1` |
| `openaiResponses` (`openai-responses`) | `POST {base}/responses` | `https://api.openai.com/v1` |
A base URL is `https`, but for a model on the same machine or network (`localhost`, a
private or link-local address, a single-label name such as a container's, `.local`,
`.internal`, `.home.arpa`), as Ollama and llama.cpp serve it; anything else is a
`FormatException` when the endpoint is made.
- **Chat completions** go to OpenAI, Mistral, Gemini's compatibility endpoint, OpenRouter
and local servers. OpenAI, Mistral and Gemini refuse fields they don't know and differ
in which they take, so each of those hosts, and OpenRouter, gets its own fields; every
other host gets what Ollama, vLLM and llama.cpp all take. Replies are read leniently:
usage at the end, with the last choice or as a running total; tool calls in pieces or
whole, with or without an index; errors as a status, in a chunk, or lost.
- **The Responses API** is stateless here (`store: false`): each request carries the
whole conversation, with the model's encrypted reasoning as it streamed it.
- **What goes back:** a model's turn holds, besides its text and calls, what its provider
wants returned with the tools' results, kept as it came: Anthropic's thinking blocks
and the Responses API's reasoning items as `LlmProviderBlock`s (sent to that protocol
alone), Gemini's thought signatures in `LlmToolCall.extra`, the Responses API's message
phase in `LlmText.extra`. Send the `LlmFinished.message` back unchanged.
- **The person's opaque ID** (`LlmRequest.personId`) goes where the provider takes one:
Anthropic's `metadata.user_id`, OpenAI's `safety_identifier`, OpenRouter's `user`.
## Testing with it
`package:halkyon_llm/testing.dart` has `FakeLlmClient`: it keeps what it was sent, and
answers with what a test queues (`replyText`, `reply` with events and a failure, `fail`),
so code that forwards or runs a tools' loop is tested without HTTP.
## How it is tested
- **Documented replies** (`test/fixtures/documented/<protocol>/`): each provider's stream
written event by event from its documentation, in our own words: text, tool calls,
errors mid-stream, the statuses, and the variants each compatible service streams.
Each is a `.http` file: `HTTP <status>`, the headers that matter, an empty line, the
body. The tests replay them in pieces of 1, 7 and 4,096 bytes, so a line or a character
cut in two is read whole.
- **Recorded replies** (`test/fixtures/recorded/<protocol>[@<host>]/`): real ones, which
`tool/record_llm.dart` writes with a key of one's own (four small requests: a text, a
tool call, the tool's result sent back with the model's whole turn, a wrong key). Their
words are the model's, so only what every such reply must hold is checked. Read them
before committing them.
```sh
cd packages/halkyon_llm
dart run tool/record_llm.dart --protocol anthropic
dart run tool/record_llm.dart --protocol openai-responses --model <a small model>
```
## Licence
BSD-3-Clause (`LICENSE`), as Serverpod's own and `halkyon_oidc`, so that others can use
it; the rest of Halkyon is AGPL. Not on pub.dev yet.

5
packages/halkyon_llm/analysis_options.yaml Normal file
View file

@ -0,0 +1,5 @@
# The workspace's lint rules (see ../../analysis_options.yaml).
include: ../../analysis_options.yaml
formatter:
trailing_commas: preserve

11
packages/halkyon_llm/lib/halkyon_llm.dart Normal file
View file

@ -0,0 +1,11 @@
/// Language models' APIs in pure Dart, streamed, over plain HTTP (ADR 0021,
/// point 5): Anthropic's Messages API and OpenAI's chat completions, and
/// every service compatible with it, behind one interface of what both
/// share: a system prompt and messages, text as it is written, images, and
/// tool calls. Nothing here keeps or logs what is asked or answered.
library;
export 'src/client.dart' show HttpLlmClient, LlmClient;
export 'src/conversation.dart';
export 'src/endpoint.dart';
export 'src/failure.dart';

337
packages/halkyon_llm/lib/src/anthropic.dart Normal file
View file

@ -0,0 +1,337 @@
import 'dart:collection';
import 'dart:convert';
import 'package:halkyon_llm/src/client.dart';
import 'package:halkyon_llm/src/conversation.dart';
import 'package:halkyon_llm/src/endpoint.dart';
import 'package:halkyon_llm/src/failure.dart';
import 'package:halkyon_llm/src/sse.dart';
/// Anthropic's Messages API, streamed
/// (https://platform.claude.com/docs/en/build-with-claude/streaming).
///
/// The request leaves `thinking` unset: Claude v5's models then think as
/// they see fit and stream each thinking block empty but for its signature
/// (`display` defaults to `omitted` there), and older ones don't think. The
/// blocks come back as [LlmProviderBlock]s, which must be sent back
/// unchanged with the tools' results.
final class AnthropicAdapter(
/// Where it sends.
final LlmEndpoint endpoint,
) implements ProtocolAdapter {
/// The API's version this adapter speaks, the only one there is
/// (https://platform.claude.com/docs/en/api/versioning).
static const apiVersion = '2023-06-01';
/// Whether this runs in a browser, where Anthropic answers a request
/// straight from the page only with the header saying it is meant
/// (a person's own key, which is theirs to expose).
static const _inBrowser = bool.fromEnvironment('dart.library.js_interop');
@override
Uri get url =>
endpoint.baseUrl.replace(path: '${endpoint.baseUrl.path}/v1/messages');
@override
Map<String, String> get headers => {
if (endpoint.apiKey.isNotEmpty) 'x-api-key': endpoint.apiKey,
'anthropic-version': apiVersion,
if (_inBrowser) 'anthropic-dangerous-direct-browser-access': 'true',
};
@override
Map<String, Object?> body(LlmRequest request) => {
'model': endpoint.model,
'max_tokens': request.maxOutputTokens,
'stream': true,
if (request.system.isNotEmpty) 'system': request.system,
'messages': [
for (final message in request.messages)
if (_content(message) case final content when content.isNotEmpty)
{'role': message.role.name, 'content': content},
],
if (request.tools.isNotEmpty)
'tools': [
for (final tool in request.tools)
{
'name': tool.name,
'description': tool.description,
'input_schema': tool.inputSchema,
},
],
if (request.personId case final personId?)
'metadata': {'user_id': personId},
};
/// A message's blocks: tools' results first, as Anthropic requires them
/// before any text of the same turn; empty text left out, as it refuses
/// empty text blocks; other protocols' blocks dropped.
static List<Map<String, Object?>> _content(LlmMessage message) => [
for (final part in message.parts.whereType<LlmToolResult>())
{
'type': 'tool_result',
'tool_use_id': part.callId,
'content': part.content,
if (part.isError) 'is_error': true,
},
for (final part in message.parts)
...switch (part) {
LlmText(:final text) when text.isNotEmpty => [
{'type': 'text', 'text': text},
],
LlmImage(:final mediaType, :final bytes) => [
{
'type': 'image',
'source': {
'type': 'base64',
'media_type': mediaType,
'data': base64Encode(bytes),
},
},
],
LlmToolCall(:final id, :final name, :final input) => [
{'type': 'tool_use', 'id': id, 'name': name, 'input': input},
],
LlmProviderBlock(protocol: LlmProtocol.anthropic, :final block) => [
block,
],
_ => const <Map<String, Object?>>[],
},
];
@override
ReplyReader reader() => _AnthropicReader();
@override
LlmFailure failureOf(
int status,
Map<String, String> headers,
List<int> body,
) {
final text = utf8.decode(body, allowMalformed: true);
final error = _errorOf(text);
final failure = failureOfStatus(
status,
headers,
error?['message'] as String? ?? text,
);
final kind = _kindOf(error?['type']);
return kind == null
? failure
: LlmFailure(
kind: kind,
detail: failure.detail,
status: status,
retryAfter: failure.retryAfter,
);
}
static Map<String, Object?>? _errorOf(String text) {
try {
return jsonObject(text)['error'] as Map<String, Object?>?;
} on Object {
return null;
}
}
}
/// The kind of failure Anthropic's error type stands for
/// (https://platform.claude.com/docs/en/api/errors); null for a type it
/// may add later, which its HTTP status then decides.
LlmFailureKind? _kindOf(Object? type) => switch (type) {
'authentication_error' ||
'permission_error' ||
'billing_error' => LlmFailureKind.key,
'rate_limit_error' => LlmFailureKind.rateLimited,
'overloaded_error' => LlmFailureKind.unavailable,
'api_error' || 'timeout_error' => LlmFailureKind.provider,
'invalid_request_error' ||
'not_found_error' ||
'request_too_large' => LlmFailureKind.request,
_ => null,
};
final class _AnthropicReader() implements ReplyReader {
/// The blocks begun, by their index, which is their place in the answer.
final _open = <int, _Block>{};
/// The blocks ended, by their index.
final _parts = SplayTreeMap<int, LlmPart>();
var _inputTokens = 0;
var _outputTokens = 0;
String? _stopReason;
@override
bool finished = false;
@override
LlmUsage get usage =>
LlmUsage(inputTokens: _inputTokens, outputTokens: _outputTokens);
@override
Iterable<LlmEvent> read(SseEvent event) sync* {
final data = jsonObject(event.data);
switch (data) {
case {
'type': 'message_start',
'message': final Map<String, Object?> message,
}:
_count(message['usage']);
case {
'type': 'content_block_start',
'index': final int index,
'content_block': final Map<String, Object?> block,
}:
_open[index] = _Block(Map.of(block));
if (block case {'type': 'text', 'text': final String text}
when text.isNotEmpty) {
yield LlmTextDelta(text);
}
case {
'type': 'content_block_delta',
'index': final int index,
'delta': final Map<String, Object?> delta,
}:
final block = _open[index];
if (block == null) {
throw const FormatException('A delta of no block');
}
switch (delta) {
case {'type': 'text_delta', 'text': final String text}:
block.text.write(text);
yield LlmTextDelta(text);
case {'type': 'input_json_delta', 'partial_json': final String json}:
block.json.write(json);
case {'type': 'thinking_delta', 'thinking': final String thinking}:
block.text.write(thinking);
case {'type': 'signature_delta', 'signature': final String signature}:
block.signature = signature;
}
case {'type': 'content_block_stop', 'index': final int index}:
final block = _open.remove(index);
if (block == null) {
throw const FormatException('The end of no block');
}
if (block.part() case final part?) {
_parts[index] = part;
}
case {'type': 'message_delta'}:
if (data['delta'] case {'stop_reason': final String reason}) {
_stopReason = reason;
}
_count(data['usage']);
case {'type': 'message_stop'}:
finished = true;
case {'type': 'error'}:
final error = data['error'];
throw LlmFailure(
kind:
_kindOf(error is Map ? error['type'] : null) ??
LlmFailureKind.provider,
detail: redacted(switch (error) {
{'message': final String message} => message,
_ => 'An error',
}),
usage: usage,
);
case {
'type':
'message_start' ||
'content_block_start' ||
'content_block_delta' ||
'content_block_stop',
}:
throw FormatException('A ${event.type} of another shape');
}
}
/// Counts a `usage` object: the input with what was written to and read
/// from the cache; the output's tokens as they stand, which
/// `message_delta` gives as a running total.
void _count(Object? usage) {
if (usage case {'input_tokens': final int input}) {
_inputTokens =
input +
_tokens(usage['cache_creation_input_tokens']) +
_tokens(usage['cache_read_input_tokens']);
}
if (usage case {'output_tokens': final int output}) {
_outputTokens = output;
}
}
static int _tokens(Object? count) => count is int ? count : 0;
@override
LlmFinished end() {
if (!finished) {
throw LlmFailure(
kind: LlmFailureKind.malformed,
detail: 'The reply ended before the answer did',
usage: usage,
);
}
return LlmFinished(
message: LlmMessage.assistant(
_parts.isEmpty ? const [LlmText('')] : _parts.values.toList(),
),
stop: switch (_stopReason) {
'end_turn' || 'stop_sequence' => LlmStop.end,
'max_tokens' || 'model_context_window_exceeded' => LlmStop.maxTokens,
'tool_use' => LlmStop.toolCalls,
'refusal' => LlmStop.refused,
_ => LlmStop.other,
},
usage: usage,
);
}
}
/// A block being streamed.
final class _Block(
/// The block as `content_block_start` gave it.
final Map<String, Object?> start,
) {
/// A text block's text, or a thinking block's thinking.
final text = StringBuffer();
/// A tool call's input, in pieces of JSON.
final json = StringBuffer();
/// A thinking block's signature.
String? signature;
/// The part the block is, once it ended; null for a kind this adapter
/// neither shows nor sends back.
LlmPart? part() => switch (start) {
{'type': 'text'} => LlmText('${start['text'] ?? ''}$text'),
{'type': 'tool_use', 'id': final String id, 'name': final String name} =>
LlmToolCall(
id: id,
name: name,
input: json.isEmpty
? switch (start['input']) {
final Map<String, Object?> input => Map.of(input),
_ => const {},
}
: jsonObject(json.toString()),
),
{'type': 'tool_use'} => throw const FormatException(
'A tool call without its name',
),
{'type': 'thinking'} => LlmProviderBlock(
protocol: LlmProtocol.anthropic,
block: {
'type': 'thinking',
'thinking': '${start['thinking'] ?? ''}$text',
'signature': signature ?? start['signature'] ?? '',
},
),
{'type': 'redacted_thinking'} => LlmProviderBlock(
protocol: LlmProtocol.anthropic,
block: start,
),
_ => null,
};
}

268
packages/halkyon_llm/lib/src/client.dart Normal file
View file

@ -0,0 +1,268 @@
import 'dart:async';
import 'dart:convert';
import 'package:halkyon_llm/src/anthropic.dart';
import 'package:halkyon_llm/src/conversation.dart';
import 'package:halkyon_llm/src/endpoint.dart';
import 'package:halkyon_llm/src/failure.dart';
import 'package:halkyon_llm/src/openai.dart';
import 'package:halkyon_llm/src/openai_responses.dart';
import 'package:halkyon_llm/src/sse.dart';
import 'package:http/http.dart' as http;
/// Sends requests to assistants, each in its endpoint's protocol, the answer
/// streamed.
abstract interface class LlmClient {
/// A client over HTTP, on [httpClient] or one of its own.
factory({http.Client? httpClient}) = HttpLlmClient;
/// Sends [request] to [endpoint]: [LlmTextDelta]s as the model writes,
/// then [LlmFinished], or an [LlmFailure] as the stream's error. Cancelling
/// the subscription aborts the request.
Stream<LlmEvent> send(LlmEndpoint endpoint, LlmRequest request);
/// Closes the HTTP client, where it is the client's own.
void close();
}
/// [LlmClient] over HTTP: the protocols' adapters, no SDKs.
final class HttpLlmClient({
http.Client? httpClient,
/// How long a reply may take to begin (a local model may first have to
/// load).
final Duration headersTimeout = const Duration(minutes: 2),
/// How long a reply may stay silent (a model may think for minutes before
/// it writes).
final Duration idleTimeout = const Duration(minutes: 5),
}) implements LlmClient {
final http.Client _http = httpClient ?? http.Client();
final bool _ownsHttp = httpClient == null;
@override
Stream<LlmEvent> send(LlmEndpoint endpoint, LlmRequest request) {
final ProtocolAdapter adapter = switch (endpoint.protocol) {
LlmProtocol.anthropic => AnthropicAdapter(endpoint),
LlmProtocol.openai => OpenAiAdapter(endpoint),
LlmProtocol.openaiResponses => OpenAiResponsesAdapter(endpoint),
};
final abort = Completer<void>();
late final StreamController<LlmEvent> events;
events = StreamController<LlmEvent>(
onListen: () => unawaited(_run(adapter, request, abort, events)),
onCancel: () {
if (!abort.isCompleted) {
abort.complete();
}
},
);
return events.stream;
}
Future<void> _run(
ProtocolAdapter adapter,
LlmRequest request,
Completer<void> abort,
StreamController<LlmEvent> events,
) async {
final reader = adapter.reader();
try {
final httpRequest =
http.AbortableRequest(
'POST',
adapter.url,
abortTrigger: abort.future,
)
..headers.addAll(adapter.headers)
..headers['content-type'] = 'application/json'
..headers['accept'] = 'text/event-stream'
..bodyBytes = utf8.encode(jsonEncode(adapter.body(request)));
final response = await _http
.send(httpRequest)
.timeout(
headersTimeout,
onTimeout: () => throw _silent(headersTimeout),
);
if (response.statusCode != 200) {
final body = await http.ByteStream(
response.stream.timeout(
idleTimeout,
onTimeout: (sink) => sink.close(),
),
).toBytes();
throw adapter.failureOf(response.statusCode, response.headers, body);
}
// Silence is timed on the bytes, before they are read: a reader
// cancelled while it waits for them would never finish cancelling.
final bytes = response.stream.timeout(
idleTimeout,
onTimeout: (sink) => sink
..addError(_silent(idleTimeout))
..close(),
);
await for (final event in parseSse(bytes)) {
for (final out in reader.read(event)) {
if (abort.isCompleted) {
return;
}
events.add(out);
}
if (reader.finished) {
break;
}
}
final finished = reader.end();
if (!abort.isCompleted) {
events.add(finished);
}
} on LlmFailure catch (failure) {
if (!abort.isCompleted) {
events.addError(failure.withUsage(reader.usage));
}
} on http.RequestAbortedException {
// The listener cancelled: nobody is left to tell.
} on http.ClientException catch (error) {
if (!abort.isCompleted) {
events.addError(
LlmFailure(
kind: LlmFailureKind.network,
detail: error.message,
usage: reader.usage,
),
);
}
} on FormatException catch (error) {
if (!abort.isCompleted) {
events.addError(
LlmFailure(
kind: LlmFailureKind.malformed,
detail: 'Unreadable reply: ${error.message}',
usage: reader.usage,
),
);
}
} on Object catch (error) {
// A reply of another shape than its protocol's: a field missing or of
// another type. Only the error's type is kept, as its text may quote
// the reply.
if (!abort.isCompleted) {
events.addError(
LlmFailure(
kind: LlmFailureKind.malformed,
detail: 'Unreadable reply: ${error.runtimeType}',
usage: reader.usage,
),
);
}
} finally {
if (!events.isClosed) {
await events.close();
}
}
}
LlmFailure _silent(Duration after) => LlmFailure(
kind: LlmFailureKind.network,
detail: 'No reply for ${after.inSeconds} s',
);
@override
void close() {
if (_ownsHttp) {
_http.close();
}
}
}
/// What a protocol's adapter provides [HttpLlmClient] with.
abstract interface class ProtocolAdapter() {
/// Where requests go.
Uri get url;
/// The request's headers but its content's type: the key, the version.
Map<String, String> get headers;
/// The request's JSON body, to stream the answer.
Map<String, Object?> body(LlmRequest request);
/// A reader of one reply's events.
ReplyReader reader();
/// The failure a reply other than `200 OK` stands for.
LlmFailure failureOf(int status, Map<String, String> headers, List<int> body);
}
/// Reads one reply's server-sent events into [LlmEvent]s.
abstract interface class ReplyReader() {
/// The events [event] stands for; throws [LlmFailure] for an error the
/// provider streams.
Iterable<LlmEvent> read(SseEvent event);
/// Whether the reply said it is over, so nothing after counts.
bool get finished;
/// The answer, once the stream is over; throws [LlmFailure] if it ended
/// before the answer did.
LlmFinished end();
/// What the provider counted so far.
LlmUsage get usage;
}
/// [LlmFailure] with what was counted before it, for a failure thrown before
/// the reader knew.
extension on LlmFailure {
LlmFailure withUsage(LlmUsage counted) => usage != LlmUsage.none
? this
: LlmFailure(
kind: kind,
detail: detail,
status: status,
retryAfter: retryAfter,
usage: counted,
);
}
/// The failure a reply's [status] stands for, with the provider's [detail]
/// and its `retry-after` header.
LlmFailure failureOfStatus(
int status,
Map<String, String> headers,
String detail,
) => LlmFailure(
kind: switch (status) {
401 || 402 || 403 => LlmFailureKind.key,
429 => LlmFailureKind.rateLimited,
503 || 529 => LlmFailureKind.unavailable,
>= 500 => LlmFailureKind.provider,
_ => LlmFailureKind.request,
},
detail: redacted(detail),
status: status,
retryAfter: _retryAfter(headers['retry-after']),
);
Duration? _retryAfter(String? header) {
final seconds = int.tryParse(header?.trim() ?? '');
return seconds == null || seconds < 0 ? null : Duration(seconds: seconds);
}
/// A provider's words fit for the logs: at most 300 characters, and what
/// looks like a key (OpenAI quotes part of a wrong one) cut.
String redacted(String detail) {
final cut = detail.replaceAllMapped(
RegExp(r'\b(sk|key)-[A-Za-z0-9_*\-]{4,}'),
(match) => '${match[1]}-…',
);
return cut.length <= 300 ? cut : '${cut.substring(0, 300)}…';
}
/// The JSON object [text] holds, or a [FormatException].
Map<String, Object?> jsonObject(String text) {
final value = jsonDecode(text);
if (value is! Map<String, Object?>) {
throw const FormatException('Not a JSON object');
}
return value;
}

254
packages/halkyon_llm/lib/src/conversation.dart Normal file
View file

@ -0,0 +1,254 @@
import 'dart:typed_data';
import 'package:halkyon_llm/src/endpoint.dart';
import 'package:halkyon_llm/src/failure.dart';
/// Who said a [LlmMessage]: the person (with what their app adds, such as
/// tools' results), or the model.
enum LlmRole() {
user,
assistant,
}
/// One turn of a conversation: its [role] and its [parts], in order.
///
/// What a role may hold is what both protocols allow: a person's turn holds
/// text, images and tools' results; the model's turn holds text, tool calls
/// and the blocks only its provider reads. Anything else is refused as the
/// message is made, not when a provider refuses it.
final class LlmMessage._(
/// Who said it.
final LlmRole role,
List<LlmPart> parts,
bool Function(LlmPart) allowed,
) {
this {
if (parts.isEmpty) {
throw ArgumentError.value(parts, 'parts', 'A message needs a part');
}
for (final part in parts) {
if (!allowed(part)) {
throw ArgumentError.value(
part,
'parts',
'Not in a turn of ${role.name}',
);
}
}
}
/// A person's turn: text, images, and the results of the model's tool
/// calls.
new user(List<LlmPart> parts)
: this._(
LlmRole.user,
parts,
(part) => part is! LlmToolCall && part is! LlmProviderBlock,
);
/// The model's turn, as [LlmFinished] hands it back: text, tool calls and
/// its provider's own blocks.
new assistant(List<LlmPart> parts)
: this._(
LlmRole.assistant,
parts,
(part) => part is! LlmImage && part is! LlmToolResult,
);
/// What it holds, in order.
final List<LlmPart> parts = List.unmodifiable(parts);
/// The text parts, joined: what a person reads of the model's turn.
String get text => parts.whereType<LlmText>().map((part) => part.text).join();
/// The tool calls the model made in this turn, in order.
List<LlmToolCall> get toolCalls => parts.whereType<LlmToolCall>().toList();
}
/// A part of a [LlmMessage].
sealed class const LlmPart();
/// Text.
final class const LlmText(
/// The text.
final String text, {
/// What the provider gave with the text of the model's turn to have back
/// with it: the Responses API's `phase` of a message (`commentary` before
/// tool calls, `final_answer`), which its newest models want returned.
final Map<String, Object?> extra = const {},
}) extends LlmPart {
@override
String toString() => 'LlmText(${text.length} characters)';
}
/// An image a person adds, for an assistant that reads images (ADR 0021,
/// D12).
final class const LlmImage({
/// Its type: `image/jpeg`, `image/png`, `image/gif` or `image/webp`, which
/// both protocols' providers take.
required final String mediaType,
/// The image's bytes.
required final Uint8List bytes,
}) extends LlmPart {
@override
String toString() => 'LlmImage($mediaType, ${bytes.length} bytes)';
}
/// A call of one of the request's tools, as the model made it.
final class const LlmToolCall({
/// The provider's ID of the call, which its result names.
required final String id,
/// The tool's name.
required final String name,
/// What the model passes, as the tool's schema describes it.
required final Map<String, Object?> input,
/// What the provider gave with the call to have back with it, as it gave
/// it: Gemini's `extra_content`, whose thought signature its models want
/// returned.
final Map<String, Object?> extra = const {},
}) extends LlmPart {
@override
String toString() => 'LlmToolCall($id, $name)';
}
/// What the app answers to a [LlmToolCall].
final class const LlmToolResult({
/// The call's [LlmToolCall.id].
required final String callId,
/// The answer, as text.
required final String content,
/// Whether the tool failed, [content] then saying why.
final bool isError = false,
}) extends LlmPart {
@override
String toString() => 'LlmToolResult($callId${isError ? ', error' : ''})';
}
/// A block of the model's turn that only its provider reads, kept as it came
/// and sent back to that provider alone: Anthropic's `thinking` and
/// `redacted_thinking` blocks, which its models write by default, and the
/// Responses API's encrypted `reasoning` items (D46); both must come back
/// unchanged with the tools' results.
final class const LlmProviderBlock({
/// The protocol it came in, which alone gets it back.
required final LlmProtocol protocol,
/// The block, as the provider wrote it.
required final Map<String, Object?> block,
}) extends LlmPart {
@override
String toString() => 'LlmProviderBlock(${protocol.name}, ${block['type']})';
}
/// A tool the model may call, which the app runs (ADR 0021, point 7).
final class const LlmTool({
/// Its name: letters, digits, `_` and `-`, 1 to 64 of them, which both
/// protocols take.
required final String name,
/// What it does, for the model.
required final String description,
/// Its input, as a JSON Schema of an object.
required final Map<String, Object?> inputSchema,
});
/// One request: the conversation so far and what the model may do.
final class const LlmRequest({
/// The conversation, ending with a person's turn.
required final List<LlmMessage> messages,
/// The ceiling on the answer's tokens, which also bounds what one request
/// may cost; a model's thinking counts towards it.
required final int maxOutputTokens,
/// What the model is told before the conversation; empty: nothing.
final String system = '',
/// The tools it may call.
final List<LlmTool> tools = const [],
/// An opaque ID of the person who asks, which the provider may use against
/// one person's abuse (ADR 0021, D30); 64 characters at most, never their
/// handle. Null with a personal key, whose provider knows its owner.
final String? personId,
});
/// The stream of an answer: [LlmTextDelta]s as the model writes, then one
/// [LlmFinished]. A failure ends it with an [LlmFailure] instead.
sealed class const LlmEvent();
/// More of the answer's text, as the model writes it.
final class const LlmTextDelta(
/// The text that follows what came before.
final String text,
) extends LlmEvent {
@override
String toString() => 'LlmTextDelta(${text.length} characters)';
}
/// The answer, whole: the last event of a stream.
final class const LlmFinished({
/// The model's turn as it is to be sent back: its text, tool calls and its
/// provider's own blocks, in the order written.
required final LlmMessage message,
/// Why the model stopped.
required final LlmStop stop,
/// What the provider counted.
required final LlmUsage usage,
}) extends LlmEvent {
@override
String toString() => 'LlmFinished(${stop.name}, $usage)';
}
/// Why the model stopped.
enum LlmStop() {
/// It finished its answer.
end,
/// It reached the request's ceiling on tokens, or the model's context.
maxTokens,
/// It calls tools, whose results the app sends in the next request.
toolCalls,
/// It declined to answer, or the provider's filter stopped it.
refused,
/// Something else the provider names.
other,
}
/// What a provider counted for one request, which limits and figures use
/// (ADR 0021, point 8).
final class const LlmUsage({
/// The request's tokens, those read from a provider's cache included.
required final int inputTokens,
/// The answer's tokens, a model's thinking included.
required final int outputTokens,
}) {
/// Nothing counted.
static const none = LlmUsage(inputTokens: 0, outputTokens: 0);
@override
bool operator ==(Object other) =>
other is LlmUsage &&
other.inputTokens == inputTokens &&
other.outputTokens == outputTokens;
@override
int get hashCode => Object.hash(inputTokens, outputTokens);
@override
String toString() => 'LlmUsage(in $inputTokens, out $outputTokens)';
}

112
packages/halkyon_llm/lib/src/endpoint.dart Normal file
View file

@ -0,0 +1,112 @@
/// The protocols the adapters speak (ADR 0021, point 5, and D46).
enum LlmProtocol(
/// Its name in settings (`ASSISTANT_<ID>_PROTOCOL`).
final String key,
/// The provider's own address, where an endpoint names none.
final String defaultBaseUrl,
) {
/// Anthropic's Messages API (`POST {base}/v1/messages`). Its base URL is
/// without `/v1`, as Anthropic's SDKs take it (`https://api.anthropic.com`).
anthropic('anthropic', 'https://api.anthropic.com'),
/// OpenAI's chat completions (`POST {base}/chat/completions`), which
/// OpenAI-compatible services speak too: Mistral, Gemini's compatibility
/// endpoint, OpenRouter, and local models through Ollama, llama.cpp or
/// vLLM. Its base URL is with `/v1`, as OpenAI's SDKs take it and as those
/// services give theirs (`https://api.openai.com/v1`).
openai('openai', 'https://api.openai.com/v1'),
/// OpenAI's Responses API (`POST {base}/responses`), its current one, on
/// which alone its newest models call tools (D46).
openaiResponses('openai-responses', 'https://api.openai.com/v1');
/// The protocol of this [key], or null.
static LlmProtocol? named(String key) {
for (final protocol in values) {
if (protocol.key == key) {
return protocol;
}
}
return null;
}
}
/// Where one assistant is reached: an operator's (ADR 0021, point 4) or a
/// person's own (point 10).
///
/// A base URL other than `https` is refused (a [FormatException]), but for a
/// model on the same machine or network (`localhost`, a private or
/// link-local address, a container's name), as Ollama and llama.cpp serve.
final class LlmEndpoint({
/// The protocol it speaks.
required final LlmProtocol protocol,
/// The model's name at the provider (`claude-sonnet-5-5`).
required final String model,
/// The key it is called with; empty for a local model that asks for none.
required final String apiKey,
String? baseUrl,
}) {
/// Its base URL, without a trailing `/`: the one given, or the protocol's
/// own.
final Uri baseUrl = _baseUrl(baseUrl ?? protocol.defaultBaseUrl);
/// The host requests go to, which the person agrees to (ADR 0021, D40).
String get host => baseUrl.host;
@override
String toString() => 'LlmEndpoint(${protocol.name}, $baseUrl, $model)';
static Uri _baseUrl(String text) {
final url = Uri.tryParse(text.trim());
if (url == null || !url.hasAuthority || url.host.isEmpty) {
throw FormatException('Not an address', text);
}
if (url.hasQuery || url.hasFragment || url.userInfo.isNotEmpty) {
throw FormatException('Only a scheme, host, port and path', text);
}
if (url.scheme != 'https' &&
!(url.scheme == 'http' && _isLocal(url.host))) {
throw FormatException('Only https, or http on this network', text);
}
final path = url.path.endsWith('/')
? url.path.substring(0, url.path.length - 1)
: url.path;
return url.replace(path: path);
}
/// Whether [host] can only be on the same machine or network: a name of
/// one label (a container's, beside the server), a name under a suffix
/// reserved for local networks, or a loopback, private or link-local
/// address.
static bool _isLocal(String host) {
final name = host.toLowerCase();
if (name.contains(':')) {
return name == '::1' ||
name.startsWith('fe80:') ||
name.startsWith('fc') ||
name.startsWith('fd');
}
final octets = name.split('.').map(int.tryParse).toList();
if (octets.length == 4 && octets.every((octet) => octet != null)) {
final [a!, b!, ...] = octets;
return a == 127 ||
a == 10 ||
(a == 172 && b >= 16 && b < 32) ||
(a == 192 && b == 168) ||
(a == 169 && b == 254);
}
return !name.contains('.') || _localSuffixes.any(name.endsWith);
}
/// localhost (RFC 6761), mDNS (RFC 6762), home networks (RFC 8375) and
/// `.internal`, which ICANN reserved for private use in 2024.
static const _localSuffixes = [
'.localhost',
'.local',
'.home.arpa',
'.internal',
];
}

53
packages/halkyon_llm/lib/src/failure.dart Normal file
View file

@ -0,0 +1,53 @@
import 'package:halkyon_llm/src/conversation.dart';
/// Why a request failed: what the stream ends with instead of [LlmFinished].
final class const LlmFailure({
/// What went wrong, as the app tells it.
required final LlmFailureKind kind,
/// The provider's own words, or the error caught, for the logs. Neither
/// the request nor the answer is ever in it.
required final String detail,
/// The HTTP status, where a provider answered with one.
final int? status,
/// How long the provider asks to wait before trying again, where it says.
final Duration? retryAfter,
/// What the provider had counted before the failure, which is still
/// billed: the request's tokens and what was written of the answer.
final LlmUsage usage = LlmUsage.none,
}) implements Exception {
@override
String toString() =>
'LlmFailure(${kind.name}${status == null ? '' : ', $status'}: $detail)';
}
/// What went wrong with a request.
enum LlmFailureKind() {
/// The provider refuses the key: wrong, revoked, without access to the
/// model, or its account out of credit or past its spending limit.
key,
/// Too many requests on the key for now; [LlmFailure.retryAfter] may say
/// how long to wait.
rateLimited,
/// The provider is overloaded or its service down for now.
unavailable,
/// The provider refused the request itself: a model it doesn't have, a
/// request too large for the model, or one it can't read.
request,
/// The provider failed on its side.
provider,
/// The provider couldn't be reached, or the connection broke or went
/// silent.
network,
/// The provider's reply couldn't be read, or ended before the answer did.
malformed,
}

447
packages/halkyon_llm/lib/src/openai.dart Normal file
View file

@ -0,0 +1,447 @@
import 'dart:convert';
import 'package:halkyon_llm/src/client.dart';
import 'package:halkyon_llm/src/conversation.dart';
import 'package:halkyon_llm/src/endpoint.dart';
import 'package:halkyon_llm/src/failure.dart';
import 'package:halkyon_llm/src/sse.dart';
/// OpenAI's chat completions, streamed, as OpenAI and the services
/// compatible with it speak them
/// (https://developers.openai.com/api/reference/resources/chat/subresources/completions/streaming-events).
///
/// They differ in what a request may hold, and OpenAI, Mistral and Gemini
/// refuse a field they don't know, so each known host gets only the fields
/// it takes ([_Dialect]), and every other host a body that the local servers
/// (Ollama, vLLM, llama.cpp) all take (ADR 0021, D45). Replies are read
/// leniently, as each of them streams usage, tool calls and errors in its
/// own way.
final class OpenAiAdapter(
/// Where it sends.
final LlmEndpoint endpoint,
) implements ProtocolAdapter {
late final _dialect = _Dialect.of(endpoint.host);
@override
Uri get url => endpoint.baseUrl.replace(
path: '${endpoint.baseUrl.path}/chat/completions',
);
@override
Map<String, String> get headers => {
if (endpoint.apiKey.isNotEmpty)
'authorization': 'Bearer ${endpoint.apiKey}',
};
@override
Map<String, Object?> body(LlmRequest request) => {
'model': endpoint.model,
'messages': [
if (request.system.isNotEmpty)
{'role': 'system', 'content': request.system},
for (final message in request.messages) ..._messages(message),
],
if (request.tools.isNotEmpty)
'tools': [
for (final tool in request.tools)
{
'type': 'function',
'function': {
'name': tool.name,
'description': tool.description,
'parameters': tool.inputSchema,
},
},
],
_dialect.ceiling: request.maxOutputTokens,
'stream': true,
if (_dialect.asksForUsage) 'stream_options': {'include_usage': true},
if (_dialect.storesUnlessTold) 'store': false,
if ((_dialect.personField, request.personId) case (final field?, final id?))
field: id,
};
/// A turn as chat completions hold it: the tools' results as messages of
/// their own, right after the model's turn that called them, then the
/// person's text and images; the model's text and calls in one message.
List<Map<String, Object?>> _messages(LlmMessage message) {
switch (message.role) {
case LlmRole.user:
final results = message.parts.whereType<LlmToolResult>();
final rest = message.parts
.where((part) => part is LlmText || part is LlmImage)
.toList();
return [
for (final result in results)
{
'role': 'tool',
'tool_call_id': result.callId,
'content': result.isError
? 'Error: ${result.content}'
: result.content,
},
if (rest.isNotEmpty)
{
'role': 'user',
'content': rest.every((part) => part is LlmText)
? rest.cast<LlmText>().map((part) => part.text).join()
: [
for (final part in rest)
switch (part) {
LlmText(:final text) => {
'type': 'text',
'text': text,
},
LlmImage(:final mediaType, :final bytes) => {
'type': 'image_url',
'image_url': {
'url':
'data:$mediaType;base64,${base64Encode(bytes)}',
},
},
_ => throw StateError('Filtered above'),
},
],
},
];
case LlmRole.assistant:
final text = message.text;
final calls = message.toolCalls;
if (text.isEmpty && calls.isEmpty) {
return const [];
}
return [
{
'role': 'assistant',
// Left out rather than null where the model only called tools:
// Gemini refuses a null where it reads a string.
if (text.isNotEmpty) 'content': text,
if (calls.isNotEmpty)
'tool_calls': [
for (final call in calls)
{
'id': call.id,
'type': 'function',
'function': {
'name': call.name,
'arguments': jsonEncode(call.input),
},
if (_dialect == _Dialect.gemini) ...call.extra,
},
],
},
];
}
}
@override
ReplyReader reader() => _ChatReader();
@override
LlmFailure failureOf(
int status,
Map<String, String> headers,
List<int> body,
) => openAiFailureOf(status, headers, body);
}
/// The failure a reply other than `200 OK` stands for, in OpenAI's shape and
/// the compatible services' ([openAiErrorOf]): by its status, but a quota
/// used up (`insufficient_quota`, a 429 at OpenAI), which is the key's.
LlmFailure openAiFailureOf(
int status,
Map<String, String> headers,
List<int> body,
) {
final text = utf8.decode(body, allowMalformed: true);
final error = openAiErrorOf(_json(text));
final failure = failureOfStatus(status, headers, error?.message ?? text);
return error?.code == 'insufficient_quota'
? LlmFailure(
kind: LlmFailureKind.key,
detail: failure.detail,
status: status,
retryAfter: failure.retryAfter,
)
: failure;
}
Object? _json(String text) {
try {
return jsonDecode(text);
} on FormatException {
return null;
}
}
/// What each host takes, of the fields that differ.
enum _Dialect(
/// The field of the ceiling on output tokens: OpenAI's reasoning models
/// refuse `max_tokens`, Mistral refuses `max_completion_tokens`, and
/// Ollama ignores it.
final String ceiling,
/// Whether usage is streamed only when asked (`stream_options`), which
/// Mistral refuses as it always streams it.
final bool asksForUsage,
/// Whether conversations are kept unless the request says `store: false`,
/// which Mistral and Gemini refuse.
final bool storesUnlessTold,
/// The field of the person's opaque ID (ADR 0021, D30), where the host
/// takes one: OpenAI's `safety_identifier` (`user` is deprecated there),
/// OpenRouter's `user`.
final String? personField,
) {
openai('max_completion_tokens', true, true, 'safety_identifier'),
mistral('max_tokens', false, false, null),
gemini('max_completion_tokens', true, false, null),
openrouter('max_completion_tokens', false, false, 'user'),
/// Any other host: what Ollama, vLLM and llama.cpp all take, each ignoring
/// what it doesn't know.
generic('max_tokens', true, false, null);
static _Dialect of(String host) => switch (host.toLowerCase()) {
'api.openai.com' => openai,
'api.mistral.ai' => mistral,
'generativelanguage.googleapis.com' => gemini,
'openrouter.ai' => openrouter,
_ => generic,
};
}
/// An error a reply holds, in the shapes the services send it: `{"error":
/// {...}}` or `{"error": "..."}`, Gemini's list of those, Mistral's flat
/// `{"object": "error", ...}`; null for none.
({String message, String? code})? openAiErrorOf(Object? json) {
switch (json) {
case [final first, ...]:
return openAiErrorOf(first);
case {'error': final String message}:
return (message: message, code: null);
case {'error': final Map<String, Object?> error}:
return (
message: _messageOf(error['message']),
code: _codeOf(error['code']),
);
case {'object': 'error'} && final Map<String, Object?> error:
return (
message: _messageOf(error['message']),
code: _codeOf(error['code']),
);
}
return null;
}
/// An error's words: its text, or for Mistral's list of what failed each
/// field's place and reason, never the value it quotes, which may be the
/// person's words.
String _messageOf(Object? message) => switch (message) {
final String text => text,
{'detail': final List<Object?> details} => [
for (final detail in details)
if (detail case {'msg': final String msg, 'loc': final List<Object?> loc})
'${loc.join('.')}: $msg',
].join('; '),
_ => 'An error',
};
String? _codeOf(Object? code) => code?.toString();
final class _ChatReader() implements ReplyReader {
final _text = StringBuffer();
final _calls = <_Call>[];
String? _finishReason;
var _inputTokens = 0;
var _outputTokens = 0;
@override
bool finished = false;
@override
LlmUsage get usage =>
LlmUsage(inputTokens: _inputTokens, outputTokens: _outputTokens);
@override
Iterable<LlmEvent> read(SseEvent event) sync* {
if (event.data.trim() == '[DONE]') {
finished = true;
return;
}
final Object? chunk = jsonDecode(event.data);
if (openAiErrorOf(chunk) case final error?) {
throw LlmFailure(
kind: LlmFailureKind.provider,
detail: redacted(error.message),
usage: usage,
);
}
if (chunk is! Map<String, Object?>) {
throw const FormatException('A chunk that is no object');
}
// Usage comes once at the end (OpenAI, Ollama, llama.cpp, vLLM,
// OpenRouter), with the last choice (Mistral), or as a running total on
// every chunk (Gemini): the last one stands, never a sum.
if (chunk['usage'] case final Map<String, Object?> usage) {
_count(usage);
}
if (chunk['choices'] case [final Map<String, Object?> choice, ...]) {
if (choice['delta'] case final Map<String, Object?> delta) {
if (_textOf(delta['content']) case final text? when text.isNotEmpty) {
_text.write(text);
yield LlmTextDelta(text);
}
if (delta['refusal'] case final String refusal
when refusal.isNotEmpty) {
_text.write(refusal);
yield LlmTextDelta(refusal);
}
if (delta['tool_calls'] case final List<Object?> calls) {
calls.indexed.forEach(_merge);
}
}
if (choice['finish_reason'] case final String reason) {
if (reason == 'error') {
throw LlmFailure(
kind: LlmFailureKind.provider,
detail: 'The provider ended the answer with an error',
usage: usage,
);
}
_finishReason = reason;
}
}
}
/// The text of a delta's content: a string, or Mistral's parts, of which
/// only the text counts (not its thinking).
static String? _textOf(Object? content) => switch (content) {
final String text => text,
final List<Object?> parts => [
for (final part in parts)
if (part case {'type': 'text', 'text': final String text}) text,
].join(),
_ => null,
};
/// Puts a piece of a tool call where it belongs: by its index; failing
/// that, its ID; failing that, its place in the list. A new ID at an index
/// already held is a new call. Whole calls (Mistral, Gemini, Ollama) are
/// pieces of one.
void _merge((int, Object?) indexed) {
final (position, piece) = indexed;
if (piece is! Map<String, Object?>) {
throw const FormatException('A tool call that is no object');
}
final id = switch (piece['id']) {
final String id when id.isNotEmpty && id != 'null' => id,
_ => null,
};
final index = switch (piece['index']) {
final int index when index >= 0 => index,
_ => null,
};
var call = index == null
? _calls.where((call) => id != null && call.id == id).firstOrNull ??
(id == null && _calls.isNotEmpty ? _calls.last : null)
: _calls.where((call) => call.index == index).lastOrNull;
if (call == null || (id != null && call.id != null && call.id != id)) {
call = _Call(index ?? position);
_calls.add(call);
}
call.id ??= id;
if (piece['function'] case final Map<String, Object?> function) {
if (function['name'] case final String name when name.isNotEmpty) {
call.name ??= name;
}
switch (function['arguments']) {
case final String arguments:
call.arguments.write(arguments);
case final Map<String, Object?> arguments:
call.arguments
..clear()
..write(jsonEncode(arguments));
}
}
for (final MapEntry(:key, :value) in piece.entries) {
if (!_knownCallFields.contains(key)) {
call.extra[key] = value;
}
}
}
static const _knownCallFields = {'id', 'index', 'type', 'function'};
void _count(Map<String, Object?> usage) {
final input = usage['prompt_tokens'];
final output = usage['completion_tokens'];
final total = usage['total_tokens'];
if (input is int) {
_inputTokens = input;
}
if (output is int) {
// Gemini leaves its thinking out of completion_tokens, but not out of
// the total, and bills it as output.
_outputTokens = total is int && input is int && total - input > output
? total - input
: output;
}
}
@override
LlmFinished end() {
if (!finished && _finishReason == null) {
throw LlmFailure(
kind: LlmFailureKind.malformed,
detail: 'The reply ended before the answer did',
usage: usage,
);
}
final calls = [
for (final (number, call) in _calls.indexed)
LlmToolCall(
id: call.id ?? 'call_$number',
name:
call.name ??
(throw const FormatException('A tool call without its name')),
input: call.arguments.isEmpty
? const {}
: jsonObject(call.arguments.toString()),
extra: Map.unmodifiable(call.extra),
),
];
return LlmFinished(
message: LlmMessage.assistant([
if (_text.isNotEmpty || calls.isEmpty) LlmText(_text.toString()),
...calls,
]),
// Gemini ends with `stop` after calling tools.
stop: calls.isNotEmpty && _finishReason == 'stop'
? LlmStop.toolCalls
: switch (_finishReason) {
'stop' => LlmStop.end,
'length' => LlmStop.maxTokens,
'tool_calls' || 'function_call' => LlmStop.toolCalls,
'content_filter' => LlmStop.refused,
_ => LlmStop.other,
},
usage: usage,
);
}
}
/// A tool call being streamed.
final class _Call(
/// Its index in the stream.
final int index,
) {
String? id;
String? name;
final arguments = StringBuffer();
/// Fields the provider gave with the call, besides OpenAI's: Gemini's
/// `extra_content` with its thought signature.
final extra = <String, Object?>{};
}

345
packages/halkyon_llm/lib/src/openai_responses.dart Normal file
View file

@ -0,0 +1,345 @@
import 'dart:collection';
import 'dart:convert';
import 'dart:math';
import 'package:halkyon_llm/src/client.dart';
import 'package:halkyon_llm/src/conversation.dart';
import 'package:halkyon_llm/src/endpoint.dart';
import 'package:halkyon_llm/src/failure.dart';
import 'package:halkyon_llm/src/openai.dart';
import 'package:halkyon_llm/src/sse.dart';
/// OpenAI's Responses API, streamed
/// (https://developers.openai.com/api/reference/resources/responses/streaming-events),
/// on which alone OpenAI's newest models call tools, as Ollama, vLLM and
/// OpenRouter speak it too (ADR 0021, D46).
///
/// Nothing is kept at the provider (`store: false`): every request carries
/// the whole conversation, the model's reasoning with it as the encrypted
/// items it streamed, which come back as [LlmProviderBlock]s and go back in
/// their place. Each item is read once it is done, as the encrypted
/// reasoning may be incomplete before.
final class OpenAiResponsesAdapter(
/// Where it sends.
final LlmEndpoint endpoint,
) implements ProtocolAdapter {
/// The least ceiling on output tokens the API takes.
static const _leastOutput = 16;
@override
Uri get url =>
endpoint.baseUrl.replace(path: '${endpoint.baseUrl.path}/responses');
@override
Map<String, String> get headers => {
if (endpoint.apiKey.isNotEmpty)
'authorization': 'Bearer ${endpoint.apiKey}',
};
@override
Map<String, Object?> body(LlmRequest request) => {
'model': endpoint.model,
if (request.system.isNotEmpty) 'instructions': request.system,
'input': [for (final message in request.messages) ..._items(message)],
if (request.tools.isNotEmpty)
'tools': [
for (final tool in request.tools)
{
'type': 'function',
'name': tool.name,
'description': tool.description,
'parameters': tool.inputSchema,
// Left out, the API would make the schema strict where it can,
// which then refuses calls the schema allows.
'strict': false,
},
],
'max_output_tokens': max(request.maxOutputTokens, _leastOutput),
'stream': true,
'store': false,
// What a stateless request has by default now; asked for still, as
// servers that followed the API earlier may not.
'include': ['reasoning.encrypted_content'],
// OpenAI's field (D30); the other servers know none.
if ((endpoint.host, request.personId) case ('api.openai.com', final id?))
'safety_identifier': id,
};
/// A turn as the Responses API's input items: the tools' results first,
/// right after the model's calls, then the person's text and images; the
/// model's reasoning, text and calls in the order written.
List<Map<String, Object?>> _items(LlmMessage message) {
switch (message.role) {
case LlmRole.user:
final rest = message.parts
.where((part) => part is LlmText || part is LlmImage)
.toList();
return [
for (final result in message.parts.whereType<LlmToolResult>())
{
'type': 'function_call_output',
'call_id': result.callId,
'output': result.isError
? 'Error: ${result.content}'
: result.content,
},
if (rest.isNotEmpty)
{
'role': 'user',
'content': rest.every((part) => part is LlmText)
? rest.cast<LlmText>().map((part) => part.text).join()
: [
for (final part in rest)
switch (part) {
LlmText(:final text) => {
'type': 'input_text',
'text': text,
},
LlmImage(:final mediaType, :final bytes) => {
'type': 'input_image',
'image_url':
'data:$mediaType;base64,${base64Encode(bytes)}',
'detail': 'auto',
},
_ => throw StateError('Filtered above'),
},
],
},
];
case LlmRole.assistant:
return [
for (final part in message.parts)
...switch (part) {
LlmText(:final text, :final extra) when text.isNotEmpty => [
{
'role': 'assistant',
'content': text,
if (extra['phase'] case final String phase) 'phase': phase,
},
],
// Without the item's own ID, which a stateless request needs
// not, and which would tie the call to reasoning another
// provider's turn never had.
LlmToolCall(:final id, :final name, :final input) => [
{
'type': 'function_call',
'call_id': id,
'name': name,
'arguments': jsonEncode(input),
},
],
LlmProviderBlock(
protocol: LlmProtocol.openaiResponses,
:final block,
) =>
[block],
_ => const <Map<String, Object?>>[],
},
];
}
}
@override
ReplyReader reader() => _ResponsesReader();
@override
LlmFailure failureOf(
int status,
Map<String, String> headers,
List<int> body,
) => openAiFailureOf(status, headers, body);
}
final class _ResponsesReader() implements ReplyReader {
/// The items done, by their place in the answer.
final _items = SplayTreeMap<int, Map<String, Object?>>();
String? _status;
String? _incompleteReason;
var _inputTokens = 0;
var _outputTokens = 0;
@override
bool finished = false;
@override
LlmUsage get usage =>
LlmUsage(inputTokens: _inputTokens, outputTokens: _outputTokens);
@override
Iterable<LlmEvent> read(SseEvent event) sync* {
// The API ends with its last event; its SDKs stop at a [DONE] too.
if (event.data.trim() == '[DONE]') {
finished = true;
return;
}
// By the data's type, as OpenAI's SDKs read it, not the event's name.
final data = jsonObject(event.data);
switch (data) {
case {
'type': 'response.output_text.delta' || 'response.refusal.delta',
'delta': final String delta,
}:
if (delta.isNotEmpty) {
yield LlmTextDelta(delta);
}
case {
'type': 'response.output_item.done',
'output_index': final int index,
'item': final Map<String, Object?> item,
}:
_items[index] = item;
case {
'type': 'response.completed' || 'response.incomplete',
'response': final Map<String, Object?> response,
}:
_finish(response);
case {
'type': 'response.failed',
'response': final Map<String, Object?> response,
}:
_count(response['usage']);
throw _failure(response['error']);
case {'type': 'error'}:
// As documented, the error's fields are the event's own; as OpenAI
// has been seen to send it, they are under `error`.
throw _failure(switch (data['error']) {
final Map<String, Object?> error => error,
_ => data,
});
}
}
void _finish(Map<String, Object?> response) {
if (response['status'] case final String status) {
_status = status;
}
if (response['incomplete_details'] case {
'reason': final String reason,
}) {
_incompleteReason = reason;
}
_count(response['usage']);
// A server that streams no items still gives them with the response.
if (_items.isEmpty) {
if (response['output'] case final List<Object?> output) {
for (final (index, item) in output.indexed) {
if (item case final Map<String, Object?> item) {
_items[index] = item;
}
}
}
}
finished = true;
}
/// Usage as the response gives it: input with what was read from the
/// cache, output with the reasoning.
void _count(Object? usage) {
if (usage case {
'input_tokens': final int input,
'output_tokens': final int output,
}) {
_inputTokens = input;
_outputTokens = output;
}
}
LlmFailure _failure(Object? error) {
final (code, message) = switch (error) {
{'message': final String message} => (
(error['code'] ?? error['type'])?.toString(),
message,
),
_ => (null, 'The provider failed the answer'),
};
return LlmFailure(
kind: switch (code) {
'insufficient_quota' => LlmFailureKind.key,
'rate_limit_exceeded' => LlmFailureKind.rateLimited,
'server_is_overloaded' || 'slow_down' => LlmFailureKind.unavailable,
'context_length_exceeded' || 'invalid_prompt' => LlmFailureKind.request,
_ => LlmFailureKind.provider,
},
detail: redacted(message),
usage: usage,
);
}
@override
LlmFinished end() {
if (!finished) {
throw LlmFailure(
kind: LlmFailureKind.malformed,
detail: 'The reply ended before the answer did',
usage: usage,
);
}
final parts = <LlmPart>[];
var refused = false;
for (final item in _items.values) {
switch (item) {
case {'type': 'message', 'content': final List<Object?> content}:
final text = StringBuffer();
for (final part in content) {
switch (part) {
case {'type': 'output_text', 'text': final String written}:
text.write(written);
case {'type': 'refusal', 'refusal': final String refusal}:
text.write(refusal);
refused = true;
}
}
if (text.isNotEmpty) {
parts.add(
LlmText(
text.toString(),
extra: {
if (item['phase'] case final String phase) 'phase': phase,
},
),
);
}
case {
'type': 'function_call',
'call_id': final String id,
'name': final String name,
}:
parts.add(
LlmToolCall(
id: id,
name: name,
input: switch (item['arguments']) {
final String arguments when arguments.isNotEmpty => jsonObject(
arguments,
),
_ => const {},
},
),
);
case {'type': 'reasoning'}:
parts.add(
LlmProviderBlock(
protocol: LlmProtocol.openaiResponses,
block: item,
),
);
}
}
final calls = parts.whereType<LlmToolCall>().isNotEmpty;
return LlmFinished(
message: LlmMessage.assistant([
...parts,
if (!calls && parts.whereType<LlmText>().isEmpty) const LlmText(''),
]),
stop: switch ((_status, _incompleteReason)) {
('incomplete', 'max_output_tokens') => LlmStop.maxTokens,
('incomplete', 'content_filter') => LlmStop.refused,
('completed', _) when calls => LlmStop.toolCalls,
('completed', _) when refused => LlmStop.refused,
('completed', _) => LlmStop.end,
_ => LlmStop.other,
},
usage: usage,
);
}
}

65
packages/halkyon_llm/lib/src/sse.dart Normal file
View file

@ -0,0 +1,65 @@
import 'dart:async';
import 'dart:convert';
/// One event of a stream of server-sent events.
final class const SseEvent({
/// Its `event:` field; `message` where it has none.
required final String type,
/// Its `data:` lines, joined by line feeds.
required final String data,
});
/// The events of a `text/event-stream` body, as the HTML Standard reads them
/// (§ 9.2.6, "Interpreting an event stream"): UTF-8 with a byte order mark
/// skipped, lines ended by CR LF, LF or CR, `:` starting a comment, a field's
/// value after its first `:` less one space, `data:` lines joined, and an
/// event dispatched at an empty line, not before: a stream that ends in the
/// middle of an event drops it. `id:` and `retry:` are read and ignored, as
/// a request here is never resumed.
Stream<SseEvent> parseSse(Stream<List<int>> bytes) async* {
var type = '';
final data = StringBuffer();
var hasData = false;
var first = true;
await for (var line
in bytes.transform(utf8.decoder).transform(const LineSplitter())) {
if (first) {
first = false;
if (line.startsWith('\uFEFF')) {
line = line.substring(1);
}
}
if (line.isEmpty) {
if (hasData) {
yield SseEvent(
type: type.isEmpty ? 'message' : type,
data: data.toString(),
);
}
type = '';
data.clear();
hasData = false;
continue;
}
if (line.startsWith(':')) {
continue;
}
final colon = line.indexOf(':');
final field = colon < 0 ? line : line.substring(0, colon);
var value = colon < 0 ? '' : line.substring(colon + 1);
if (value.startsWith(' ')) {
value = value.substring(1);
}
switch (field) {
case 'event':
type = value;
case 'data':
if (hasData) {
data.write('\n');
}
data.write(value);
hasData = true;
}
}
}

91
packages/halkyon_llm/lib/testing.dart Normal file
View file

@ -0,0 +1,91 @@
/// An [LlmClient] for the tests of what uses this package: it answers from a
/// script and keeps what it was sent.
library;
import 'dart:async';
import 'dart:collection';
import 'package:halkyon_llm/halkyon_llm.dart';
/// An [LlmClient] answering each request with the next reply scripted.
final class FakeLlmClient() implements LlmClient {
final _replies = Queue<_Reply>();
/// What it was sent, in order.
final List<(LlmEndpoint, LlmRequest)> sent = [];
/// How many answers were cancelled before they ended.
int cancelled = 0;
/// Whether [close] was called.
bool closed = false;
/// Scripts the next reply: [events], then [failure] if given.
void reply(List<LlmEvent> events, {LlmFailure? failure}) =>
_replies.add(_Reply(events, failure));
/// Scripts an answer of [text], in two pieces, counted as [usage].
void replyText(
String text, {
LlmUsage usage = const LlmUsage(inputTokens: 100, outputTokens: 20),
}) {
final half = text.length ~/ 2;
reply([
LlmTextDelta(text.substring(0, half)),
LlmTextDelta(text.substring(half)),
LlmFinished(
message: LlmMessage.assistant([LlmText(text)]),
stop: LlmStop.end,
usage: usage,
),
]);
}
/// Scripts a failure of [kind] before anything is written, with [usage]
/// counted.
void fail(LlmFailureKind kind, {LlmUsage usage = LlmUsage.none}) => reply(
const [],
failure: LlmFailure(kind: kind, detail: 'scripted', usage: usage),
);
/// Whether every reply scripted was given.
bool get allReplied => _replies.isEmpty;
@override
Stream<LlmEvent> send(LlmEndpoint endpoint, LlmRequest request) {
sent.add((endpoint, request));
if (_replies.isEmpty) {
throw StateError('No reply scripted for request ${sent.length}');
}
final reply = _replies.removeFirst();
var ended = false;
late final StreamController<LlmEvent> events;
events = StreamController(
onListen: () async {
for (final event in reply.events) {
if (events.isClosed) {
return;
}
events.add(event);
await Future<void>.delayed(Duration.zero);
}
if (reply.failure case final failure?) {
events.addError(failure);
}
ended = true;
await events.close();
},
onCancel: () {
if (!ended) {
cancelled++;
}
},
);
return events.stream;
}
@override
void close() => closed = true;
}
final class _Reply(final List<LlmEvent> events, final LlmFailure? failure);

22
packages/halkyon_llm/pubspec.yaml Normal file
View file

@ -0,0 +1,22 @@
name: halkyon_llm
description: >-
Language models' APIs in pure Dart, streamed, over plain HTTP: Anthropic's
Messages API, OpenAI's chat completions (and every service compatible with
them) and OpenAI's Responses API, behind one interface of what they share:
a system prompt and messages, text as it is written, images, and tool calls.
version: 0.1.0
# In the Halkyon repository; publishing it to pub.dev is asked first (ADR 0021, D15).
publish_to: none
resolution: workspace
environment:
sdk: ^3.13.0
dependencies:
# The requests, aborted when the listener cancels (AbortableRequest, 1.5.0 on).
http: ^1.6.0
dev_dependencies:
# The recording tool's options.
args: ^2.7.0
test: ^1.31.1

611
packages/halkyon_llm/test/anthropic_test.dart Normal file
View file

@ -0,0 +1,611 @@
import 'dart:async';
import 'dart:convert';
import 'dart:typed_data';
import 'package:halkyon_llm/halkyon_llm.dart';
import 'package:http/http.dart' as http;
import 'package:http/testing.dart';
import 'package:test/test.dart';
import 'support/fixture.dart';
final _endpoint = LlmEndpoint(
protocol: LlmProtocol.anthropic,
model: 'claude-sonnet-5-5',
apiKey: 'sk-ant-test',
);
final _question = LlmRequest(
system: 'Du hilfst beim Japanischlernen.',
messages: [
LlmMessage.user([const LlmText('Was heißt „Hund“ auf Japanisch?')]),
],
maxOutputTokens: 4096,
personId: 'p_8Zq3',
);
final _documented = fixturesOf('documented', 'anthropic');
/// What [request] streams, against [fixture], each event, then the error
/// the stream ended with, if any.
Future<(List<LlmEvent>, LlmFailure?)> _reply(
Fixture fixture, {
LlmRequest? request,
Sent? sent,
int chunk = 7,
}) async {
final client = LlmClient(
httpClient: replying(fixture, sent: sent, chunk: chunk),
);
final events = <LlmEvent>[];
try {
await client.send(_endpoint, request ?? _question).forEach(events.add);
} on LlmFailure catch (failure) {
return (events, failure);
}
return (events, null);
}
String _text(List<LlmEvent> events) =>
events.whereType<LlmTextDelta>().map((delta) => delta.text).join();
void main() {
group('the request', () {
test(
"goes to the Messages API with the key and the API's version",
() async {
final sent = Sent();
await _reply(_documented['text']!, sent: sent);
expect(sent.request.method, 'POST');
expect(
sent.request.url,
Uri.parse('https://api.anthropic.com/v1/messages'),
);
expect(sent.request.headers, containsPair('x-api-key', 'sk-ant-test'));
expect(
sent.request.headers,
containsPair('anthropic-version', '2023-06-01'),
);
expect(
sent.request.headers['content-type'],
startsWith('application/json'),
);
expect(
sent.request.headers.keys,
isNot(contains('anthropic-dangerous-direct-browser-access')),
reason: 'only in a browser',
);
},
);
test('takes a base URL of its own, with its path', () async {
final sent = Sent();
final client = LlmClient(
httpClient: replying(_documented['text']!, sent: sent),
);
await client
.send(
LlmEndpoint(
protocol: LlmProtocol.anthropic,
model: 'm',
apiKey: 'k',
baseUrl: 'https://proxy.example.org/anthropic/',
),
_question,
)
.drain<void>();
expect(
sent.request.url,
Uri.parse('https://proxy.example.org/anthropic/v1/messages'),
);
});
test('streams the conversation with the system prompt, the ceiling and the '
"person's opaque ID", () async {
final sent = Sent();
await _reply(_documented['text']!, sent: sent);
expect(sent.body, {
'model': 'claude-sonnet-5-5',
'max_tokens': 4096,
'stream': true,
'system': 'Du hilfst beim Japanischlernen.',
'messages': [
{
'role': 'user',
'content': [
{'type': 'text', 'text': 'Was heißt „Hund“ auf Japanisch?'},
],
},
],
'metadata': {'user_id': 'p_8Zq3'},
});
});
test('leaves out an empty system prompt, no tools and no person', () async {
final sent = Sent();
await _reply(
_documented['text']!,
sent: sent,
request: LlmRequest(
messages: [
LlmMessage.user([const LlmText('Hallo')]),
],
maxOutputTokens: 100,
),
);
expect(
sent.body.keys,
unorderedEquals(['model', 'max_tokens', 'stream', 'messages']),
);
});
test('sends tools, images, tool calls, results first, and its own blocks '
'back unchanged', () async {
final sent = Sent();
const thinking = LlmProviderBlock(
protocol: LlmProtocol.anthropic,
block: {'type': 'thinking', 'thinking': '', 'signature': 'EqQB'},
);
await _reply(
_documented['text']!,
sent: sent,
request: LlmRequest(
messages: [
LlmMessage.user([
const LlmText('Was steht da?'),
LlmImage(
mediaType: 'image/png',
bytes: Uint8List.fromList([1, 2, 3]),
),
]),
LlmMessage.assistant([
thinking,
const LlmText(''),
const LlmProviderBlock(
protocol: LlmProtocol.openai,
block: {'type': 'other'},
),
const LlmToolCall(
id: 'toolu_1',
name: 'find_cards',
input: {'query': '犬'},
),
]),
LlmMessage.user([
const LlmText('Und?'),
const LlmToolResult(
callId: 'toolu_1',
content: '[]',
isError: true,
),
]),
],
tools: const [
LlmTool(
name: 'find_cards',
description: 'Finds cards.',
inputSchema: {
'type': 'object',
'properties': {
'query': {'type': 'string'},
},
},
),
],
maxOutputTokens: 100,
),
);
expect(sent.body['messages'], [
{
'role': 'user',
'content': [
{'type': 'text', 'text': 'Was steht da?'},
{
'type': 'image',
'source': {
'type': 'base64',
'media_type': 'image/png',
'data': 'AQID',
},
},
],
},
{
'role': 'assistant',
'content': [
{'type': 'thinking', 'thinking': '', 'signature': 'EqQB'},
{
'type': 'tool_use',
'id': 'toolu_1',
'name': 'find_cards',
'input': {'query': '犬'},
},
],
},
{
'role': 'user',
'content': [
{
'type': 'tool_result',
'tool_use_id': 'toolu_1',
'content': '[]',
'is_error': true,
},
{'type': 'text', 'text': 'Und?'},
],
},
]);
expect(sent.body['tools'], [
{
'name': 'find_cards',
'description': 'Finds cards.',
'input_schema': {
'type': 'object',
'properties': {
'query': {'type': 'string'},
},
},
},
]);
});
test('leaves out a turn with nothing to send', () async {
final sent = Sent();
await _reply(
_documented['text']!,
sent: sent,
request: LlmRequest(
messages: [
LlmMessage.user([const LlmText('a')]),
LlmMessage.assistant([const LlmText('')]),
LlmMessage.user([const LlmText('b')]),
],
maxOutputTokens: 100,
),
);
expect(sent.body['messages'], hasLength(2));
});
});
group('documented replies', () {
for (final chunk in [1, 7, 4096]) {
test('text streams as written, in pieces of $chunk bytes', () async {
final (events, failure) = await _reply(
_documented['text']!,
chunk: chunk,
);
expect(failure, isNull);
expect(_text(events), '„Hund“ heißt auf Japanisch 犬 (いぬ, inu).');
final finished = events.last as LlmFinished;
expect(finished.stop, LlmStop.end);
expect(
finished.usage,
const LlmUsage(inputTokens: 412, outputTokens: 38),
);
expect(finished.message.role, LlmRole.assistant);
expect(
finished.message.text,
'„Hund“ heißt auf Japanisch 犬 (いぬ, inu).',
);
});
}
test(
'a thinking block is kept, its signature whole, to be sent back',
() async {
final (events, _) = await _reply(_documented['text']!);
final parts = (events.last as LlmFinished).message.parts;
expect(parts, hasLength(2));
final block = parts.first as LlmProviderBlock;
expect(block.protocol, LlmProtocol.anthropic);
expect(block.block, {
'type': 'thinking',
'thinking': '',
'signature': 'EqQBCgIYAhIMxk3Lr9vQm2HbW7pTGgy1d2nXo8Rf4KsJ0eQiMNp6',
});
expect(parts.last, isA<LlmText>());
},
);
test('a tool call comes whole, its input put together', () async {
final (events, failure) = await _reply(_documented['tool']!);
expect(failure, isNull);
expect(_text(events), 'Ich sehe im Studienset nach.');
final finished = events.last as LlmFinished;
expect(finished.stop, LlmStop.toolCalls);
expect(
finished.usage,
const LlmUsage(inputTokens: 472, outputTokens: 89),
);
final [call] = finished.message.toolCalls;
expect(call.id, 'toolu_01T1x1fJ34qAmk2tNTrN7Up6');
expect(call.name, 'find_cards');
expect(call.input, {'query': 'Hund', 'limit': 5});
});
test(
'an error streamed mid-answer fails it, with what was counted',
() async {
final (events, failure) = await _reply(
_documented['overloaded_midstream']!,
);
expect(_text(events), '„Hund“');
expect(events.whereType<LlmFinished>(), isEmpty);
expect(failure!.kind, LlmFailureKind.unavailable);
expect(failure.detail, 'Overloaded');
expect(
failure.usage,
const LlmUsage(inputTokens: 412, outputTokens: 1),
);
},
);
final statuses = {
'status_400': (LlmFailureKind.request, 400, null),
'status_401': (LlmFailureKind.key, 401, null),
'status_402': (LlmFailureKind.key, 402, null),
'status_429': (
LlmFailureKind.rateLimited,
429,
const Duration(seconds: 30),
),
'status_529': (LlmFailureKind.unavailable, 529, null),
};
for (final MapEntry(key: name, value: (kind, status, retryAfter))
in statuses.entries) {
test('$name fails as ${kind.name}', () async {
final (events, failure) = await _reply(_documented[name]!);
expect(events, isEmpty);
expect(failure!.kind, kind);
expect(failure.status, status);
expect(failure.retryAfter, retryAfter);
expect(failure.usage, LlmUsage.none);
});
}
test("keeps the provider's words for the logs", () async {
final (_, failure) = await _reply(_documented['status_401']!);
expect(failure!.detail, 'invalid x-api-key');
});
});
group('replies off the documentation', () {
Fixture stream(String events) => Fixture.parse(
'inline',
Uint8List.fromList(
utf8.encode('HTTP 200\ncontent-type: text/event-stream\n\n$events'),
),
);
const start =
'event: message_start\ndata: {"type":"message_start","message":'
'{"usage":{"input_tokens":10,"output_tokens":1}}}\n\n';
const text =
'event: content_block_start\ndata: {"type":"content_block_start",'
'"index":0,"content_block":{"type":"text","text":""}}\n\n'
'event: content_block_delta\ndata: {"type":"content_block_delta",'
'"index":0,"delta":{"type":"text_delta","text":"Hi"}}\n\n'
'event: content_block_stop\ndata: {"type":"content_block_stop",'
'"index":0}\n\n';
const end =
'event: message_delta\ndata: {"type":"message_delta","delta":'
'{"stop_reason":"end_turn"},"usage":{"output_tokens":5}}\n\n'
'event: message_stop\ndata: {"type":"message_stop"}\n\n';
test('events and deltas of kinds yet to come are passed over', () async {
final (events, failure) = await _reply(
stream(
'${start}event: future_event\ndata: {"type":"future_event"}\n\n'
'event: content_block_start\ndata: {"type":"content_block_start",'
'"index":0,"content_block":{"type":"future_block"}}\n\n'
'event: content_block_delta\ndata: {"type":"content_block_delta",'
'"index":0,"delta":{"type":"future_delta"}}\n\n'
'event: content_block_stop\ndata: {"type":"content_block_stop",'
'"index":0}\n\n'
'${text.replaceAll('"index":0', '"index":1')}$end',
),
);
expect(failure, isNull);
expect((events.last as LlmFinished).message.parts, [isA<LlmText>()]);
});
test('a redacted thinking block is kept as it came', () async {
final (events, _) = await _reply(
stream(
'${start}event: content_block_start\ndata: {"type":"content_block_start",'
'"index":0,"content_block":{"type":"redacted_thinking","data":"EmwKAhgB"}}\n\n'
'event: content_block_stop\ndata: {"type":"content_block_stop",'
'"index":0}\n\n$end',
),
);
final block =
(events.last as LlmFinished).message.parts.single as LlmProviderBlock;
expect(block.block, {'type': 'redacted_thinking', 'data': 'EmwKAhgB'});
});
test('usage with input in message_delta takes the later count', () async {
final (events, _) = await _reply(
stream(
'$start$text${end.replaceAll('"output_tokens":5', '"input_tokens":12,"cache_read_input_tokens":3,"output_tokens":5')}',
),
);
expect(
(events.last as LlmFinished).usage,
const LlmUsage(inputTokens: 15, outputTokens: 5),
);
});
test('an empty answer is an empty text', () async {
final (events, _) = await _reply(stream('$start$end'));
expect((events.last as LlmFinished).message.text, '');
});
for (final (reason, stop) in [
('max_tokens', LlmStop.maxTokens),
('model_context_window_exceeded', LlmStop.maxTokens),
('stop_sequence', LlmStop.end),
('refusal', LlmStop.refused),
('pause_turn', LlmStop.other),
]) {
test('stop reason $reason is ${stop.name}', () async {
final (events, _) = await _reply(
stream('$start$text${end.replaceAll('end_turn', reason)}'),
);
expect((events.last as LlmFinished).stop, stop);
});
}
test('a reply that ends before message_stop fails as malformed, with what '
'was counted', () async {
final (events, failure) = await _reply(stream('$start$text'));
expect(_text(events), 'Hi');
expect(failure!.kind, LlmFailureKind.malformed);
expect(failure.usage, const LlmUsage(inputTokens: 10, outputTokens: 1));
});
test("a tool call whose input isn't JSON fails as malformed", () async {
final (_, failure) = await _reply(
stream(
'${start}event: content_block_start\ndata: {"type":"content_block_start",'
'"index":0,"content_block":{"type":"tool_use","id":"t","name":"n","input":{}}}\n\n'
'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,'
'"delta":{"type":"input_json_delta","partial_json":"{\\"query\\": "}}\n\n'
'event: content_block_stop\ndata: {"type":"content_block_stop",'
'"index":0}\n\n$end',
),
);
expect(failure!.kind, LlmFailureKind.malformed);
});
test('a field of another type fails as malformed, without quoting the reply', () async {
final (_, failure) = await _reply(
stream(
'event: content_block_start\ndata: {"type":"content_block_start",'
'"index":"Geheimnis","content_block":{"type":"text","text":""}}\n\n',
),
);
expect(failure!.kind, LlmFailureKind.malformed);
expect(failure.detail, isNot(contains('Geheimnis')));
});
test('a status without a JSON body fails by its status', () async {
final (_, failure) = await _reply(
Fixture.parse(
'inline',
Uint8List.fromList(
utf8.encode('HTTP 502\ncontent-type: text/html\n\n<html>'),
),
),
);
expect(failure!.kind, LlmFailureKind.provider);
expect(failure.status, 502);
});
test("what looks like a key in the provider's words is cut", () async {
final (_, failure) = await _reply(
Fixture.parse(
'inline',
Uint8List.fromList(
utf8.encode(
'HTTP 401\n\n{"type":"error","error":{"type":"authentication_error",'
'"message":"Incorrect API key provided: sk-ant-api03-abcdEFGH1234"}}',
),
),
),
);
expect(failure!.detail, 'Incorrect API key provided: sk-…');
});
});
group('the connection', () {
test('a provider out of reach fails as network', () {
final client = LlmClient(
httpClient: MockClient(
(_) => throw http.ClientException('Connection refused'),
),
);
expect(
client.send(_endpoint, _question).drain<void>(),
throwsA(
isA<LlmFailure>().having(
(f) => f.kind,
'kind',
LlmFailureKind.network,
),
),
);
});
test('a reply that goes silent fails as network', () {
final client = HttpLlmClient(
httpClient: MockClient.streaming(
(_, _) async => http.StreamedResponse(
Stream<List<int>>.fromFuture(Completer<List<int>>().future),
200,
),
),
idleTimeout: const Duration(milliseconds: 20),
);
expect(
client.send(_endpoint, _question).drain<void>(),
throwsA(
isA<LlmFailure>().having(
(f) => f.kind,
'kind',
LlmFailureKind.network,
),
),
);
});
test('a reply that never begins fails as network', () {
final client = HttpLlmClient(
httpClient: MockClient.streaming(
(_, _) => Completer<http.StreamedResponse>().future,
),
headersTimeout: const Duration(milliseconds: 20),
);
expect(
client.send(_endpoint, _question).drain<void>(),
throwsA(
isA<LlmFailure>().having(
(f) => f.kind,
'kind',
LlmFailureKind.network,
),
),
);
});
test('cancelling the answer aborts the request', () async {
final sent = Sent();
final body = StreamController<List<int>>();
final client = LlmClient(
httpClient: MockClient.streaming((request, bodyStream) async {
sent.request = request;
await bodyStream.drain<void>();
if (request case http.Abortable(:final abortTrigger?)) {
abortTrigger.whenComplete(() => sent.aborted = true).ignore();
}
return http.StreamedResponse(body.stream, 200);
}),
);
final first = Completer<void>();
final subscription = client.send(_endpoint, _question).listen((_) {
if (!first.isCompleted) {
first.complete();
}
});
body.add(
utf8.encode(
'event: content_block_start\ndata: {"type":"content_block_start",'
'"index":0,"content_block":{"type":"text","text":"Hi"}}\n\n',
),
);
await first.future;
await subscription.cancel();
await pumpEventQueue();
expect(sent.aborted, isTrue);
await body.close();
});
});
}

15
packages/halkyon_llm/test/fixtures/documented/anthropic/overloaded_midstream.http vendored Normal file
View file

@ -0,0 +1,15 @@
HTTP 200
content-type: text/event-stream; charset=utf-8
event: message_start
data: {"type":"message_start","message":{"id":"msg_01Qm2vYB8xkN4cH6sGtRpL3J","type":"message","role":"assistant","model":"claude-sonnet-5-5","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":412,"output_tokens":1}}}
event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"„Hund“"}}
event: error
data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}}

4
packages/halkyon_llm/test/fixtures/documented/anthropic/status_400.http vendored Normal file
View file

@ -0,0 +1,4 @@
HTTP 400
content-type: application/json
{"type":"error","error":{"type":"invalid_request_error","message":"model: claude-sonet-5-5 is not a valid model"},"request_id":"req_011CSHoEeqs5C35K2UUqR7G0"}

5
packages/halkyon_llm/test/fixtures/documented/anthropic/status_401.http vendored Normal file
View file

@ -0,0 +1,5 @@
HTTP 401
content-type: application/json
request-id: req_018EeWyXxfu5pfWkrYcMdjWG
{"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"},"request_id":"req_018EeWyXxfu5pfWkrYcMdjWG"}

4
packages/halkyon_llm/test/fixtures/documented/anthropic/status_402.http vendored Normal file
View file

@ -0,0 +1,4 @@
HTTP 402
content-type: application/json
{"type":"error","error":{"type":"billing_error","message":"Your credit balance is too low to access the Claude API."},"request_id":"req_011CSHoEeqs5C35K2UUqR7G1"}

Some files were not shown because too many files have changed in this diff Show more