Reference · Talking agents
- The language
- Agents and tools
- Model providers
- The Anthropic API
- Talking agents
- Standard library
- Glossary
- Package docs
- Telemetry
- Robots
- Servers
- Deploying
- tin.ad
- The tin command
- How it works
On this page
Talking agents
Voice agents: speech-to-speech sessions with tools, barge-in and restarts. Implemented for OpenAI Realtime, Gemini Live and Nova Sonic and tested against a mock; no real provider has produced a spoken reply yet.
A voice agent holds one live, two-way connection to a speech-to-speech model. Audio and text go in
while audio, transcripts and tool calls come out, at the same time. The design follows the bidirectional agent in
strands-agents/harness-sdk (its event taxonomy, its loop and its restart rules), written as a language feature: the
declaration, the events, the capabilities and the tool rules are checked by the compiler. The tutorial chapter is
Talking agents.
What has been verified. Whole sessions run end to end with the offline "mock" model, in tests and
in the playground with a fake microphone. The three real providers are implemented and tested against recorded fixtures;
OpenAI Realtime has connected and authenticated live, but no real provider has produced a spoken reply yet
(details, status).
Declaring a voice agent
/// The weather in a city.
tool fn weather(city: string): string => "18C in {city}"
voice agent Concierge {
model: "mock"
system: "Answer in one short sentence."
voice: "marin"
tools: [weather]
maxTokens: 4000
restartAfter: 300s
on heard(text) => print("user: {text}")
on said(text) => print("agent: {text}")
on toolCall(name, input) => print("tool {name} {input}")
}
fn main() {
let s = Concierge.open()
s.say("weather in 'Rome'")
var done = false
while !done {
match s.next() {
Said(t) => s.stop()
HungUp(r) => { done = true }
_ => {}
}
}
} tool weather {"city":"Rome"}
agent: mock: 18C in Rome | Field | Meaning |
|---|---|
model | a speech-to-speech model (below) or "mock". Required. A literal is checked at compile time; a computed name when the session connects. |
system | instructions, as for a text agent |
voice | the provider's voice name (marin, Puck, tiffany, …) |
tools | tool fns the model may call mid-conversation (not other agents) |
restartAfter | a Duration: reconnect proactively after this long (default: the provider's limit minus a margin) |
maxTokens | accepted for symmetry; providers bound a live response themselves |
on … | heard(text), said(text), text(t), toolCall(name, input), toolResult(name, output, ok), bargeIn() |
maxTurns and code are compile errors on a voice agent: a live session is not a turn loop
the program drives, and code as action needs a reply to check before it runs. Calling a voice agent like a text
agent is an error with a fix:
voice agent V { model: "mock" }
fn main() { print(V("hello")) } error: V is a voice agent: it holds a live conversation, not one question
--> main.tin:2:19
|
2 | fn main() { print(V("hello")) }
| ^
= note: open a session: `let s = V.open()`, or run a whole call: `V.talk(mic(), speaker())`
= fix: open a session voice agent V { model: "openai:gpt-5.1" }
fn main() {} error: 'openai:gpt-5.1' is not a speech-to-speech model
--> main.tin:1:24
|
1 | voice agent V { model: "openai:gpt-5.1" }
| ^^^^^^^^^^^^^^^^
= note: a voice agent uses one of: openai-realtime:<model>, gemini-live:<model>, bedrock-sonic:<model>, or "mock" agent T { model: "openai-realtime:gpt-realtime-mini" }
fn main() {} error: 'openai-realtime:gpt-realtime-mini' is a speech-to-speech model; a text agent can't use it
--> main.tin:1:18
|
1 | agent T { model: "openai-realtime:gpt-realtime-mini" }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= note: declare a talking agent: `voice agent T { ... }`, and call it with .open() or .talk(mic(), speaker()) Sessions
| Call | Does |
|---|---|
A.open(): Session | connects (one connection per call; several can be open at once, each with its own history) |
A.talk(input: AudioIn, output: AudioOut): string | a whole call: the input streams to the model, the reply plays, a barge-in clears the output; returns the transcript (user: … / assistant: … lines) when the input ends and the model has been quiet for 2 s, or the session closes |
s.say(text) | sends a user message as text |
s.send(chunk: Audio) | streams audio (resampled to the model's rate) |
s.next(): VoiceEvent | waits for the next event; after the end, HungUp(reason) every time |
s.interrupt() | cancels the reply being spoken |
s.stop() | closes the connection (waits for running tools) |
s.save(): string | the conversation as JSON, in the shape A.save() gives, so a text agent can restore it |
VoiceEvent
A prelude enum. The provider events (connection start/restart/warning/stop, response start/stop, audio start/delta/stop, text, reasoning, transcript start/delta/stop, barge-in, usage, tool-use blocks) are folded into what a program acts on. Reasoning is kept in the history, not surfaced.
| Case | When |
|---|---|
Connected(model) | the connection is up (again after each restart) |
Speech(Audio) | a piece of the reply's audio |
Hearing(text) / Heard(text) | a piece of / the final transcript of what the user said |
Saying(text) / Said(text) | a piece of / the final transcript of what the agent said |
Wrote(text) | a text reply (models that answer in text as well) |
CallingTool { name, input } | the model called a tool; it runs now, while audio keeps flowing |
ToolReturned { name, output, ok } | the result, which goes back to the model with the others of its group |
BargedIn | the user talked over the reply |
TurnDone | a response finished (a tool call is a turn of its own) |
TokensUsed { input, output } | usage for a response |
RestartSoon(seconds) | a proactive restart is coming |
Reconnected { reason, interrupted } | the connection was replaced ("timeout" or "scheduled"); interrupted if a turn was cut |
HungUp(reason) | the end: "stopped", "ended" (the provider closed it) |
voice agent V { model: "mock" }
fn main() {
let s = V.open()
match s.next() { Said(t) => print(t), HungUp(r) => print(r) }
} error: match is not exhaustive: missing Connected, Speech, Hearing, Heard, Saying, Wrote, CallingTool, ToolReturned { name: _, output: _, ok: true }, BargedIn, TurnDone, TokensUsed, RestartSoon, Reconnected { reason: _, interrupted: true }
--> main.tin:4:3
|
4 | match s.next() { Said(t) => print(t), HungUp(r) => print(r) }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= note: add the missing arms, or a catch-all `_ => ...`
= fix: add arms for Connected, Speech, Hearing, Heard, Saying, Wrote, CallingTool, ToolReturned { name: _, output: _, ok: true }, BargedIn, TurnDone, TokensUsed, RestartSoon, Reconnected { reason: _, interrupted: true } Audio, mic and speaker
Audio is 16-bit mono sound held by the host; a program holds a handle. The host keeps the last 8192
chunks, so play or send a chunk soon after you get it.
let a = tone(440Hz, 250ms)
print(a.seconds())
print(a.rate())
print(a.level() > 0.1)
print(silence(1s).level())
print(a) 0.25
24000
true
0.0
Audio(0.25s) mic(): AudioIn (read(): Option<Audio>) and speaker(): AudioOut
(play(a), clear()) come from the host:
| Host | mic() | speaker() |
|---|---|---|
tin run --mic in.wav --speaker out.wav | the WAV file, in 100 ms chunks at real-time pace | everything played, written as a 24 kHz WAV at exit |
tin run --mic - | raw 24 kHz PCM16 on stdin | |
tin run with sox installed | the default microphone (rec) | the default speaker (play), cleared on barge-in |
tin run on macOS without sox | none (an error names the flag) | afplay, once per reply: not streamed |
| the playground / a browser host | getUserMedia + an AudioWorklet, echo cancellation on | an AudioWorklet queue that clear() empties at once |
| none of these | a panic that says how to give one | |
Tools, approval, taint, capabilities
- The same tools. The model gets the same JSON Schemas as a text agent's. A group of tool calls
runs concurrently, while the connection keeps streaming; the results go back together. A tool not in
tools:never runs, even if the model names it; a failing tool is a failed result the model hears. - Approval. A
guarded tool fnasks a person first (a dialog in the playground, a prompt or--yeson the CLI, denied undertin test). Handlers may only call the agent's own tools. - Taint. Everything a session hears or says (
next(),talk(),save()) is model output: it may not reach anextern fnexcept throughapprove. - Capabilities.
micandspeakerare capabilities likenet:tin capsreports which functions and agents reach them.
import js "./shell.mjs"
extern fn run(cmd: string): string
voice agent V { model: "mock" }
fn main() {
let s = V.open()
match s.next() {
Heard(t) => print(run(t))
_ => {}
}
} error: in main: a value from a model's reply reaches extern fn run without anyone approving it
--> main.tin:4:1
|
4 | fn main() {
| ^^^^^^^
= note: main gets text from agent.ask / agent.try and passes something computed from it to run
= note: put the call in a `guarded tool fn` and call it with `approve`, so a person sees the exact arguments first
= note: or match the reply against cases and call with values the program writes itself Barge-in
When the user speaks over a reply, the provider says so (OpenAI: speech starts while the reply is still playing;
Gemini: interrupted; Nova Sonic: an INTERRUPTED content end). The session ends the response,
emits BargedIn, runs on bargeIn, and talk() clears the speaker in the same step,
so the old reply stops at once. s.interrupt() does the same from the program's side.
Restarts
Live connections have a limit. Each provider declares when to reconnect proactively: OpenAI after 45 minutes,
Gemini after 9, Nova Sonic after 7 (or restartAfter). Ten seconds before, the session emits
RestartSoon; at the deadline it waits up to 10 s for the current turn to finish, then swaps the
connection: sends wait at a gate, the conversation is replayed (Gemini resumes the server session with its handle
instead), a tool result that arrives after the swap is handed over as text (Nova), and Reconnected says
whether a turn was cut. A provider timeout (OpenAI session_expired, Gemini GoAway, Nova
ModelTimeoutException) restarts the same way, reactively.
Speech-to-speech models
model: | Protocol | Needs | Browser | Tested |
|---|---|---|---|---|
"openai-realtime:gpt-realtime-mini" | WebSocket wss://api.openai.com/v1/realtime, 24 kHz, server VAD, transcription with gpt-4o-mini-transcribe | OPENAI_API_KEY (OPENAI_REALTIME_URL, OPENAI_REALTIME_TRANSCRIBE optional) | yes: the key goes in the WebSocket subprotocol, since a browser can't set headers | fixtures; live: connects and authenticates, replies not yet verified (the key here had no credits) |
"gemini-live:gemini-live-2.5-flash-preview" | WebSocket BidiGenerateContent, 16 kHz in, 24 kHz out, session resumption | GEMINI_API_KEY or GOOGLE_API_KEY | yes | fixtures only: no working Gemini key was available |
"bedrock-sonic:amazon.nova-sonic-v1:0" | HTTP/2 InvokeModelWithBidirectionalStream, event-stream body with each event signed (SigV4), 16 kHz | AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY (+ AWS_SESSION_TOKEN), AWS_REGION (default us-east-1). Bedrock API keys are refused by this operation. | no: a browser fetch can't send and receive on one request at once | fixtures, and the signatures match the AWS SDK signer byte for byte; live: not yet (no access keys here) |
"mock" | offline, scripted | nothing | yes | every test |
The mock hears speech by loudness (a stretch of speech, then 400 ms of quiet, is a turn; it hears it as
"(1.2 s of speech)"), calls the tools a turn names, answers "mock: …" with a tone of about
60 ms per character in 100 ms chunks, and treats speech while it answers as a barge-in.
npm run smoke:bidi runs two live sessions per provider that has credentials: a whole call through a Tin
voice agent (a recorded question, a tool call mid-call, the reply's audio, an answer that uses the tool) and a
barge-in. It never prints keys and keeps to a budget of sessions. npm run smoke:bidi -- mock runs the same
cases offline.
From JavaScript
A host passes voice to instantiate: models(name) returns a
BidiModel (start / send / receive / stop, optional restart and interrupt; see src/bidi.ts),
and mic() / speaker() return an audio source (read()) and sink
(play, clear). src/voice-models.ts builds the three providers from environment
variables; src/audio-node.ts has the Node backends; src/audio-browser.ts and
src/voice-worker.ts are the page and Web Worker halves used by the playground.
Not supported
- Voice agents without a JavaScript host: not on
--target wasior--native. - WebRTC transport (one browser visitor is served by the WebSocket path), video or image input, and tool interrupts in the middle of a call.
- A voice agent as another agent's tool, or another agent as a voice agent's tool (wrap it in a
tool fn). - Record and replay of live sessions: tests use the mock and scripted provider fixtures instead.