Tin
Reference · Talking agents

On this page

Talking agents

Voice agents: speech-to-speech sessions with tools, barge-in and restarts. Implemented for OpenAI Realtime, Gemini Live and Nova Sonic and tested against a mock; no real provider has produced a spoken reply yet.

A voice agent holds one live, two-way connection to a speech-to-speech model. Audio and text go in while audio, transcripts and tool calls come out, at the same time. The design follows the bidirectional agent in strands-agents/harness-sdk (its event taxonomy, its loop and its restart rules), written as a language feature: the declaration, the events, the capabilities and the tool rules are checked by the compiler. The tutorial chapter is Talking agents.

What has been verified. Whole sessions run end to end with the offline "mock" model, in tests and in the playground with a fake microphone. The three real providers are implemented and tested against recorded fixtures; OpenAI Realtime has connected and authenticated live, but no real provider has produced a spoken reply yet (details, status).

Declaring a voice agent

/// The weather in a city.
tool fn weather(city: string): string => "18C in {city}"

voice agent Concierge {
  model: "mock"
  system: "Answer in one short sentence."
  voice: "marin"
  tools: [weather]
  maxTokens: 4000
  restartAfter: 300s
  on heard(text) => print("user: {text}")
  on said(text) => print("agent: {text}")
  on toolCall(name, input) => print("tool {name} {input}")
}

fn main() {
  let s = Concierge.open()
  s.say("weather in 'Rome'")
  var done = false
  while !done {
    match s.next() {
      Said(t) => s.stop()
      HungUp(r) => { done = true }
      _ => {}
    }
  }
}
tool weather {"city":"Rome"}
agent: mock: 18C in Rome
FieldMeaning
modela speech-to-speech model (below) or "mock". Required. A literal is checked at compile time; a computed name when the session connects.
systeminstructions, as for a text agent
voicethe provider's voice name (marin, Puck, tiffany, …)
toolstool fns the model may call mid-conversation (not other agents)
restartAftera Duration: reconnect proactively after this long (default: the provider's limit minus a margin)
maxTokensaccepted for symmetry; providers bound a live response themselves
on …heard(text), said(text), text(t), toolCall(name, input), toolResult(name, output, ok), bargeIn()

maxTurns and code are compile errors on a voice agent: a live session is not a turn loop the program drives, and code as action needs a reply to check before it runs. Calling a voice agent like a text agent is an error with a fix:

voice agent V { model: "mock" }
fn main() { print(V("hello")) }
error: V is a voice agent: it holds a live conversation, not one question
 --> main.tin:2:19
  |
2 | fn main() { print(V("hello")) }
  |                   ^
  = note: open a session: `let s = V.open()`, or run a whole call: `V.talk(mic(), speaker())`
  = fix: open a session
voice agent V { model: "openai:gpt-5.1" }
fn main() {}
error: 'openai:gpt-5.1' is not a speech-to-speech model
 --> main.tin:1:24
  |
1 | voice agent V { model: "openai:gpt-5.1" }
  |                        ^^^^^^^^^^^^^^^^
  = note: a voice agent uses one of: openai-realtime:<model>, gemini-live:<model>, bedrock-sonic:<model>, or "mock"
agent T { model: "openai-realtime:gpt-realtime-mini" }
fn main() {}
error: 'openai-realtime:gpt-realtime-mini' is a speech-to-speech model; a text agent can't use it
 --> main.tin:1:18
  |
1 | agent T { model: "openai-realtime:gpt-realtime-mini" }
  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  = note: declare a talking agent: `voice agent T { ... }`, and call it with .open() or .talk(mic(), speaker())

Sessions

CallDoes
A.open(): Sessionconnects (one connection per call; several can be open at once, each with its own history)
A.talk(input: AudioIn, output: AudioOut): stringa whole call: the input streams to the model, the reply plays, a barge-in clears the output; returns the transcript (user: … / assistant: … lines) when the input ends and the model has been quiet for 2 s, or the session closes
s.say(text)sends a user message as text
s.send(chunk: Audio)streams audio (resampled to the model's rate)
s.next(): VoiceEventwaits for the next event; after the end, HungUp(reason) every time
s.interrupt()cancels the reply being spoken
s.stop()closes the connection (waits for running tools)
s.save(): stringthe conversation as JSON, in the shape A.save() gives, so a text agent can restore it

VoiceEvent

A prelude enum. The provider events (connection start/restart/warning/stop, response start/stop, audio start/delta/stop, text, reasoning, transcript start/delta/stop, barge-in, usage, tool-use blocks) are folded into what a program acts on. Reasoning is kept in the history, not surfaced.

CaseWhen
Connected(model)the connection is up (again after each restart)
Speech(Audio)a piece of the reply's audio
Hearing(text) / Heard(text)a piece of / the final transcript of what the user said
Saying(text) / Said(text)a piece of / the final transcript of what the agent said
Wrote(text)a text reply (models that answer in text as well)
CallingTool { name, input }the model called a tool; it runs now, while audio keeps flowing
ToolReturned { name, output, ok }the result, which goes back to the model with the others of its group
BargedInthe user talked over the reply
TurnDonea response finished (a tool call is a turn of its own)
TokensUsed { input, output }usage for a response
RestartSoon(seconds)a proactive restart is coming
Reconnected { reason, interrupted }the connection was replaced ("timeout" or "scheduled"); interrupted if a turn was cut
HungUp(reason)the end: "stopped", "ended" (the provider closed it)
voice agent V { model: "mock" }
fn main() {
  let s = V.open()
  match s.next() { Said(t) => print(t), HungUp(r) => print(r) }
}
error: match is not exhaustive: missing Connected, Speech, Hearing, Heard, Saying, Wrote, CallingTool, ToolReturned { name: _, output: _, ok: true }, BargedIn, TurnDone, TokensUsed, RestartSoon, Reconnected { reason: _, interrupted: true }
 --> main.tin:4:3
  |
4 |   match s.next() { Said(t) => print(t), HungUp(r) => print(r) }
  |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  = note: add the missing arms, or a catch-all `_ => ...`
  = fix: add arms for Connected, Speech, Hearing, Heard, Saying, Wrote, CallingTool, ToolReturned { name: _, output: _, ok: true }, BargedIn, TurnDone, TokensUsed, RestartSoon, Reconnected { reason: _, interrupted: true }

Audio, mic and speaker

Audio is 16-bit mono sound held by the host; a program holds a handle. The host keeps the last 8192 chunks, so play or send a chunk soon after you get it.

let a = tone(440Hz, 250ms)
print(a.seconds())
print(a.rate())
print(a.level() > 0.1)
print(silence(1s).level())
print(a)
0.25
24000
true
0.0
Audio(0.25s)

mic(): AudioIn (read(): Option<Audio>) and speaker(): AudioOut (play(a), clear()) come from the host:

Hostmic()speaker()
tin run --mic in.wav --speaker out.wavthe WAV file, in 100 ms chunks at real-time paceeverything played, written as a 24 kHz WAV at exit
tin run --mic -raw 24 kHz PCM16 on stdin
tin run with sox installedthe default microphone (rec)the default speaker (play), cleared on barge-in
tin run on macOS without soxnone (an error names the flag)afplay, once per reply: not streamed
the playground / a browser hostgetUserMedia + an AudioWorklet, echo cancellation onan AudioWorklet queue that clear() empties at once
none of thesea panic that says how to give one

Tools, approval, taint, capabilities

  • The same tools. The model gets the same JSON Schemas as a text agent's. A group of tool calls runs concurrently, while the connection keeps streaming; the results go back together. A tool not in tools: never runs, even if the model names it; a failing tool is a failed result the model hears.
  • Approval. A guarded tool fn asks a person first (a dialog in the playground, a prompt or --yes on the CLI, denied under tin test). Handlers may only call the agent's own tools.
  • Taint. Everything a session hears or says (next(), talk(), save()) is model output: it may not reach an extern fn except through approve.
  • Capabilities. mic and speaker are capabilities like net: tin caps reports which functions and agents reach them.
import js "./shell.mjs"
extern fn run(cmd: string): string
voice agent V { model: "mock" }
fn main() {
  let s = V.open()
  match s.next() {
    Heard(t) => print(run(t))
    _ => {}
  }
}
error: in main: a value from a model's reply reaches extern fn run without anyone approving it
 --> main.tin:4:1
  |
4 | fn main() {
  | ^^^^^^^
  = note: main gets text from agent.ask / agent.try and passes something computed from it to run
  = note: put the call in a `guarded tool fn` and call it with `approve`, so a person sees the exact arguments first
  = note: or match the reply against cases and call with values the program writes itself

Barge-in

When the user speaks over a reply, the provider says so (OpenAI: speech starts while the reply is still playing; Gemini: interrupted; Nova Sonic: an INTERRUPTED content end). The session ends the response, emits BargedIn, runs on bargeIn, and talk() clears the speaker in the same step, so the old reply stops at once. s.interrupt() does the same from the program's side.

Restarts

Live connections have a limit. Each provider declares when to reconnect proactively: OpenAI after 45 minutes, Gemini after 9, Nova Sonic after 7 (or restartAfter). Ten seconds before, the session emits RestartSoon; at the deadline it waits up to 10 s for the current turn to finish, then swaps the connection: sends wait at a gate, the conversation is replayed (Gemini resumes the server session with its handle instead), a tool result that arrives after the swap is handed over as text (Nova), and Reconnected says whether a turn was cut. A provider timeout (OpenAI session_expired, Gemini GoAway, Nova ModelTimeoutException) restarts the same way, reactively.

Speech-to-speech models

model:ProtocolNeedsBrowserTested
"openai-realtime:gpt-realtime-mini"WebSocket wss://api.openai.com/v1/realtime, 24 kHz, server VAD, transcription with gpt-4o-mini-transcribeOPENAI_API_KEY (OPENAI_REALTIME_URL, OPENAI_REALTIME_TRANSCRIBE optional)yes: the key goes in the WebSocket subprotocol, since a browser can't set headersfixtures; live: connects and authenticates, replies not yet verified (the key here had no credits)
"gemini-live:gemini-live-2.5-flash-preview"WebSocket BidiGenerateContent, 16 kHz in, 24 kHz out, session resumptionGEMINI_API_KEY or GOOGLE_API_KEYyesfixtures only: no working Gemini key was available
"bedrock-sonic:amazon.nova-sonic-v1:0"HTTP/2 InvokeModelWithBidirectionalStream, event-stream body with each event signed (SigV4), 16 kHzAWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY (+ AWS_SESSION_TOKEN), AWS_REGION (default us-east-1). Bedrock API keys are refused by this operation.no: a browser fetch can't send and receive on one request at oncefixtures, and the signatures match the AWS SDK signer byte for byte; live: not yet (no access keys here)
"mock"offline, scriptednothingyesevery test

The mock hears speech by loudness (a stretch of speech, then 400 ms of quiet, is a turn; it hears it as "(1.2 s of speech)"), calls the tools a turn names, answers "mock: …" with a tone of about 60 ms per character in 100 ms chunks, and treats speech while it answers as a barge-in.

npm run smoke:bidi runs two live sessions per provider that has credentials: a whole call through a Tin voice agent (a recorded question, a tool call mid-call, the reply's audio, an answer that uses the tool) and a barge-in. It never prints keys and keeps to a budget of sessions. npm run smoke:bidi -- mock runs the same cases offline.

From JavaScript

A host passes voice to instantiate: models(name) returns a BidiModel (start / send / receive / stop, optional restart and interrupt; see src/bidi.ts), and mic() / speaker() return an audio source (read()) and sink (play, clear). src/voice-models.ts builds the three providers from environment variables; src/audio-node.ts has the Node backends; src/audio-browser.ts and src/voice-worker.ts are the page and Web Worker halves used by the playground.

Not supported

  • Voice agents without a JavaScript host: not on --target wasi or --native.
  • WebRTC transport (one browser visitor is served by the WebSocket path), video or image input, and tool interrupts in the middle of a call.
  • A voice agent as another agent's tool, or another agent as a voice agent's tool (wrap it in a tool fn).
  • Record and replay of live sessions: tests use the mock and scripted provider fixtures instead.