<!-- Talking agents · Tin · https://tin.ad/docs/ref/bidi/ · generated from the built page -->

[Reference](https://tin.ad/docs/ref/)

-  [The language](https://tin.ad/docs/ref/language/)
-  [Agents and tools](https://tin.ad/docs/ref/agents/)
-  [Model providers](https://tin.ad/docs/ref/providers/)
-  [The Anthropic API](https://tin.ad/docs/ref/anthropic/)
-  [Talking agents](https://tin.ad/docs/ref/bidi/)
- [Declaring a voice agent](#declare)
- [Sessions](#session)
- [VoiceEvent](#events)
- [Audio, mic and speaker](#audio)
- [Tools, approval, taint, capabilities](#rules)
- [Barge-in](#bargein)
- [Restarts](#restart)
- [Speech-to-speech models](#providers)
- [From JavaScript](#host)
- [Not supported](#limits)
-  [Standard library](https://tin.ad/docs/ref/stdlib/)
-  [Glossary](https://tin.ad/docs/ref/glossary/)
-  [Package docs](https://tin.ad/docs/ref/api/)
-  [Telemetry](https://tin.ad/docs/ref/telemetry/)
-  [Robots](https://tin.ad/docs/ref/robot/)
-  [Servers](https://tin.ad/docs/ref/server/)
-  [Deploying](https://tin.ad/docs/ref/deploy/)
-  [tin.ad](https://tin.ad/docs/ref/cloud/)
-  [The tin command](https://tin.ad/docs/ref/cli/)
-  [How it works](https://tin.ad/docs/ref/internals/)

**Reference · Talking agents

- [The language](https://tin.ad/docs/ref/language/)
- [Agents and tools](https://tin.ad/docs/ref/agents/)
- [Model providers](https://tin.ad/docs/ref/providers/)
- [The Anthropic API](https://tin.ad/docs/ref/anthropic/)
- [Talking agents](https://tin.ad/docs/ref/bidi/)
- [Standard library](https://tin.ad/docs/ref/stdlib/)
- [Glossary](https://tin.ad/docs/ref/glossary/)
- [Package docs](https://tin.ad/docs/ref/api/)
- [Telemetry](https://tin.ad/docs/ref/telemetry/)
- [Robots](https://tin.ad/docs/ref/robot/)
- [Servers](https://tin.ad/docs/ref/server/)
- [Deploying](https://tin.ad/docs/ref/deploy/)
- [tin.ad](https://tin.ad/docs/ref/cloud/)
- [The tin command](https://tin.ad/docs/ref/cli/)
- [How it works](https://tin.ad/docs/ref/internals/)

On this page

- [Declaring a voice agent](#declare)
- [Sessions](#session)
- [VoiceEvent](#events)
- [Audio, mic and speaker](#audio)
- [Tools, approval, taint, capabilities](#rules)
- [Barge-in](#bargein)
- [Restarts](#restart)
- [Speech-to-speech models](#providers)
- [From JavaScript](#host)
- [Not supported](#limits)

# Talking agents

Voice agents: speech-to-speech sessions with tools, barge-in and restarts. Implemented for OpenAI Realtime, Gemini Live and Nova Sonic and tested against a mock; no real provider has produced a spoken reply yet.

A `voice agent` holds one live, two-way connection to a speech-to-speech model. Audio and text go in
while audio, transcripts and tool calls come out, at the same time. The design follows the bidirectional agent in
strands-agents/harness-sdk (its event taxonomy, its loop and its restart rules), written as a language feature: the
declaration, the events, the capabilities and the tool rules are checked by the compiler. The tutorial chapter is
[Talking agents](https://tin.ad/docs/learn/voice/).

**What has been verified.** Whole sessions run end to end with the offline `"mock"` model, in tests and
in the playground with a fake microphone. The three real providers are implemented and tested against recorded fixtures;
OpenAI Realtime has connected and authenticated live, but **no real provider has produced a spoken reply yet**
([details](#providers), [status](https://tin.ad/docs/status/#providers)).

## Declaring a voice agent

```tin
/// The weather in a city.
tool fn weather(city: string): string => "18C in {city}"

voice agent Concierge {
  model: "mock"
  system: "Answer in one short sentence."
  voice: "marin"
  tools: [weather]
  maxTokens: 4000
  restartAfter: 300s
  on heard(text) => print("user: {text}")
  on said(text) => print("agent: {text}")
  on toolCall(name, input) => print("tool {name} {input}")
}

fn main() {
  let s = Concierge.open()
  s.say("weather in 'Rome'")
  var done = false
  while !done {
    match s.next() {
      Said(t) => s.stop()
      HungUp(r) => { done = true }
      _ => {}
    }
  }
}
```

<!-- output -->
```text
tool weather {"city":"Rome"}
agent: mock: 18C in Rome
```

| Field | Meaning |
| --- | --- |
| `model` | a speech-to-speech model ([below](#providers)) or `"mock"`. Required. A literal is checked at compile time; a computed name when the session connects. |
| `system` | instructions, as for a text agent |
| `voice` | the provider's voice name (`marin`, `Puck`, `tiffany`, …) |
| `tools` | `tool fn`s the model may call mid-conversation (not other agents) |
| `restartAfter` | a `Duration`: reconnect proactively after this long (default: the provider's limit minus a margin) |
| `maxTokens` | accepted for symmetry; providers bound a live response themselves |
| `on …` | `heard(text)`, `said(text)`, `text(t)`, `toolCall(name, input)`, `toolResult(name, output, ok)`, `bargeIn()` |

 `maxTurns` and `code` are compile errors on a voice agent: a live session is not a turn loop
the program drives, and code as action needs a reply to check before it runs. Calling a voice agent like a text
agent is an error with a fix:

```tin
voice agent V { model: "mock" }
fn main() { print(V("hello")) }
```

<!-- compiler output -->
```text
error: V is a voice agent: it holds a live conversation, not one question
 --> main.tin:2:19
  |
2 | fn main() { print(V("hello")) }
  |                   ^
  = note: open a session: `let s = V.open()`, or run a whole call: `V.talk(mic(), speaker())`
  = fix: open a session
```

```tin
voice agent V { model: "openai:gpt-5.1" }
fn main() {}
```

<!-- compiler output -->
```text
error: 'openai:gpt-5.1' is not a speech-to-speech model
 --> main.tin:1:24
  |
1 | voice agent V { model: "openai:gpt-5.1" }
  |                        ^^^^^^^^^^^^^^^^
  = note: a voice agent uses one of: openai-realtime:<model>, gemini-live:<model>, bedrock-sonic:<model>, or "mock"
```

```tin
agent T { model: "openai-realtime:gpt-realtime-mini" }
fn main() {}
```

<!-- compiler output -->
```text
error: 'openai-realtime:gpt-realtime-mini' is a speech-to-speech model; a text agent can't use it
 --> main.tin:1:18
  |
1 | agent T { model: "openai-realtime:gpt-realtime-mini" }
  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  = note: declare a talking agent: `voice agent T { ... }`, and call it with .open() or .talk(mic(), speaker())
```

## Sessions

| Call | Does |
| --- | --- |
| `A.open(): Session` | connects (one connection per call; several can be open at once, each with its own history) |
| `A.talk(input: AudioIn, output: AudioOut): string` | a whole call: the input streams to the model, the reply plays, a barge-in clears the output; returns the transcript (`user: …` / `assistant: …` lines) when the input ends and the model has been quiet for 2 s, or the session closes |
| `s.say(text)` | sends a user message as text |
| `s.send(chunk: Audio)` | streams audio (resampled to the model's rate) |
| `s.next(): VoiceEvent` | waits for the next event; after the end, `HungUp(reason)` every time |
| `s.interrupt()` | cancels the reply being spoken |
| `s.stop()` | closes the connection (waits for running tools) |
| `s.save(): string` | the conversation as JSON, in the shape `A.save()` gives, so a text agent can `restore` it |

## VoiceEvent

A prelude enum. The provider events (connection start/restart/warning/stop, response start/stop, audio
start/delta/stop, text, reasoning, transcript start/delta/stop, barge-in, usage, tool-use blocks) are folded into
what a program acts on. Reasoning is kept in the history, not surfaced.

| Case | When |
| --- | --- |
| `Connected(model)` | the connection is up (again after each restart) |
| `Speech(Audio)` | a piece of the reply's audio |
| `Hearing(text)` / `Heard(text)` | a piece of / the final transcript of what the user said |
| `Saying(text)` / `Said(text)` | a piece of / the final transcript of what the agent said |
| `Wrote(text)` | a text reply (models that answer in text as well) |
| `CallingTool { name, input }` | the model called a tool; it runs now, while audio keeps flowing |
| `ToolReturned { name, output, ok }` | the result, which goes back to the model with the others of its group |
| `BargedIn` | the user talked over the reply |
| `TurnDone` | a response finished (a tool call is a turn of its own) |
| `TokensUsed { input, output }` | usage for a response |
| `RestartSoon(seconds)` | a proactive restart is coming |
| `Reconnected { reason, interrupted }` | the connection was replaced (`"timeout"` or `"scheduled"`); `interrupted` if a turn was cut |
| `HungUp(reason)` | the end: `"stopped"`, `"ended"` (the provider closed it) |

```tin
voice agent V { model: "mock" }
fn main() {
  let s = V.open()
  match s.next() { Said(t) => print(t), HungUp(r) => print(r) }
}
```

<!-- compiler output -->
```text
error: match is not exhaustive: missing Connected, Speech, Hearing, Heard, Saying, Wrote, CallingTool, ToolReturned { name: _, output: _, ok: true }, BargedIn, TurnDone, TokensUsed, RestartSoon, Reconnected { reason: _, interrupted: true }
 --> main.tin:4:3
  |
4 |   match s.next() { Said(t) => print(t), HungUp(r) => print(r) }
  |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  = note: add the missing arms, or a catch-all `_ => ...`
  = fix: add arms for Connected, Speech, Hearing, Heard, Saying, Wrote, CallingTool, ToolReturned { name: _, output: _, ok: true }, BargedIn, TurnDone, TokensUsed, RestartSoon, Reconnected { reason: _, interrupted: true }
```

## Audio, mic and speaker

`Audio` is 16-bit mono sound held by the host; a program holds a handle. The host keeps the last 8192
chunks, so play or send a chunk soon after you get it.

```tin
let a = tone(440Hz, 250ms)
print(a.seconds())
print(a.rate())
print(a.level() > 0.1)
print(silence(1s).level())
print(a)
```

<!-- output -->
```text
0.25
24000
true
0.0
Audio(0.25s)
```

`mic(): AudioIn` (`read(): Option<Audio>`) and `speaker(): AudioOut`
(`play(a)`, `clear()`) come from the host:

| Host | `mic()` | `speaker()` |
| --- | --- | --- |
| `tin run --mic in.wav --speaker out.wav` | the WAV file, in 100 ms chunks at real-time pace | everything played, written as a 24 kHz WAV at exit |
| `tin run --mic -` | raw 24 kHz PCM16 on stdin |  |
| `tin run` with sox installed | the default microphone (`rec`) | the default speaker (`play`), cleared on barge-in |
| `tin run` on macOS without sox | none (an error names the flag) | `afplay`, once per reply: not streamed |
| the playground / a browser host | `getUserMedia` + an AudioWorklet, echo cancellation on | an AudioWorklet queue that `clear()` empties at once |
| none of these | a panic that says how to give one |  |

## Tools, approval, taint, capabilities

- **The same tools.** The model gets the same JSON Schemas as a text agent's. A group of tool calls
runs concurrently, while the connection keeps streaming; the results go back together. A tool not in
`tools:` never runs, even if the model names it; a failing tool is a failed result the model hears.

- **Approval.** A `guarded tool fn` asks a person first (a dialog in the playground, a prompt
or `--yes` on the CLI, denied under `tin test`). Handlers may only call the agent's own tools.

- **Taint.** Everything a session hears or says (`next()`, `talk()`,
`save()`) is model output: it may not reach an `extern fn` except through `approve`.

- **Capabilities.** `mic` and `speaker` are capabilities like `net`:
`tin caps` reports which functions and agents reach them.

```tin
import js "./shell.mjs"
extern fn run(cmd: string): string
voice agent V { model: "mock" }
fn main() {
  let s = V.open()
  match s.next() {
    Heard(t) => print(run(t))
    _ => {}
  }
}
```

<!-- compiler output -->
```text
error: in main: a value from a model's reply reaches extern fn run without anyone approving it
 --> main.tin:4:1
  |
4 | fn main() {
  | ^^^^^^^
  = note: main gets text from agent.ask / agent.try and passes something computed from it to run
  = note: put the call in a `guarded tool fn` and call it with `approve`, so a person sees the exact arguments first
  = note: or match the reply against cases and call with values the program writes itself
```

## Barge-in

When the user speaks over a reply, the provider says so (OpenAI: speech starts while the reply is still playing;
Gemini: `interrupted`; Nova Sonic: an `INTERRUPTED` content end). The session ends the response,
emits `BargedIn`, runs `on bargeIn`, and `talk()` clears the speaker in the same step,
so the old reply stops at once. `s.interrupt()` does the same from the program's side.

## Restarts

Live connections have a limit. Each provider declares when to reconnect proactively: OpenAI after 45 minutes,
Gemini after 9, Nova Sonic after 7 (or `restartAfter`). Ten seconds before, the session emits
`RestartSoon`; at the deadline it waits up to 10 s for the current turn to finish, then swaps the
connection: sends wait at a gate, the conversation is replayed (Gemini resumes the server session with its handle
instead), a tool result that arrives after the swap is handed over as text (Nova), and `Reconnected` says
whether a turn was cut. A provider timeout (OpenAI `session_expired`, Gemini `GoAway`, Nova
`ModelTimeoutException`) restarts the same way, reactively.

## Speech-to-speech models

| `model:` | Protocol | Needs | Browser | Tested |
| --- | --- | --- | --- | --- |
| `"openai-realtime:gpt-realtime-mini"` | WebSocket `wss://api.openai.com/v1/realtime`, 24 kHz, server VAD, transcription with `gpt-4o-mini-transcribe` | `OPENAI_API_KEY` (`OPENAI_REALTIME_URL`, `OPENAI_REALTIME_TRANSCRIBE` optional) | yes: the key goes in the WebSocket subprotocol, since a browser can't set headers | fixtures; live: connects and authenticates, replies not yet verified (the key here had no credits) |
| `"gemini-live:gemini-live-2.5-flash-preview"` | WebSocket BidiGenerateContent, 16 kHz in, 24 kHz out, session resumption | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | yes | fixtures only: no working Gemini key was available |
| `"bedrock-sonic:amazon.nova-sonic-v1:0"` | HTTP/2 InvokeModelWithBidirectionalStream, event-stream body with each event signed (SigV4), 16 kHz | `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` (+ `AWS_SESSION_TOKEN`), `AWS_REGION` (default us-east-1). Bedrock API keys are refused by this operation. | no: a browser `fetch` can't send and receive on one request at once | fixtures, and the signatures match the AWS SDK signer byte for byte; live: not yet (no access keys here) |
| `"mock"` | offline, scripted | nothing | yes | every test |

 The mock hears speech by loudness (a stretch of speech, then 400 ms of quiet, is a turn; it hears it as
`"(1.2 s of speech)"`), calls the tools a turn names, answers `"mock: …"` with a tone of about
60 ms per character in 100 ms chunks, and treats speech while it answers as a barge-in.

`npm run smoke:bidi` runs two live sessions per provider that has credentials: a whole call through a Tin
voice agent (a recorded question, a tool call mid-call, the reply's audio, an answer that uses the tool) and a
barge-in. It never prints keys and keeps to a budget of sessions. `npm run smoke:bidi -- mock` runs the same
cases offline.

## From JavaScript

A host passes `voice` to `instantiate`: `models(name)` returns a
`BidiModel` (start / send / receive / stop, optional restart and interrupt; see `src/bidi.ts`),
and `mic()` / `speaker()` return an audio source (`read()`) and sink
(`play`, `clear`). `src/voice-models.ts` builds the three providers from environment
variables; `src/audio-node.ts` has the Node backends; `src/audio-browser.ts` and
`src/voice-worker.ts` are the page and Web Worker halves used by the playground.

## Not supported

- Voice agents without a JavaScript host: not on `--target wasi` or `--native`.
- WebRTC transport (one browser visitor is served by the WebSocket path), video or image input, and tool
interrupts in the middle of a call.

- A voice agent as another agent's tool, or another agent as a voice agent's tool (wrap it in a `tool fn`).
- Record and replay of live sessions: tests use the mock and scripted provider fixtures instead.
