Tin

Learn · Real programs · 29 of 34

Talking agents

Goal. Hold a live spoken conversation with a voice agent, with tools and interruptions.

The agents so far answer one question at a time: a prompt goes in, the loop runs, a reply comes out. A talking agent is different. It holds one live connection to a speech-to-speech model, and audio goes both ways at the same time. While the model speaks, you can talk over it, and it can call your tools mid-sentence.

In Tin that is a voice agent. It looks like the agents you know, and the compiler checks it the same way.

A session

Concierge.open() connects and gives you a Session. say sends text, send sends audio, and next() waits for the next thing the conversation does:

session.tintin run session.tin
/// The weather in a city.
tool fn weather(city: string): string => "18C and clear in {city}"

voice agent Concierge {
  model: "mock"
  system: "You are a hotel concierge. One short sentence."
  tools: [weather]
}

fn main() {
  let s = Concierge.open()
  s.say("weather in 'Oslo'")
  var answered = false
  loop {
    match s.next() {
      CallingTool { name, input } => print("calling {name} {input}")
      ToolReturned { name, output, ok } => print("{name} gave: {output}")
      Said(text) => {
        print("said: {text}")
        answered = true
      }
      TurnDone => if answered { s.stop() }
      HungUp(reason) => {
        print("hung up: {reason}")
        break
      }
      _ => {}
    }
  }
}
calling weather {"city":"Oslo"}
weather gave: 18C and clear in Oslo
said: mock: 18C and clear in Oslo
hung up: stopped

Every event is a VoiceEvent, an ordinary enum, so match works on it and the compiler knows its cases. The tool call is a turn of its own: the model asks, the runtime runs weather while the connection stays open, and the answer comes in the next turn. That is why the loop hangs up after the spoken answer, not after the first TurnDone.

The common events: Heard(text) and Said(text) are final transcripts of the user and the agent; Hearing and Saying are the pieces as they arrive; Speech(chunk) is a piece of the reply's audio; CallingTool and ToolReturned bracket a tool; BargedIn means the user talked over the reply; HungUp(reason) is the last one, and next() keeps returning it after the call ends.

Handlers and audio

As with text agents, on handlers run as things happen: on heard, on said, on toolCall, on toolResult, on bargeIn. A Speech chunk is an Audio value: 16-bit sound held by the host, not copied into your program. You can ask how long it is, how loud, or play it.

handlers.tintin run handlers.tin
voice agent Echo {
  model: "mock"
  // handlers run as the conversation goes, before next() hands the event on
  on said(text) => print("[said] {text}")
}

fn main() {
  let s = Echo.open()
  s.say("hello there")
  var audio = 0.0
  loop {
    match s.next() {
      Speech(chunk) => {
        audio = audio + chunk.seconds()
      }
      TurnDone => s.stop()
      HungUp(reason) => break
      _ => {}
    }
  }
  print("{audio.fixed(2)} s of reply audio")
}
[said] mock: hello there
1.02 s of reply audio

"mock" answers with a tone, about 60 ms per character, so programs and tests have real audio to work with offline.

A whole call

Most programs don't need the event loop. talk wires a microphone to the model and the model to a speaker, clears the speaker the moment you interrupt, and returns the transcript when the call ends:

fn main() {
  let transcript = FrontDesk.talk(mic(), speaker())
  print(transcript)
}

mic() and speaker() come from the host. In the playground they are your browser's microphone and speakers (it asks first). In the terminal they are files or devices:

tin run call.tin --mic question.wav --speaker reply.wav     # recorded audio in, the reply written out
tin run call.tin                                           # the default microphone and speaker (needs sox)

What the compiler checks

A voice agent is not a function you ask a question, so calling it like one is an error with a fix:

called.tintin check called.tin
voice agent Concierge {
  model: "mock"
}

fn main() {
  print(Concierge("what's the weather?"))
}
error: Concierge is a voice agent: it holds a live conversation, not one question
 --> called.tin:6:9
  |
6 |   print(Concierge("what's the weather?"))
  |         ^^^^^^^^^
  = note: open a session: `let s = Concierge.open()`, or run a whole call: `Concierge.talk(mic(), speaker())`
  = fix: open a session

Its model has to be a speech-to-speech model. A text model would accept the connection and then never speak:

model.tintin check model.tin
voice agent Concierge {
  model: "openai:gpt-5.1"
}

fn main() {}
error: 'openai:gpt-5.1' is not a speech-to-speech model
 --> model.tin:2:10
  |
2 |   model: "openai:gpt-5.1"
  |          ^^^^^^^^^^^^^^^^
  = note: a voice agent uses one of: openai-realtime:<model>, gemini-live:<model>, bedrock-sonic:<model>, or "mock"

The rules from the earlier chapters hold. A tool not in tools: never runs, a guarded tool fn asks a person first, and what the model says may not reach JavaScript unapproved. The microphone and the speaker are capabilities, so tin caps shows who can listen:

listen.tintin caps listen.tin
/// Listen to the room for a moment.
tool fn listen(): string => match mic().read() {
  Some(chunk) => "heard {chunk.seconds()} s"
  None => "silence"
}

/// The time at the desk.
tool fn deskTime(): string => "9:41"

voice agent FrontDesk { model: "mock", tools: [deskTime] }
voice agent Spy { model: "mock", tools: [listen] }

fn main() {}
voice agent FrontDesk
  tools:           deskTime
  can call freely: deskTime
  needs approval:  nothing
  capabilities:    model

voice agent Spy
  tools:           listen
  can call freely: listen
  needs approval:  nothing
  capabilities:    model, mic

tools
  listen    free     mic
  deskTime  free     pure

Real voices

Change "mock" to a speech-to-speech model and nothing else:

model:NeedsRuns in
"openai-realtime:gpt-realtime-mini"OPENAI_API_KEYthe CLI and the browser
"gemini-live:gemini-live-2.5-flash-preview"GEMINI_API_KEYthe CLI and the browser
"bedrock-sonic:amazon.nova-sonic-v1:0"access keys (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)the CLI only

A live connection does not last forever (Nova Sonic about 8 minutes, Gemini about 10, OpenAI an hour). Tin reconnects shortly before the limit, waits for a pause in the conversation to do it, and keeps the conversation. The reference says what has been tested against each provider.

Tasks

Task 1: A host who asks first

book is guarded. Print every tool result and whether it worked, then hang up after the spoken answer. Run it without approval first: what does the model hear?

task-1.tintin run task-1.tin
/// Book a table for a number of people.
guarded tool fn book(people: int): string => "booked for {people}"

voice agent Host {
  model: "mock"
  tools: [book]
}

fn main() {
  let s = Host.open()
  s.say("book a table for 4")
  // print each tool result and whether it worked, then hang up after the spoken answer
}

Checking needs JavaScript; the solution below works without it.

Solution
task-1.solution.tintin run task-1.solution.tin
/// Book a table for a number of people.
guarded tool fn book(people: int): string => "booked for {people}"

voice agent Host {
  model: "mock"
  tools: [book]
}

fn main() {
  let s = Host.open()
  s.say("book a table for 4")
  var answered = false
  loop {
    match s.next() {
      ToolReturned { name, output, ok } => print("{name} ok={ok}: {output}")
      Said(text) => {
        answered = true
      }
      TurnDone => if answered { s.stop() }
      HungUp(reason) => break
      _ => {}
    }
  }
}
book ok=false: Tool call denied by the user. Do not retry it; continue without it.

Recap

  • A voice agent keeps one speech-to-speech session open.
  • Events are typed, and match must handle each kind.
  • The microphone is a capability the compiler reports.

In the reference: Talking agents · VoiceEvent · Speech-to-speech models