Tablebox DocsSite 日本語

Agent

The Agent is an AI that people talk to by voice or text and can interrupt. It runs on your AI provider account with a model that hears and speaks directly (OpenAI Realtime or Gemini Live), listens to the broadcast it is called on, answers in voice and text, calls your app's tools and reads the context your app gives it.

Paths

{source} is a broadcast path, usually the caller's own (alice/camera.hang).

Written by Path Track Content
The caller {source} its hang catalog and audio The voice the Agent hears: the first audio rendition
The caller {source} context.json (optional) A JSON snapshot track: context for the model
The caller {source} input.json (optional) Each frame one JSON object: {"text": string}, a typed message, or {"call": id, "output": value}, a tool's result
The Agent .agent/{source} a hang catalog and audio The Agent's voice (Opus)
The Agent the same transcript.json Each frame {"id", "role": "user" or "agent", "text", "final": boolean, "interrupted"?: true}; a later frame with the same id replaces that item's text
The Agent the same calls.json Each frame a tool call {"id", "name", "arguments": object}
The Agent the same status.json A JSON snapshot track {"state", "code"?, "message"?}; state is connecting, listening, thinking, speaking or failed

Reading .agent/{source} makes the Agent hear {source}, so grant it as you grant {source}: subscribe .agent/{id}/** lets each person call the Agent on their own broadcasts. transcript.json and calls.json are MoQ JSON streams (json.Stream), so a reader that comes late reads the conversation so far.

TypeScript, browser · agent.ts
// A typed conversation with the Agent: messages go on input.json of {me}/chat, and the
// conversation comes from .agent/{me}/chat.
import { type Connection, json } from 'tablebox.io';

interface Item { id: string; role: 'user' | 'agent'; text: string; final: boolean; interrupted?: true }

export async function chat(connection: Connection, me: string, show: (item: Item) => void) {
  const input = connection.publish(`${me}/chat`).createTrack('input.json');
  const agent = await connection.read(`.agent/${me}/chat`);
  const transcript = new json.Stream.Consumer<Item>({ track: agent.track('transcript.json').subscribe() });
  // A later item with the same id replaces that item's text.
  void (async () => { for await (const item of transcript) show(item); })();
  return (text: string) => input.writeJson({ text });
}

Talking

Reading any track of .agent/{source} starts one conversation for {source}, shared by every reader of the path; it lasts while the path is read and ends 20 s after its last reader leaves. The Agent hears the source's audio and its typed messages and answers in voice and text; both sides' words appear on transcript.json as they are recognised and spoken. To talk by voice, call it on the broadcast that carries your microphone, such as {id}/camera.hang from Get started, and play .agent/{id}/camera.hang with the official player. A source with no audio talks by typed messages alone.

When the caller starts speaking or sends a text while the Agent speaks, the Agent stops at once and marks that answer interrupted. When the provider ends a session at its length limit, the Agent opens a new one and gives it the conversation so far as text.

Tools and context

The model calls the tools listed in AGENT_TOOLS. Each call appears on calls.json, and the app answers on input.json with the call's id; an answer that has not come within 10 s reaches the model as a failed call. The newest context.json value, up to 64 KiB of its JSON, is given to the model whenever it changes.

Variables

Variable Meaning
AGENT_PROVIDER openai (OpenAI Realtime, the default) or gemini (Gemini Live)
OPENAI_API_KEY, GEMINI_API_KEY The chosen provider's key, required
AGENT_MODEL The provider's model; gpt-realtime-2.1 for OpenAI and gemini-3.8-live for Gemini when unset
AGENT_VOICE The provider's voice; the provider's default when unset
AGENT_INSTRUCTIONS The instructions given to the model
AGENT_TOOLS A JSON array of {"name", "description", "parameters"}, parameters a JSON Schema; none when unset

The variables are the project's, so one configuration serves every conversation of the project.

Failures