# Proximity voice

> Give rooms voice where each listener hears the closest speakers it can see.

Rooms can have voice chat in which each listener hears the speakers near
it. The room decides who hears whom; `@irt/client/voice` joins the voice
server and plays what the room allows.

## How it works

- Voice is audio only and runs on a voice server (LiveKit) in each
  region, outside your VMs. It runs no tenant code.
- A project turns voice on with its `voice` setting. Each connection then
  gets a voice token from the router after `welcome`, in a `voice`
  message (type 7) with the voice server's URL. The voice identity is the
  player ID, stable across reconnects.
- **Who hears whom comes from the room.** With `voice: {}` in
  `defineRoom`, a listener hears the players whose entities it can see
  under the room's [interest](/docs/guides/state-sync#interest), ranked by
  the interest limit's `priority`, capped at `maxSpeakers`.
  An entity belongs to the player in its `owner` string field, unless you
  pass `owner`. `audible(room, listener)` replaces the rule, for teams,
  radio channels or a whole room on one call.
- Each tick the room works out every connection's **speakers** (who it
  hears) and **listeners** (who hears it) and sends them, only when they
  change and at most 4 times a second, in a `peers` message (type 9).
- The client subscribes only to its speakers, and sets its own
  microphone track's permissions to its listeners, so a modified client
  can't overhear anyone and can only expose its own voice. A subscribe
  that arrives before a speaker's permission is retried when the
  permission comes.
- By default each speaker plays through its own `<audio>` element, with
  volume falling linearly to silence at `range`. `spatial: true` pans
  speakers in 3D with Web Audio, played through one `<audio>` element,
  because Chrome's echo cancellation ignores audio played straight from
  Web Audio.

## Room

```ts
// room.ts
import { defineRoom, defineSchema, entity, grid, type RoomEntity } from "@irt/room";

export const Avatar = entity("avatar", { owner: "string", x: "f32", y: "f32" });
export const schema = defineSchema({ entities: [Avatar] });

type A = RoomEntity<typeof Avatar>;

export default defineRoom({
  schema,
  interest: {
    ...grid({ cellSize: 16, radius: 1, dims: "xy", around: (_room, c) => c.data.avatar }),
    limit: {
      max: 64,
      priority: (_room, c, e) => -Math.hypot((e as A).x - (c.data.avatar as A).x, (e as A).y - (c.data.avatar as A).y),
    },
  },
  voice: { maxSpeakers: 6 },
  onJoin(room, conn) {
    conn.data.avatar = [...room.entities.all(Avatar)].find((e) => e.owner === conn.player) ?? room.entities.create(Avatar, { owner: conn.player, x: 0, y: 0 });
  },
  onPlayerLeave(room, player) {
    for (const e of room.entities.all(Avatar)) if (e.owner === player) room.entities.delete(e);
  },
});
```

`Voice` options on `defineRoom({ voice })`:

| Option | Default | Meaning |
|---|---|---|
| `maxSpeakers` | `8`, or `256` with `audible` | Speakers each listener hears, at most 256. The project's `voice.maxSpeakers` setting is a ceiling: a room gets the smaller of the two |
| `owner(entity)` | its `owner` string field | The player an entity speaks for |
| `audible(room, listener)` | speakers it can see | Player IDs the listener hears, replacing the default rule |

## Whole room on one call

Party games and calls want everyone to hear everyone, whatever they can
see. Return every connection's player from `audible`; the listener's own
ID is dropped for you:

```ts
// call-room.ts
import { defineRoom, defineSchema } from "@irt/room";

export const schema = defineSchema({});

export default defineRoom({
  schema,
  voice: { audible: (room) => [...room.connections.values()].map((c) => c.player) },
});
```

With `audible` the room's own cap defaults to 256, and the project's
`voice.maxSpeakers` setting, 64 by default, caps it, so a call of up to
65 players needs no setting. On the client, leave
out `position` so every speaker plays at full volume; see
[Client](#client).

## Muting a player

`room.muteVoice(player)` mutes a player in this room, for good or until
`{ expiresAt }` in wall-clock milliseconds; `room.unmuteVoice(player)`
lifts it, and `room.voiceMutes()` lists them. The room leaves a muted
player out of every listener's speakers, whatever `audible` or the
default rule says, so no client the room drives subscribes to it, and
its own client sees no listeners. Mutes are kept in room state, like
bans, so they last through hibernation and a room host restart.

```ts
calls: {
  mute: (room, conn, args: { player: string }) => {
    if (conn.player === room.creator) room.muteVoice(args.player);
  },
},
```

The voice server enforces a mute too: the player can't publish while
it lasts, and its voice tokens can't either, so neither a modified
client nor a reconnect gets around it. A player on the voice server is
muted at once, and one who joins it with a token from before the mute
is muted as it joins; should the voice server miss either, within a
minute. When the mute ends, `Voice` turns the player's microphone back
on if it was on, unless the player muted it with `setMuted`. The server
API's `voice/mute` is a one-off mute of the player's tracks, which its
own client can undo.

## Client

```ts
// client.ts
import { connect } from "@irt/client";
import { joinVoice } from "@irt/client/voice";
import { Avatar, schema } from "./room.js";

const room = await connect({ router: "wss://eu.irt.io", project: "my-game", room: "plaza", schema });

const voice = joinVoice(room, {
  position(player) {
    const a = [...room.entities.all(Avatar)].find((e) => e.owner === player);
    return a && { x: a.x, y: a.y };
  },
  range: 24,
});

document.querySelector("#talk")!.addEventListener("click", () => void voice.enableMic());
document.querySelector("#mute")!.addEventListener("click", () => void voice.setMuted(!voice.muted));
```

```ts
function joinVoice(room: RoomClient, opts?: VoiceOptions): Voice;

interface VoiceOptions {
  position?(player: string): { x: number; y: number; z?: number } | undefined;
  spatial?: boolean; // default false
  range?: number; // default 20
}

class Voice {
  readonly project: string;                      // the room connection's project
  readonly muted: boolean;                       // this player's mute, from setMuted
  readonly micOn: boolean;                       // enableMic called and not muted
  readonly participants: ReadonlySet<string>;    // other players on the voice server
  readonly speaking: ReadonlySet<string>;        // players speaking now, this one included
  readonly mutedPlayers: ReadonlySet<string>;    // other players on voice sending no audio
  on(event: "participants" | "speaking" | "mutedPlayers", fn: (players: ReadonlySet<string>) => void): () => void;
  on(event: "micError", fn: (error: Error) => void): () => void;
  enableMic(): Promise<void>;
  setMuted(muted: boolean): Promise<void>;
  disconnect(): Promise<void>;

  // Player controls; see below
  volume(player: string): number;                // 0 to 1, default 1
  setVolume(player: string, volume: number, keep?: boolean): void;
  readonly blocked: ReadonlySet<string>;
  setBlocked(player: string, blocked: boolean): void;
  devices(): Promise<{ inputs: VoiceDevice[]; outputs: VoiceDevice[] }>;  // VoiceDevice: { id, label }
  readonly input: string | undefined;            // chosen microphone; undefined for the default
  readonly output: string | undefined;           // chosen speaker; undefined for the default
  readonly canSetOutput: boolean;                // the browser has setSinkId
  setInput(deviceId: string): Promise<void>;
  setOutput(deviceId: string): Promise<void>;
  readonly level: number;                        // microphone input level, 0 to 1
  readonly pushToTalk: boolean;
  setPushToTalk(on: boolean): Promise<void>;
  talk(down: boolean): Promise<void>;
}
```

- **`position`** places players for volume and panning, and a speaker
  it returns undefined for is silent. Leave it out and every speaker
  plays at full volume, unpanned, as on a call; `spatial` and `range`
  then do nothing.
- **`participants`** is the set of other players' IDs on the voice
  server, whether or not the room has this player hear them: everyone
  a voice chat list should show. A player in the room who hasn't joined
  voice, or whose voice dropped, isn't in it. The `participants` event
  fires with the new set each time a player joins or leaves voice.
- **`speaking`** is the set of player IDs the voice server hears talking
  now, this player's included, so it covers speakers this client doesn't
  hear too. The `speaking` event fires with the new set each time it
  changes.
- **`enableMic`** while the room mutes this player publishes nothing
  until the mute ends, then turns the microphone on. Called before voice
  has joined, it turns the microphone on when it does.
- **`micError`** fires when the microphone fails to start, as when the
  player refuses permission, whether from `enableMic` or on joining.
  The microphone is then off again, so `micOn` is false and the next
  `enableMic` asks again; `enableMic` also rejects with the error. An
  error named `NotAllowedError` is a refusal.
- **`mutedPlayers`** is the other players on the voice server sending no
  audio: muted with `setMuted`, or not yet past `enableMic`. The
  `mutedPlayers` event fires with the new set; `voice.muted` stays this
  player's own mute. A player who isn't on the voice server
  is in neither set.
- The three sets empty when voice drops and fill again after it rejoins, so
  keep the same listeners across reconnects; there's nothing to rebind.
  `voice.livekit` is replaced on each reconnect, so listeners on it
  would need binding again.

### Player controls

These are the player's own settings, which the project shows in its UI;
the [UI kit](/docs/ui-kit)'s `<irt-voice-settings>` and
`<irt-voice-chat>` are built on them.

- **`setVolume(player, volume)`** sets how loud this player hears
  another, from 0 to 1, multiplied into the distance attenuation on both
  the `<audio>` and the `spatial` path. It can't go above 1: an
  `<audio>` element's volume is capped there, and gain past it would
  need Web Audio, which Chrome's echo cancellation ignores. With `keep`
  false it isn't saved, for a slider being dragged; save the value it
  stops at.
- **`setBlocked(player, true)`** is a local mute: the client never
  subscribes to a blocked player, whatever the room picks, so a blocked
  player costs no bandwidth. Other players still hear them. To silence
  a player for everyone, the room calls `muteVoice`.
- Volumes and blocks are kept in localStorage per project by player
  ID, so a player blocked in one room stays blocked in the next. Where
  the browser blocks storage they last for the page.
- **`devices()`** lists microphones and speakers. Labels are empty until
  the page has microphone permission, so list them again after
  `enableMic`. `setInput` and `setOutput` switch devices, and a device
  chosen before voice joins is used when it does. Speakers can only be
  chosen where the browser has `setSinkId`, which `canSetOutput`
  reports; elsewhere `devices()` lists none and `setOutput` throws.
- **`level`** is the microphone's input level from 0 to 1, 0 while it
  isn't sending. Read it on animation frames for a meter.
- **`setPushToTalk(true)`** keeps the microphone muted except between
  `talk(true)` and `talk(false)`. `Voice` doesn't bind keys; the page
  does, and should ignore key presses in text fields:

```ts
// push-to-talk.ts
import { connect } from "@irt/client";
import { joinVoice } from "@irt/client/voice";

const room = await connect({ router: "wss://eu.irt.io", project: "my-party", room: "lobby" });
const voice = joinVoice(room);
await voice.setPushToTalk(true);

const typing = (e: KeyboardEvent) => e.target instanceof HTMLInputElement || e.target instanceof HTMLTextAreaElement;
addEventListener("keydown", (e) => {
  if (e.code === "KeyV" && !e.repeat && !typing(e)) void voice.talk(true);
});
addEventListener("keyup", (e) => {
  if (e.code === "KeyV") void voice.talk(false);
});
document.querySelector("#mic")!.addEventListener("click", () => void voice.enableMic());
```

### Indicators

```ts
// indicators.ts
import { connect } from "@irt/client";
import { joinVoice } from "@irt/client/voice";

const room = await connect({ router: "wss://eu.irt.io", project: "my-party", room: "lobby" });
const voice = joinVoice(room);

function badge(player: string): string {
  if (voice.mutedPlayers.has(player)) return "muted";
  return voice.speaking.has(player) ? "speaking" : "";
}

function show(): void {
  const onVoice = [...room.presence].filter((p) => voice.participants.has(p.player));
  console.log(onVoice.map((p) => `${p.player} ${badge(p.player)}`));
}

for (const event of ["participants", "speaking", "mutedPlayers"] as const) voice.on(event, show);
```

The voice follows the room connection: it leaves the voice server when
the connection drops, joins again with the new token after the
reconnect, and ends when the connection closes for good. Without the
project's `voice` setting the router sends no token, and `joinVoice`
does nothing. `room.peers` (`{ speakers, listeners }`) and the `peers`
event are on every `RoomClient`, so a project can show who's talking
without `@irt/client/voice`, which is a separate entry so projects without
voice don't load `livekit-client`.

`livekit-client` is most of a voice game's bundle, about 570 KB
minified. Import `@irt/client/voice` dynamically, once the game has
connected or when the player opts in, and the scaffold's `build-web.mjs`, which bundles with esbuild's
`splitting`, puts it in a chunk of its own that loads only then:

```ts
// lazy-voice.ts
import { connect } from "@irt/client";

const room = await connect({ router: "wss://eu.irt.io", project: "my-party", room: "lobby" });

const { joinVoice } = await import("@irt/client/voice");
const voice = joinVoice(room);
document.querySelector("#talk")!.addEventListener("click", () => void voice.enableMic());
```

A build script of your own needs `splitting: true`, `format: "esm"` and
`outdir` rather than `outfile` for this; without `splitting`, esbuild
inlines the dynamic import into the main bundle. Keep `enableMic` in a
click handler of its own, after the import has resolved, since browsers
grant the microphone only from a user gesture.

## Testing

Test clients expose the room's decision, so proximity voice is testable
without a voice server:

```ts
// voice.test.ts
import { testRoom } from "@irt/testing";
import { expect, it } from "vitest";
import room, { Avatar } from "./room.js";

it("hears nearby players", async () => {
  const t = testRoom(room);
  const alice = await t.connect({ player: "alice" });
  const bob = await t.connect({ player: "bob" });
  await t.host.tick();
  expect(alice.speakers).toEqual(["bob"]);
  expect(bob.listeners).toEqual(["alice"]);
  const far = [...t.live.entities.all(Avatar)].find((a) => a.owner === "bob")!;
  far.x = 500;
  await t.host.tick(10);
  expect(alice.speakers).toEqual([]);
});
```

## Turning it on

Voice is a project setting:

| Setting | Default | Meaning |
|---|---|---|
| `voice.enabled` | `false` | Connections get a voice token |
| `voice.maxSpeakers` | `64` | 1 to 64. A ceiling on every room's speakers per listener: a room gets the smaller of this and its own `maxSpeakers`, whose default of 8 is unchanged for proximity voice. A change applies to each room from its next wake |

Declare them in [`irt.config.ts`](/docs/cli#irtconfigts):

```ts
settings: { voice: { enabled: true } },
```

Locally, voice is off unless you run a LiveKit server and point `irt dev`
at it; `peers` messages arrive either way, so the room's rule can be
checked without one:

```sh
docker run -d --rm --network host livekit/livekit-server:v1.9.1 --dev --bind 127.0.0.1
IRT_LIVEKIT_URL=ws://127.0.0.1:7880 npx @irt/cli dev
```

`IRT_LIVEKIT_KEY` and `IRT_LIVEKIT_SECRET` default to LiveKit's dev
mode `devkey` and `secret`. The token `irt dev` mints has the grants the
router's does: identity the player ID, LiveKit room `{project}/{room}`,
publishing only the microphone, and subscribing.

## Common mistakes

- **Leaving out `voice` in `defineRoom`.** Without it the room sends no
  `peers`, and every client hears no one.
- **An interest that shows everyone everything.** With no interest, every
  connection sees every entity, so a listener hears the first
  `maxSpeakers` players rather than the nearest.
- **A `priority` that ranks far entities higher.** Priority is "higher is
  more important"; use the negative of the distance.
- **Entities without an owner.** The default rule only counts entities
  with an `owner` string field naming a player; pass `owner` otherwise.
- **Calling `enableMic` outside a click.** Browsers only grant the
  microphone from a user gesture.
- **Expecting voice to keep a room awake.** Rooms hibernate 30 seconds
  after their last connection closes, whatever voice is doing.