Invariants and load runs

Run irtio simulate against a running room to check synchronization, bandwidth, and behavior under load. Bots use the client SDK and real connections. Start locally, then repeat against a deployed project.

Running it

Start the room, then point the simulator at it.

npx irtio dev              # in one terminal
npx irtio simulate         # in another

The defaults are 5 bots for 10 seconds against ws://localhost:7070.

Bot 0 joins first and creates the room when no --room was given, and the rest join the code it came back with. Without that, twenty bots would create twenty empty rooms and every invariant would pass while nothing was tested.

What it needs from your project

The command loads irtio/schema.ts (or irtio/schema.js, or schema.ts) and bundles it the way irtio dev does. That module has to export a schema as schema or as its default export, and the error message says so when it does not.

Everything the default bots do is derived from that schema: which collections a client may own, what each field’s declared type will accept, which server RPCs take no return value. That is what lets the command work on a room it has never seen.

Two things follow from this that will surprise you if you do not expect them:

• No schema module means a relay run. The command falls back to a schema-less relay simulation, where bots broadcast small payloads on a timer, and it says so in the report. That is the honest thing to do for a relay room, and it is a clear signal that you ran it from the wrong directory.
• A physics room needs irtio/world.ts. With a shared world-builder module exporting gravity, and optionally setup, bodies, and intents, the bots predict physics locally exactly as a browser client does, and body-field corrections are judged as real mispredictions in world units. Without it the bots interpolate instead, those corrections count as body sync, and the report tells you the numbers mean less than they look like they do. The command also builds the world twice and compares the engine’s own snapshots byte for byte. A builder that reads Math.random(), a clock, or module state fails the run there, before its nondeterminism can be misread as bad netcode.

The invariants

liveness fails when bots receive frames but no join snapshot or delta. It reports n/a when no bot receives any frames, including runs with no bots.

denial-rate counts expected refusals from ctx.deny(reason) separately from handler errors. It appears when a run makes RPC calls and reports n/a until there are 20 calls. Configure the ceiling with --denial-rate-max or denialsRateMax in spawnBots; use 1 to allow any refusal rate.

party-integrity is listed only on a run that queued (see Matchmaking below). A run that built its own room has no parties to be intact and no ticket to be over-filled, so the row is absent rather than reported as unread.

misprediction and snaps have no cap by default, so they report their measured peak and mean without ever failing. They only become gates when you set a threshold yourself: pass --misprediction-max <units> or --snaps-max <n> on the CLI, or the equivalent mispredictionMagnitudeMax / snapsMax through spawnBots. correction-storm is the one invariant of the three that is not opt-in — it fails on its own default of 5 corrections/s per bot, and --corrections-max (or correctionsPerSecMax through spawnBots) raises it rather than waives it.

handler-error is the one invariant with a budget rather than a hard zero, because a room that refuses an illegal RPC is working correctly. Raise handlerErrorsMax for a run that expects some rejections.

Tick health

tick-health reads the dev server’s tick counters. ok means overruns stayed within --overruns-max; FAIL means they exceeded it. n/a means the counters were unavailable, not that the check passed. The n/a line names its reason: a probe that timed out under the run’s own load says so, separately from a server that refused the connection or answered 404 (a deployed tenant does not serve the inspector). A run with no baseline reading at its start is also n/a, rather than a window measured against zero. For a deployed project, inspect irtio logs alongside the run.

Prediction on all three engines

When your project has a shared world-builder module, the bots predict physics locally exactly as a browser client does, and body-field corrections are judged as real disagreements instead of counted as authority arriving. That works on all three blessed engines. The engine comes from the room’s own physics.engine: a rapier3d world is handed to joinRoom({ physics }), and a rapier2d or matter2d world to joinRoom({ physics2d }), along with whichever of timestep, setup, bodies, intents, settle (matter2d only), epsilon, maxPredictedBodies and smoothingHalfLifeMs it exports. If the room will not load, the shape of the exported gravity is the fallback, where a { x, y, z } means rapier3d and a { x, y } means matter2d.

Two 2D worlds are recognised and honestly not predicted from, and the run says which in one sentence: a module exporting no bodies (there is nothing to build), and an inert one, meaning no intents, zero gravity, and no settle where matter2d would accept one, so nothing in a local world built from it could ever move a body. A room that applies its forces by hand inside tick() has no steering hook to share and is the usual reason for the second. Export the block your client passes to joinRoom({ physics2d }) and the run predicts with it.

Before any engine’s bots start, the world-builder is built twice and compared, so a builder that reads Math.random(), a clock or module state fails the run rather than showing up later as mispredictions blamed on netcode. Both Rapier checks compare two engine snapshots; matter.js has no snapshot, so its check steps both worlds sixty times and compares every body’s pose and velocity.

The report prints a prediction line when any bot held a local world:

  prediction: 4/4 bot(s) held a local world · peak 12 proxied, 0 absent · 284 misprediction(s), 4527 within-epsilon, 0 proxied, 0 body-sync

The three kinds are what the local world holds for an instance, and each means something different when the server corrects it:

• predicted — simulated locally. A correction here is a real misprediction, unless its values matched the prediction within the epsilon, in which case it is within-epsilon: authority confirming the local world rather than disagreeing with it.
• proxied — a collider at the pose the renderer draws, which predicted bodies stand on and which never moves on its own. A correction here is the server disagreeing with what you drew, which is the interpolation residual. It is counted and reported, and it has no threshold.
• absent — no body in the local world at all. Predicted bodies pass straight through these, so a non-zero absent on a collection anything stands on is a gameplay bug rather than a fidelity tradeoff, and the run says so in words. Raise maxProxyBodies, or give the collection a body factory on the client.

epsilon is the one number that surprises people. It is a distance in your world’s own units, and its default (0.05) is sized for a metre-scale world. matter.js’s own convention is pixels, and at pixel scale that default calls every ordinary contact resolution a misprediction. A world in pixels should export its own.

Matchmaking

By default every bot lands in one room: bot 0 creates it and the rest join the code it came back with, because twenty bots creating twenty empty rooms would pass every invariant while testing nothing. --queue turns that off and puts the bots through the front door your game actually has.

npx irtio simulate --queue duo --control https://irt.io --bots 8
npx irtio simulate --queue duo --control https://irt.io --bots 8 --party 2

Each bot calls the real match API, waits for a ticket, and joins whatever room the ticket names — through the ordinary join, so nothing about it knows a matchmaker filled the room. --party <n> groups each n consecutive bots into one party: one of them mints the code, all of them send it, and the queue fills only when the whole party is waiting.

--queue needs --control, and is refused without it before any socket opens. Quick-match is a control-plane surface and irtio dev runs none, so there is no queue at a dev server to join. Point it at a real control plane: your project’s, or a test one your suite starts.

The report gains a matchmaking section:

  matchmaking
  8 ticket(s), 4 room(s), wait p50 12 ms p95 41 ms max 44 ms, 0 timeout(s), parties intact 4/4
  every bot was answered, and every party landed in one room.

A ticket that never came is a failure, named per bot with the code the matchmaker used, and the run exits 1. A queue that did not fill is one of those: it comes back E_NO_MATCH at the deadline rather than hanging, and it counts as a timeout in the line above. Timeouts are excluded from the wait percentiles, because folding a deadline into a wait distribution turns “nobody came” into “everybody waited exactly the timeout”.

party-integrity is the invariant that comes with it: every party’s members share a room, and no room holds more bots than its ticket said it seats. It is on by default and there is nothing to raise — a split party is a bug in the matchmaker, not a threshold.

Reading the report

The command prints the bot count, duration, endpoint, and room code, then one line per invariant, then a summary.

  ok   schema-validity   every snapshot and delta decoded
  ok   visibility-leak   no collection reached a role that cannot see it
  ok   bandwidth         peak 9120 B/s in per bot, budget 128000 B/s
  ok   handler-error     0 error reply/frame(s), tolerated 0
  ok   correction-storm  184 correction(s), peak 3/s per bot, threshold 5/s
  ok   misprediction     peak 2.4 units, mean 0.7 over 184 correction(s)
  ok   snaps             no snaps (every correction replayed within the resim window)
  ok   disconnects       every bot stayed connected
  ok   tick-health       0 tick overrun(s) in the run window, tolerated 0, worst tick 4.1 ms

  612 frames/s · in 9.1 kB/s/bot · out 1.2 kB/s/bot · 184 corrections · misprediction mean 0.7 max 2.4 · 0 snap(s) · convergence 34 ms (p50 of 412)
  trace: .irtio/sim/trace-2026-08-27T09-14-02-113Z.json

  every invariant held across 20 bots.

The numbers worth watching:

• Convergence is the headline. It is the median milliseconds from one bot writing a value to another bot’s view showing it, sampled from real writes crossing the seam. When it says “convergence not sampled”, nothing converged during the run, which usually means the bots never owned anything to write.
• Bytes in per bot is what one player’s connection costs. Compare it to the budget you intend to hold, not to the 128,000 B/s default, which is a ceiling rather than a target. If your room filters by area of interest, a budget near the default would pass even if filtering silently regressed to sending everything.
• Corrections counts the server overruling a client. A steady trickle is normal in a room with a strict validate. A storm means either your validator is too tight for what your client predicts, or the client is predicting something the server computes differently.
• Snaps counts corrections that arrived too late to replay and had to jump the client’s state instead. Those are the visible ones. Zero is the number you want.

A failing run names each broken invariant and its first few offenders in the same line, and the process exits 1.

Making the bots cheat

--cheat inverts what the default bots do. Instead of small steps around a sensible value, they write the far end of an integer range and five-order-of-magnitude jumps on a float: type-valid, play-illegal, exactly what a modified client sends. Every one of those should draw a CORRECT from your room.

npx irtio simulate --bots 10 --cheat

Cheating is per bot. --cheat means all of them; --cheat-bot 0 means bot 0 alone, and the flag repeats. One hostile client among honest ones is the situation most rooms actually meet, and it is also the one that tells you the most: the report says which bot cheated, how many corrections it drew, and whether the honest bots drew any (they should not, and a validator that corrects legal play is a different bug).

npx irtio simulate --bots 5 --cheat-bot 0

If a cheating bot draws no corrections at all, that is not a clean bill of health, and the report says so on its own line:

  adversarial
  HOLE bot 0 cheated and drew 0 corrections: the room accepted every illegal write it sent.
       Add rules to `validate` in irtio/room.ts: reject by returning `prev`, or clamp the value.

It means the room accepted every out-of-range and teleporting write the bot threw at it, which is what the permissive validate from irtio init does until you write rules into it. See Server authority for how to write those rules.

Injecting network conditions

Every bot is a real socket, so the run can make that socket behave badly. --conditions takes a JSON object and applies it to every bot:

npx irtio simulate --bots 4 --conditions '{"rttMs":200,"jitterMs":20,"loss":0.02}'

Two rules are worth knowing before you read a result.

Loss, duplication and reorder touch state frames only. Writes, deltas, corrections and messages can be dropped; the join handshake, pings and RPC replies never are. A run under 100% loss therefore still connects, still plays, and still ends, which is what makes the setting useful rather than just a way to hang a run.

The same seed injects the same delays. The dice are seeded per bot index, so two runs of the same command drop the same frames.

The report says what was asked for and what actually happened, per bot:

  adversarial
  bot 0   rtt 200 ms, jitter 20 ms, loss 2%
      dropped 4 out / 31 in, duplicated 0/0, reordered 0/0

For a split run, --conditions-bot 0:'{"rttMs":200}' gives one bot its own conditions. Past two or three bots that flag gets unreadable, and a scenario is the better home: a scenario takes conditions as a function of the bot index, in TypeScript, where a lagged shooter against a clean target is one line.

This is a different instrument from the latency option in @irtio/testing. That one runs in process on a fake clock with no sockets, which is what makes testRoom fast and exactly repeatable. This one is a real socket being treated badly in real time. The vocabulary is the same on purpose; the two are not interchangeable, and a number measured under one does not transfer to the other.

Hit registration

A bot can record that it fired at something:

bot.shot({ collection: 'targets', target: 'T', aim: { x: seen.x, y: seen.y } });

That call sends nothing. Your own RPC still does the shooting; this sits beside it and says what was aimed at, and when. Afterwards the run correlates each shot with the recorded authoritative timeline and prints where the server actually had the target when the shot arrived:

  hit registration (6 shot(s))
  bot 0 at targets.T: tick 7 (estimated), aimed (x 32.08, y 23.89), authority (x 17.43, y 36), missed by 19.02
  mean miss 17.97 over 6 shot(s). Nothing here compensates for lag; this is what a player at this
  latency would experience.

The word estimated is there because it is: the receive tick is derived from when the bot fired plus the uplink half of whatever latency was injected into it. The point is to let you see the size of the problem on your own room before deciding whether it is one.

Exact rows, and rewound ones

An estimate is only needed while the bot has to guess which tick its shot landed on. If your own RPC answers with the tick it judged the shot at, hand that number back and the row becomes exact:

const result = await bot.room.call.fire({ x: seen.x, y: seen.y });
bot.shot({
  collection: 'targets',
  target: 'T',
  aim: { x: seen.x, y: seen.y },
  serverTick: result.tick,     // the tick the room says it judged this shot at
  rewound: result.rewound,     // whether it used room.rewind to get there
});

The row is then correlated to exactly that frame, with no clock compared against any other clock, and (estimated) is gone. A room that uses lag compensation reports the tick its rewind answered from, which is the tick the shooter was looking at, and the report says so:

  hit registration (8 shot(s))
  bot 0 at targets.T: tick 22 (rewound), aimed (x -468, y 100), authority (x -468, y 100), missed by 0.00
  mean miss 0.00, worst 0.00 over 8 shot(s). 8 of 8 shot(s) were resolved through room.rewind,
  against the world the shooter was looking at.

The miss distance means the same thing in both reports: how far the aim was from where authority had the target at the tick the shot was judged. Lag compensation is what moves that tick, and the number follows it. Running the same room twice, once with the rewind and once without, is the measurement.

Hit rows need the recorded timeline, so they need a scenario and a local irtio dev room.

The truth seam

--truth takes a save of the room at the end of the run, decodes it, and compares it against what each client actually received:

npx irtio simulate --bots 4 --truth

  truth seam
  save 00001788039894659-46c9 at tick 43, blob format v1
  ok   bot 0   7 record(s) held, and every one matched the save
  every record the 3 client(s) held matched the decoded save, and none was missing from it.

Read that verdict exactly as it is written. It says that everything each client held agrees with authoritative state, and that nothing a client held is missing from it. It does not say the client received everything the room holds: a client legitimately never sees entities outside its area of interest or outside its role’s visibility, and the comparison is per client for that reason.

A difference names the bot, the collection, the field, and both values:

  FAIL bot 0   2 difference(s) across 6 record(s) held
       bot 0 has targets.T.x = 6.924, the save has 33.085

A difference fails the run, the way any other measured violation does. A run that could not take a save at all does not: the section says why, and an unread number is never reported as a passing one.

The save is taken when the run ends, so a room that keeps changing on its own after that moment will differ from its clients, and the run will say so. That is a true statement about the run rather than a false alarm, but it is worth knowing before you point --truth at a room with a busy tick loop.

--truth needs a local irtio dev room, which is where a save can be taken and read back.

Writing your own bots

The default script wanders. It cannot know that your world is 800 units wide or that a player has to cross a boundary for the thing you want to measure to happen. When you need a specific behaviour, write a script and drive it with spawnBots from @irtio/bots, which is what the command uses underneath.

A script is one async function per bot. It gets a seeded random generator, a wait that returns early when the run ends, and the bot’s real client room.

// irtio/bot.ts
import type { BotScript } from '@irtio/bots';

import type { schema } from './schema.js';

const WIDTH = 800;
const HEIGHT = 500;
const SPEED = 12;

/**
 * Sweeps each bot across the world on its own fixed heading, bouncing off the edges. Every bot
 * covers ground on every run, which a random walk cannot promise.
 */
export const sweep: BotScript<typeof schema> = async (bot) => {
  if (bot.role !== 'player') return;

  // The golden angle, so no two bots trace the same path whatever the bot count.
  const heading = bot.index * 2.399963229728653 + 0.31;
  let dx = Math.cos(heading) * SPEED;
  let dy = Math.sin(heading) * SPEED;

  while (!bot.stopped) {
    const me = bot.room.state.players.get(bot.id);
    if (me) {
      if (me.x + dx < 0 || me.x + dx > WIDTH) dx = -dx;
      if (me.y + dy < 0 || me.y + dy > HEIGHT) dy = -dy;
      me.x += dx;
      me.y += dy;
      bot.room.flush();
    }
    await bot.wait(50);
  }
};

Then a test that spawns them, waits for the run to finish, and asserts on the report. This one needs a room to talk to, so it skips itself when there is nothing running. The CI page shows how to start one first.

// irtio/simulate.test.ts
import { spawnBots } from '@irtio/bots';
import { describe, expect, it } from 'vitest';

import { sweep } from './bot.js';
import { schema } from './schema.js';

const url = process.env.IRTIO_URL ?? '';

describe.skipIf(url === '')('20 bots on one room', () => {
  it('holds every invariant inside a tighter bandwidth budget', async () => {
    const runner = await spawnBots(20, {
      url,
      schema,
      role: 'player',
      name: (i) => `bot-${i}`,
      script: sweep,
      durationMs: 6_000,
      // A quarter of the global default. This room filters by area of interest, so the
      // default ceiling would pass even if filtering stopped working.
      budgetBytesPerSec: 32_000,
    });

    // `done()` waits for the scripts to run their full duration. `stop()` on its own would
    // end the run immediately and report on whatever had happened so far.
    await runner.done();
    const report = await runner.stop();

    for (const invariant of report.invariants) {
      expect(invariant.ok, `${invariant.name}: ${invariant.detail}`).toBe(true);
    }
    expect(report.convergenceLagMs).toBeLessThan(200);

    await runner.trace.save('.irtio/sim/trace.json');
  }, 60_000);
});

spawnBots takes the same three threshold options the CLI’s flags set — mispredictionMagnitudeMax, snapsMax, and correctionsPerSecMax — plus two the CLI does not expose: budgetBytesPerSec and handlerErrorsMax. Setting the first two of those five is how you turn misprediction and snaps from reports into gates, the same as --misprediction-max and --snaps-max do from the command line. It also takes seed (bot i uses seed + i, so a run replays exactly), role and name as a value or a function of the bot index, flushMs, and rpc implementations shared by every bot.

runner.report() is safe to call mid-run, which is how you assert on a snapshot of the numbers without stopping the bots. runner.scriptErrors holds anything a script threw, and runner.done() rejects with the first one.

Choosing bot behavior

Use default bots for broad schema coverage. Write a script for a specific path through the game, such as entering an area, taking a turn, or reaching a win condition. Use scenarios to assert against the recorded server timeline.

Reading a trace

Every run writes a trace, and --trace <path> chooses where. It is JSON: one entry per frame per bot, in time order, headers only.

{
  "startedAt": 1756284842113,
  "bots": 20,
  "dropped": 0,
  "entries": [
    { "at": 12, "bot": 0, "dir": "out", "type": 1, "frame": "HELLO", "bytes": 48 },
    { "at": 31, "bot": 0, "dir": "in", "type": 2, "frame": "WELCOME", "bytes": 412,
      "note": "joined 7KQ2 as c1/player" },
    { "at": 1084, "bot": 3, "dir": "in", "type": 8, "frame": "CORRECT", "bytes": 37,
      "note": "corrected players (clientTick 208)" }
  ]
}

at is milliseconds since the run started, and startedAt turns that back into wall time. dir is out for client to server and in for server to client. note is set when a frame is worth a word: a decode failure, an error code, an RPC name, a correction’s client tick.

When an invariant goes red, find the first offending frame in the report line, then find its timestamp in the trace and read the frames around it. The interesting part is almost never the frame itself, it is what the bot sent in the fifty milliseconds before it.

One caveat: the trace is a fixed-size ring of 4096 frames per bot. A long or busy run overwrites its own beginning, and dropped counts how many entries went. A non-zero dropped means you are reading a tail, not the whole run, so shorten --seconds or reduce --bots when you need the start.

Limits worth knowing

A cold, freshly-deployed project fails its first run on E_STARTING. --url and --key point the run at a deployed project instead of irtio dev. A tenant that was asleep answers the first joins with E_STARTING, a non-fatal error frame, and handler-error tolerates zero of those:

FAIL handler-error  1 error reply/frame(s), tolerated 0; bot 0: E_STARTING: your game server is starting

That is the invariant working as written, not a bug in your room — it cannot tell a waking tenant from a room answering badly, and there is no flag to exempt it. Warm the tenant first, by connecting once from a browser or by running a short throwaway simulate, so the run you measure starts against a project that is already up. See Reference: errors for E_STARTING.

The per-IP connection cap is 120 new connections per minute, and it cannot be raised for a deployed project. A run of more bots than that from one address is refused with E_RATE_LIMITED. Run against local irtio dev for a bigger count, or split the bots across several source addresses.

Live runs check role visibility, not spatial visibility. A bot connects over a socket and has no server truth to compare against, so visibility-leak here judges whether a frame named a collection outside the bot’s role view. Rooms using visibility: 'spatial-grid' need toHaveNoVisibilityLeaks in a testRoom test, which does have real authority to compare against and does catch spatial leaks. Keep that assertion even when a bot run covers the same room.

Hundreds of bots on one machine measure the machine. A run of twenty real client sessions is real load on a shared CI box. Runs in the hundreds are worth doing, but treat them as a local or staging exercise and read the numbers as a profile rather than a pass or fail.

A physics room is judged more loosely by default. Contact-heavy moments produce one correction opportunity per tick, so a busy physics room on slow hardware can burst past the cursor-tuned correction threshold without anything being wrong. Raise correctionsPerSecMax for those rooms rather than accepting a red run as noise.

Related

• Scenarios for your own assertions against the server’s recorded timeline, on top of these invariants
• Testing in CI for wiring both kinds of test into a push