An AI that watches you work

When I tell people the warehouse safety simulator has an AI instructor watching the trainee, they picture something cinematic: a vision model streaming the rendered frame, understanding the scene like a human supervisor would. That is not what shipped, and the reason it is not is the most useful thing the project taught me.

The scene already knows everything

A game engine is not a camera feed. It is a database wearing a camera as a costume. At any moment, Unity can tell you — exactly, for free — where the trainee is standing, what they are looking at, how fast the forklift is moving, and whether the fire route is blocked.

So the instructor never "sees" pixels. It receives a compact, structured account of the moment:

{
  "zone": "loading-dock",
  "gaze": "pallet-stack-04",
  "hazardsActive": ["forklift-crossing"],
  "trainee": { "speed": 1.4, "helmet": true },
  "violations": []
}

The AI's job is judgment, not perception: given this situation and this training goal, is the trainee doing the right thing, and if not, what is the shortest useful correction?

This split — engine handles perception, model handles judgment — cut latency, cost, and hallucination all at once. The model cannot misread a scene it never has to read.

Feedback is a design problem

The second lesson had nothing to do with AI. Early versions of the instructor were right and useless: it interrupted constantly, narrated the obvious, and trainees started ignoring it — the exact behavior safety training exists to prevent.

What fixed it was ruthless interaction design:

• Silence is the default. The instructor speaks only on a violation, a near-miss, or a direct question.
• Corrections name the behavior, not the person. "Load is blocking route B" lands better than "you blocked route B."
• Every session ends in a written debrief. What you saw, what you did, what you missed — a report a supervisor can actually file.

An AI that talks less is an AI people listen to.

Where this goes

The pattern — structured world-state in, short judgment out — is not a warehouse trick. It works for any 3D or simulation product that wants intelligence inside the scene: museum guides that know which exhibit you have lingered on, onboarding flows that watch what users actually do instead of where they click.

The model is the easy part now. The craft is in deciding what the world tells it, and when it is allowed to speak.