What jobs can an AI actually take inside a 3D world? Role by role
AI agent 3D world use casesAI tour guideAI docentvirtual world AI rolesembodied AI applicationsself-hosted 3D platform AIGenesis virtual world
The access problem is already solved: one domain, one Key, and an environment that runs Node 18 — and an AI can walk into your self-hosted 3D world as a humanoid character. But for many teams the harder question starts right after: it got in — what do we actually assign it to do?
Between "can connect" and "can work" sits a job list. This article lays that list out: four kinds of work you can assign today, what each one depends on, what you have to prepare yourself, and what they all still cannot do. What is implemented and what is on the roadmap are kept separate.
One: think in job roles, not in API lists
A common mistake is going the other way around: reading the interface list first and trying to figure out what those interfaces could be combined into.
It works better to first answer "what kind of character do I want in this world," then check the capability list to see whether it fits. This article is organized by role; each role comes with the implemented capabilities it relies on, the scenarios it suits, and the part it cannot do.
Two: four kinds of work you can assign now
1. Guiding: tours and walk-throughs
The job: visitors enter the world and don't know where to go; it walks them from one point of interest to the next and talks along the way.
Implemented capabilities it relies on:
- Move to coordinates, continuous follow (with configurable stop distance and maximum duration — designed for guiding);
- Speech goes through the regular chat pipeline, delivered within 30 meters, with a bubble above the visitor's head;
- Every object in the world can carry an "AI description" written in the editor (up to 500 characters), delivered along with the spatial radar — so it knows what each stop is.
Suits: exhibition routing, campus walk-throughs, themed tours in online galleries.
What you prepare: lay out the stops on the route and write a clear AI description for each. If the description is vague, all it knows is "there is something there."
2. Explaining: standing by the exhibit and answering questions
The job: a visitor stops in front of an exhibit and asks; it answers using the exhibit's materials.
Implemented capabilities it relies on:
- Spatial radar: it knows who is nearby, how far, and facing where (Key-identity sensing radius up to 200 meters);
- Object descriptions are delivered with the radar, so it knows what it is standing next to;
- Chat context survives reconnects, so a network hiccup doesn't break the conversation.
What you prepare: the industry knowledge base is built locally by you — the platform only provides three raw inputs (object descriptions, chat history, spatial radar) and does not host knowledge bases. In other words, how well it answers depends on how clearly you organized your material.
To be honest about: it cannot see. It knows nothing about visitors' expressions or gazes — only the structured information the system feeds it. "Reading the room" is not something it can do today.
3. Greeting and standing by: the eyes at the entrance
The job: wait at the entrance, greet people as they approach, point them in the right direction; stay quiet when nobody is around.
Implemented capabilities it relies on:
- Real-time event stream with three configurable levels (off / aggregated per second / per-event); new visitors appearing and moving are both observable;
- Movement speed is server-capped at human scale (1–20 m/s, default 9), so it cannot teleport up to a visitor;
- After 5 minutes idle it leaves the world automatically and does not hold a slot.
Suits: exhibition entrance greeting, event check-in guidance, duty roles during opening hours.
To be honest about: this is "standing at the door," not "watching the whole floor" — its awareness is bounded by the radar radius; beyond it, it doesn't know.
4. Practicing and process guidance: walking through the steps
The job: guide a visitor through a predefined procedure or learning path, answering questions, correcting course, prompting the next step.
Implemented capabilities it relies on: moving, speaking, following, interaction actions, and per-command execution receipts (accepted / completed / rejected, with explicit notice when superseded) — the primitives needed for process orchestration are all there.
What you prepare: the process content and domain knowledge, orchestrated entirely by your own AI client. The platform provides the channel that lets it move and speak inside the world; the process logic lives outside that channel.
To be honest about: the actual outcome of this category depends entirely on your process design and knowledge base quality. The platform side cannot guarantee training results — and this article will not invent a number it hasn't measured.
Three: the common foundation under all four roles
| Capability | Status |
|---|---|
| Three-step access (discover capability manifest → Key for a 15-minute token → connect to the agent channel) | Implemented |
| Spatial radar, object descriptions, real-time event stream, chat context | Implemented |
| Move, follow, rotate, speak, interact, action receipts | Implemented |
| Humanoid presentation, explicit AI identity (join notice, AI prefix overhead) | Implemented |
| Backend: create / disable / delete agents, issue and revoke Keys | Implemented |
| Zero-dependency sample client (full chain runnable on Node 18+) | Implemented |
Four: what none of them can do
All four roles share one capability set — and one boundary:
| Cannot do today | Why |
|---|---|
| See the scene | Structured radar only; no camera-style vision |
| Listen and speak by voice | Voice channel is off by default; transcription and speech are handled by your own AI client |
| Host a knowledge base | Built by you locally; the platform provides raw inputs only |
| Terrain adherence | The server holds no terrain or collision data; movement is flat-plane plus bounds |
| Teleport or touch assets | Same rules as human visitors: no inventory, no shop |
| Act autonomously | It is a programmable character; behavior comes from your model and prompts |
One product rule worth noting: the AI identity is always explicit — a join notice and an AI prefix overhead. Humans always know they are talking to an AI.
Five: still on the roadmap (planned, not current)
Listed for completeness — none of these are implemented yet:
- Templated docents (planned): pick a template in the backend, fill in a knowledge base, get a ready "docent" — today, assigning any of the four roles requires a code-capable client;
- World-wrapped MCP Server (planned): let MCP-capable AI tools enter a world directly;
- Agent store (planned) and agent-to-agent interconnection (planned).
Six: FAQ
Q: Which of the four roles is easiest to start with?
A: Guiding and greeting. They rely mostly on platform-side capabilities (movement, speech, radar); what you prepare is stop descriptions and greeting rules. Explaining and practicing add a knowledge-base layer, and their timeline depends on how fast you can organize your material.
Q: How many people does this take?
A: Platform-side configuration is done in the backend. The real workload is content — stop descriptions, knowledge base, process design. That work belongs to your business staff, not to a platform team.
Q: What if the AI says something wrong?
A: Chat history lands on your own server in real time and is archived per configuration, with audit and revocation mechanisms. Before launch, limit the knowledge base scope, set fallback answers for sensitive topics, and spot-check the archive. Backend tooling for behavior governance is still being refined.
Q: Can these roles coexist?
A: Yes. Measured push traffic is about 1 KB/s per agent — one hundred agents in the same scene is about 0.8 Mbps — and server load is roughly 1% of a single core. Concurrency caps are set in the backend.
Seven: Source and repositories
The three addresses have identical content; the first two are faster for visitors in mainland China. The repositories include deployment guides and a demo entry.
- Gitee (faster in mainland China): https://gitee.com/miduoxinxijeji/miduo.git
- GitCode (mirror): https://gitcode.com/qq_35054471/virtual-world
- GitHub: https://github.com/miduo100/3d-virtual-world
About Genesis
Genesis is a self-hosted 3D virtual world system built on Three.js + WebGL, helping individuals and businesses build their own 3D spaces. Accessible directly from a browser, compatible with both PC and mobile, it supports multiplayer online, federated teleportation, a shop system, and Agent integration—where an AI can enter your world as an embodied character. Your data runs on your own server, never passing through a third-party platform—so every world truly belongs to its owner.
Want AI to take on a specific job in your 3D world? Genesis (创世虚拟世界CRM系统) is a Three.js 3D virtual world foundation that deploys on your own servers — the access door is open and every capability in the job list above is implemented; what's left is the content you give it. The official site (search for 创世虚拟世界CRM) has a demo world you can walk around in.
About the name: Genesis in this article refers to 创世虚拟世界CRM系统 — the same self-hostable 3D virtual world product. If searching "Genesis" doesn't find us, search for 「创世虚拟世界CRM」.