How We Taught an AI Character to Feel Human | AR Spatially

How We Taught an AI Character to Feel Human: Gestures, Lip Sync, and the Limits of Realtime AI

By AR Spatially14 min read
How We Taught an AI Character to Feel Human: Gestures, Lip Sync, and the Limits of Realtime AI

How We Taught an AI Character to Feel Human: Gestures, Lip Sync, and the Limits of Realtime AI

We're building AR Spatially — a spatial augmented reality platform. A month and a half ago, we decided we didn't want a "voice interface" living inside it. We wanted a character. Someone you talk to the way you'd talk to a person — or at least a good NPC in a game. You point your camera at a printed brochure, a 40-centimeter-tall figure appears on it, and you start a conversation.

On paper, this sounds like "bolt on the Realtime API." In practice, none of the real difficulty was in making the assistant talk — that took a day. The difficulty was making it look alive: getting the face to move, gestures to land on meaning, keeping it from turning into a talking doll.

Here's a full breakdown of what we tried — including the parts we had to throw out entirely.

The Architecture, in Two Paragraphs

Voice runs entirely on OpenAI's Realtime API: one model handles recognition, response generation, and speech synthesis together. Transport is LiveKit — the client publishes the microphone into a room, and a server-side Python worker joins the same room and holds the connection to OpenAI.

Client (Android) ──POST /voice/token──► gateway (signs a LiveKit token, 5-min TTL)
   │
   └─ WebRTC ──► LiveKit ◄── Python worker ──WS──► OpenAI Realtime
                    │
   ◄── assistant's audio track
   ◄── word-by-word transcript, synced to audio
   ◄── data packets: gestures and emotions

The client knows nothing about OpenAI. It just publishes the mic, listens to audio, and animates the face and body. The API key, model choice, prompt, and rate limits all live server-side.

The rest of this post is about the two things that turned out to be real engineering problems: gestures and lip sync.

Problem One: Gestures — Getting the Model to Control a Body

We wanted the assistant to gesture: wave hello, look thoughtful when considering something, react physically to the conversation.

The Realtime API has nothing built for this. What it has is function calling — designed for a completely different purpose: the model fetches data, then continues its answer knowing the result. We repurposed that mechanism into "trigger this animation."

There's exactly one tool:

python
@function_tool
async def play_gesture(self, context: RunContext, name: GestureName) -> str:
    ...

The parameter type is a Python enum, so the model physically cannot request a gesture that doesn't exist. The tool's description, from the model's point of view, is the method's docstring — which is also where all the behavioral logic lives: when each gesture is appropriate, and a hard rule against narrating the gesture in words ("never say 'I'm waving'").

The worker then publishes a data packet into the room, and the client catches it and plays the clip:

json
{"name": "hello", "v": 1}

This works. But it comes with two limitations we had to accept rather than solve.

Limitation 1: Tool Calls and Speech Compete for the Same Turn

In the Realtime API, a response that contains a function_call does not contain audio. Speech arrives on the next turn, after the model has received the tool's result. Every tool call is a structural pause in the conversation.

We hit this in its most painful form. We had a mandatory tool the model was supposed to call on every single reply — to pick a conversational tone. The result: the assistant would smile and say absolutely nothing.

We fixed it in three layers, and at each layer we were sure we'd found the one cause.

Layer 1: the mandatory tool itself. We removed the per-turn tool entirely. Picking the "talking" animation got handed off to the client — it already knows whether the assistant is speaking or not. The tool now exists only for meaningful gestures, called at the model's discretion.

Layer 2: what the tool returns. The tool used to reply to the model with a bare string: "ok". It turns out a bare acknowledgment conditions the next turn poorly — the model could treat it as a natural place to end the reply. Now the tool returns an instruction instead of a confirmation:

Gesture is now playing on screen. Continue and speak your reply to the user now, in their language; never mention or narrate the gesture.

This is locked in with a regression test that explicitly checks the return value is not "ok" and contains an instruction to keep speaking. It looks like an odd thing to test — but it's guarding a week of debugging.

Layer 3: the prompt. The character's persona is described in detail — 516 lines, 17 sections. And buried in that description were lines like "might trail off mid-sentence." For a piece of creative writing, that's a personality trait. For a realtime model, it's a literal instruction to generate a turn with no audio.

We had to add a section to the prompt with absolute priority: "you always answer out loud" — ranked above every rule about character and mystery.

The takeaway: in voice agents, the prompt isn't just about meaning — it's also mechanics. A literary metaphor turns into real silence in production.

Limitation 2: One Gesture Per Response

The Realtime API can't call tools asynchronously, in parallel with speech. The model can trigger one animation per reply, and that's it. If you want the character to gesture throughout a sentence, reacting to its own words as it says them — there's no mechanism for that.

This is a hard constraint of the API, and for now, we live with it.

Problem Two: Lip Sync — Six Approaches, Two Full Teardowns

This is the longest and most instructive part. We wanted facial expression to actually reflect speech. We only landed on something that worked on the sixth attempt.

Approach 1: Random Mouth Shapes

Just jitter the mouth shapes randomly while audio is playing.

It looks exactly like it sounds: the character appears to be chewing. Zero correlation to what it's actually saying. Discarded immediately.

Approach 2: Volume → Mouth Openness

Compute the RMS (root mean square) of the incoming audio frame, map it to how open the mouth is. A classic trick — gives you the recognizable "talking puppet" effect.

First problem: human speech is quiet. Real-world RMS on-device measured 0.03–0.15. With a linear mapping, that gave us 2–3 degrees of mouth opening — the character looked like it was talking through clenched teeth. We fixed that with a cube-root curve instead of linear, plus a 6x gain.

But a second, unfixable problem remained: volume doesn't correlate with mouth shape. "Ah" and "M" at the same volume look identical, even though they're fundamentally different shapes. There's no way to get this to look convincing.

Approaches 3–7: Procedural Lip Sync via Facial Bones

We decided to drive the face bones directly — jaw and mouth corners. One calendar day, five iterations, each one rolling back the previous idea:

  • Manual Euler angles for jaw and mouth corners. Failed: eyeballing a 3D bone rotation by hand is basically impossible, and there are a lot of bones.

  • Copying a pose from a baked animation clip. We took a frame from an existing animation as a reference. Better, but there's one clip and many phonemes.

  • Checking axis alignment against the rig. This is where we found a real bug: the code was mirroring the left and right mouth corners — intuitive, but wrong. Both bones' local axes point the same direction, so they need to rotate the same way. A classic trap: model symmetry is not the same thing as local coordinate symmetry.

  • Speech as a gate, not a multiplier. We were interpolating angles by volume — which meant quiet passages got half the amplitude, and the mouth started to "mumble." That's when we recognized the actual split: volume should decide whether it's speaking, and something else entirely should decide how open the mouth is.

  • A phase cursor driven by a timeline, instead of recomputing frame by frame.

The result of this entire branch: we tore it out completely. Procedural bone-based lip sync means manually tuning dozens of constants, and it still doesn't look natural — because underneath it all, you're still driven by volume, not by content.

The real lesson here is organizational, not technical: we spent a full day on five iterations inside the wrong architecture. The right move was to step out of it after iteration two, not to keep optimizing a dead end.

Approach 8 (The One That Worked): Text → Visemes → Morph Targets

Here's what actually worked.

LiveKit can deliver a synchronized transcript — the spoken text arrives on the client word by word, in sync with the audio frame. In other words, alongside the voice, we also get exactly what's being said, right now. We pinned this explicitly on the server, even though it's the default behavior:

python
room_output_options=RoomOutputOptions(
    transcription_enabled=True,
    sync_transcription=True,
)

The comment next to it explains why we made it explicit: this became a hard contract for the client's lip sync, and a silent SDK default change would have broken the character's face.

On the client, there's a small G2P-lite (grapheme-to-phoneme) mapper: a character-by-character mapping from text to visemes — mouth shapes. It supports Cyrillic, Latin, and Arabic script, including diacritics. Viseme duration depends on the sound class:

  • Sound Class Duration
  • Vowel 160 ms
  • Fricative consonant 110 ms
  • Plosive consonant 70 ms
  • Consonant cluster 50 ms

The core architectural idea is two independent inputs:

Which mouth shape — comes from the transcript. How open the mouth is — comes from the audio's RMS.

Text drives the shape. Sound drives the amplitude. That separation was exactly what every previous approach was missing. If the text doesn't arrive in time for some reason (network lag), there's a fallback to pseudo-random jitter based on volume, at a 90ms step — a graceful degradation back to Approach 2.

This already looks a lot like real speech. But to be honest: it's still pseudo-lip-sync. We don't know the real phoneme boundaries — we know letters, and we estimate durations with heuristics.

Two Traps Inside the Working Approach

Letters arrive faster than they can be displayed. The transcript streams in at roughly 13 characters per second, while the viseme queue chews through about 11. The gap got dropped essentially at random, and the mouth would twitch like a machine gun. Fix: a run of consonants with no shape of its own collapses into a single micro-shape.

The mouth is closed at the start of a word. The silence branch unconditionally flushed the viseme queue, and the transcript for the first word arrives a few frames before volume crosses the silence threshold. The first word's visemes were simply getting eaten. We added a 300ms grace window — enough time for lip sync to start before the sound has "officially" started.

The Face, Fully Assembled

In the end, the character's face is six independent channels, each owning its own set of morph targets:

  • Channel Parameters
  • Breathing ~4s period (≈15 breaths/min), weight 0.55
  • Blinking every 2.5–6s, phases 70/50/120ms
  • Idle smile every 8–20s, weight 0.4
  • Emotion (from the model) 200/2000/600ms
  • Lip sync queue of 24 visemes (≈2–3s of speech)
  • Speech emphasis brows and cheeks on stressed syllables, 2s cooldown

Composition is just plain assignment — no priority system needed, because each channel owns a non-overlapping set of morphs by design. Every frame, weights reset to zero and get rebuilt from scratch — the reset itself is the release mechanism.

One exception to "channels are independent": speech owns the face. The moment the assistant starts talking, emotion gets cut instantly — otherwise the smile and the articulation fight over the same muscles.

Problem Three, Unsolved: The Body

This is where we're still losing, and it's the biggest open question.

While the assistant is speaking, the client picks one of nine talking animations at random. The model isn't involved in this at all — it has no idea what its body is doing.

The result is what you'd expect: sometimes the character gestures out of sync with the content. It's saying something serious, and its body throws a playful gesture.

Technically, the clips vary in amplitude: three move only the arms, six move the whole body, up to stepping in place. There's a hard AR constraint here too: horizontal pelvis drift can't pull the character off its anchor point, so root motion drift is strictly zero. And since the clips don't loop perfectly, an 0.8-second crossfade is mandatory on every transition, including the loop seam itself.

The right fix is for the AI to drive the body together with speech, the way it already drives one gesture. But it runs straight into the same limitation from Part One: there's no async tool calling, and one call per reply isn't enough to produce body language.

  • The Numbers
  • Parameter Value
  • Character model 20 morph targets, 78 bones, 9 talking animations
  • Character height (brochure / preview) 40 cm / 1.7 m
  • Gestures available to the model 3
  • Character prompt 516 lines, 17 sections
  • Vowel / fricative / plosive / cluster 160 / 110 / 70 / 50 ms
  • Viseme queue 24 (≈2–3s of speech)
  • Silence threshold (RMS) / gain 0.06 / ×6
  • Real on-device speech RMS 0.03–0.15
  • Jitter fallback (no text) 90 ms
  • Grace window at word start 300 ms
  • Body animation crossfade 0.8 s
  • Unit tests on this feature 214 across 22 files, pure JVM
  • Localization 22 keys × 6 languages

One note on testing: time and the random number generator are both injected, and the 3D model itself gets parsed directly inside the test — so a test verifying morph names match the actual model file catches code/model drift without ever running on a device. For a feature where half the bugs are "the code's mental model of the character doesn't match reality," a test like that is worth ten ordinary ones.

What's Still Unsolved

We're deliberately not ending this on "and it all worked out."

Body control. Nine animations picked at random, landing in context by luck. We want the AI to drive the body together with speech. That's blocked by the lack of async tool calls in the Realtime API.

Real lip sync. What we have is a heuristic layered on top of letters. What we want is actual phoneme boundaries and emotionally colored articulation — the same line sounding and looking different when the character is happy versus when it's uncertain.

The model itself. We have AI-generated video clips where the character looks exactly the way we want it to. The 3D model was built to match those clips — it came out close, but not quite there. Catching up to a generated reference with a hand-built 3D model turned out to be its own, and possibly the most underrated, problem in this entire project.

What We'd Do Differently

Don't keep iterating inside an architecture that's already given you two bad results. Five attempts at bone-based lip sync in one day is five ways of not leaving a dead end. The working solution only showed up after we tore the whole branch out.

Separate your signal sources. Lip sync started working the moment mouth shape and amplitude came from two different sources. Before that, we spent five iterations trying to squeeze the content of speech out of its volume — which was never going to work.

Remember that a prompt is mechanics, not just personality. "Might trail off mid-sentence," written as a character trait, turns into real silence in production.

Treat API limitations as architectural, not as bugs to route around. Tool calls and speech competing for the same turn isn't something you hack past — your interaction design has to be built around it from the start.

This feature is currently in active development behind feature flags — a few more iterations before release. We'll follow up with a look at how it behaves in motion once it's further along.