How Augmented Reality Works: The Technology Behind AR

Quick Recap: What Makes Something "AR"?

Augmented reality overlays digital content onto the physical world in real time, through a device a person is actually holding or wearing in that location — not a pre-rendered video, but a live composite that reacts to where the viewer is and how they move. We covered the basic definition and the boundaries of the term in What Is Augmented Reality? A Simple Guide for 2026. This piece goes one level deeper: what's actually happening inside the device to make that live overlay possible.

The Core Components of AR Technology

An AR experience isn't one piece of technology — it's several systems working together, each handling a different part of the problem.

Cameras & Sensors

Before a device can add anything to the world, it has to perceive the world. The RGB camera captures the visual feed a viewer sees on screen. Alongside it, most AR-capable devices carry a set of additional sensors: an accelerometer and gyroscope (grouped as the IMU, or inertial measurement unit) that track how the device is moving and tilting, and, on many newer phones and headsets, a depth sensor or LiDAR that measures the actual distance to surfaces in the room.

The camera alone can tell you what a scene looks like. The sensors tell the device where it is and how far away things are — information the camera image doesn't carry by itself.

Tracking & Mapping (SLAM)

This is the part that makes AR feel stable rather than jittery. SLAM stands for Simultaneous Localization and Mapping: the device builds a map of its surroundings and tracks its own position within that map, at the same time, continuously, as it moves.

It does this by picking out visual features in the camera feed — corners, edges, textured patches — and following how they shift from frame to frame. Combined with the motion data from the IMU, that lets the device calculate its position and orientation dozens of times per second. This is why a well-tracked AR object appears to stay locked to a real surface even as the viewer walks around it, rather than sliding or drifting.

Image Recognition

Image recognition is a related but distinct capability: instead of mapping open space, it identifies a specific, known image — a product package, a poster, a page in a brochure — and uses that as a trigger or reference point for AR content. We've written about the mechanics of this separately in How Does AR Image Recognition Work? A Simple Explanation, so we won't repeat it here. The short version for this piece: recognition answers "what am I looking at," while SLAM answers "where am I in space." AR experiences often use both.

Rendering

Once the device knows where it is and what it's looking at, it has to place a digital object into the scene convincingly. That's rendering, and three things determine whether it looks believable rather than pasted on.

Lighting estimation reads the real-world light in the scene and applies matching shading to the digital object, so it doesn't look flat or lit from the wrong direction. Occlusion handles what happens when a real object passes in front of the digital one — a hand reaching in front of a virtual character should hide part of it, not clip through it. Scale and perspective keep the object sized and angled correctly as the viewer's distance and angle change.

Get these three wrong, and even accurate tracking won't stop an AR object from looking like a sticker floating over the video.

Display

Finally, the composited result has to reach the viewer's eyes. Most AR today is viewed on a phone or tablet screen, where the camera feed and the digital overlay are combined into a single video image. Headsets and glasses split into two rendering approaches: video passthrough, which works the same way as a phone (camera feed plus overlay, shown on an internal screen), and optical see-through, where the wearer looks directly through a lens and the digital content is projected onto it. The underlying tracking, mapping, and rendering pipeline is largely the same either way — the display is just the last step.

Marker-Based vs Markerless AR

In practice, most AR experiences fall into one of two approaches, and the difference matters for what's technically and logistically possible.

Marker-based AR anchors content to a specific, pre-defined image — a logo, a QR-style pattern, a printed page. The system doesn't need to understand the whole room; it just needs to recognize that one image and knows exactly where to place content relative to it. This makes it reliable and fast to set up, but the content only appears when that specific marker is in view.

Markerless AR, built on SLAM, doesn't need a predefined image at all. It reads the general environment — flat surfaces, spatial features — and lets content be placed anywhere the system can map, not just on a known trigger. This is more flexible but generally needs more processing and a bit more movement from the user before tracking stabilizes.

Neither approach is universally "better" — a printed brochure with a fixed layout is a natural fit for marker-based tracking, while a virtual furniture placement app needs markerless mapping of an arbitrary living room.

Cloud Anchors & Persistent AR

One limitation of basic AR tracking is that it resets every session — a digital object placed in a room during one visit doesn't automatically remember where it was the next time someone opens the app, and it's invisible to a second person's device at the same time.

Cloud anchors solve this by saving the spatial map data to a server instead of keeping it only on one device. When another device — or the same device later — scans the same physical space, it can request that saved map, localize itself against it, and see the digital content in the same real-world position as before. This is what makes shared, multi-user AR possible (two people looking at the same virtual object from different angles, at the same time) and what makes persistent AR possible (content that "stays" at a location across separate visits, days or weeks apart).

Putting It Together: From Camera Feed to AR Experience

Stripped down, a live AR session runs through roughly this sequence, repeated many times per second: the camera captures a frame and the sensors report motion and depth; SLAM uses that data to update the device's map and its position within it; if the experience uses image recognition, the system checks the frame against known reference images to see if one is in view; the renderer places the digital content into the scene, adjusting its lighting, occlusion, and scale to match; and the display shows the combined result to the viewer.

As an illustration of how these pieces come together in a real project, AR Spatially's own Quantum Choice Chronicles uses image tracking on a printed brochure as the recognition layer — scanning specific pages triggers the AR content built for them.

Why This Matters If You're Evaluating AR

None of this requires writing code to be useful. Knowing the difference between marker-based and markerless, or between a one-off tracked object and a persistent cloud-anchored one, changes the questions worth asking a vendor or an internal team: What kind of tracking does this experience actually need? Does content need to persist across visits or across users, or is a single-session experience enough? Is the plan technically a form of AR at all, or a video edited to look like one?

That vocabulary is usually enough to tell a real technical proposal apart from a good-looking pitch deck.

See It Applied to a Real Project

Reading about the pipeline is one thing — seeing how tracking, recognition, and persistence come together in an actual campaign or exhibit is another. If it's useful to walk through how this would apply to a specific project, AR Spatially is happy to talk it through.