Electronics Guide

Articles in This Category

Fundamental Technologies

AR and MR systems depend on several sophisticated technologies working in concert. Display systems must render images that appear to exist in real space, which calls for optics that can present content at a comfortable focal distance and align it precisely with each eye. Tracking systems must determine the device's position and orientation in six degrees of freedom, with motion-to-photon latency low enough that virtual objects stay locked to the world rather than drifting or swimming as the head moves.

Environmental understanding lets virtual objects interact convincingly with the physical world, which requires real-time reconstruction of surfaces, detection of objects, and interpretation of scene semantics so that content can rest on a table or hide behind a wall. Input systems capture the user's intent through gaze, gestures, voice, and physical controllers, translating natural human behavior into commands. All of these capabilities must run continuously within the strict power and thermal limits of a head-worn device.

Optical See-Through and Video Passthrough

Two architectures dominate head-worn AR and MR, and the choice between them shapes nearly every other design decision. In an optical see-through device, the user looks at the real world directly through a transparent combiner, and the display adds light on top of it. Microsoft's HoloLens 2 and Magic Leap 2 take this approach. In a video passthrough device, the real world is opaque; outward-facing cameras capture it, the system processes the images, and opaque internal displays present the composited result. Apple Vision Pro and Meta Quest 3 work this way.

Optical see-through preserves the full resolution, dynamic range, and instantaneous latency of natural vision, and it keeps the user visually connected to the room. Its weaknesses follow from the physics of additive light: virtual content can only add brightness, never subtract it, so black renders as transparent and bright rooms wash out the imagery. Occlusion, in which a virtual object hides the real surface behind it, requires a per-pixel dimming layer that few devices provide. Magic Leap 2 introduced segmented dynamic dimming precisely to address this limitation.

Video passthrough sidesteps those problems. Because the entire view is synthetic, the system controls occlusion, contrast, and color completely, and it can fade smoothly between full passthrough and fully virtual scenes. The cost is that every photon the user sees has passed through a camera, an image signal processor, a compositor, and a display, so passthrough imagery inherits the sensors' noise, dynamic range, and geometric distortion. It also introduces a latency budget that does not exist in optical see-through, since even the real world now arrives late. Vendors treat that budget as a headline specification: Apple states that the R1 coprocessor in Vision Pro streams images from its cameras to the displays within twelve milliseconds.

Display Technologies

Near-eye displays combine an image source with optics that place a virtual image at a comfortable focal distance and steer it into the eye. Image sources include micro-OLED and microLED panels, liquid crystal on silicon, conventional LCD, and MEMS laser-scanning engines. The combiner optics vary more widely: diffractive or reflective waveguides, birdbath beam splitters, freeform prisms, and folded "pancake" catadioptric stacks each occupy a different point in the trade space of field of view, weight, thickness, efficiency, and cost.

Waveguide combiners are what make slim optical see-through glasses possible. Light from a small projector couples into a thin glass or plastic substrate, travels by total internal reflection, and couples out toward the eye, with intermediate gratings replicating the exit pupil so that the image stays visible as the eye moves. The approach is compact and largely transparent, but only a small fraction of the projected light reaches the pupil, so the image engine must be extremely bright to compete with daylight. Diffractive designs can also produce color nonuniformity and stray "rainbow" artifacts from ambient light sources.

Field of view remains the most visible constraint on see-through devices. HoloLens 2 pairs laser-scanning engines with diffractive waveguides to deliver roughly 52 degrees measured diagonally, about 43 degrees horizontally by 29 degrees vertically. Magic Leap 2 widened this to approximately 70 degrees diagonally. Both figures fall well short of natural binocular vision, which spans well over 100 degrees horizontally, and enlarging the field of view generally demands a larger image source, a larger eyebox, and more light, all of which add weight and power.

Passthrough devices face a different limit, namely angular resolution. Apple Vision Pro uses two micro-OLED panels with roughly 23 million pixels in total and a pixel pitch near 7.5 micrometers, supporting refresh rates of 90, 96, and 100 hertz; the 96-hertz mode exists so that 24-frame-per-second film content divides evenly. Pancake optics, used in Vision Pro and Quest 3, fold the light path through a polarization-based reflection sequence to shorten the headset considerably, at the cost of discarding most of the light and therefore demanding brighter, more power-hungry panels.

A difficulty common to nearly all current designs is the vergence-accommodation conflict. Conventional stereoscopic displays fix the focus distance even as the eyes converge on objects at different apparent depths, and the mismatch between these normally coupled cues can cause eye strain and fatigue. Varifocal optics, which mechanically or electrically shift the focal plane to match where the user is looking, light-field and multifocal displays, holographic elements, and retinal projection all aim to restore the correct focal cue. Each adds demands on the processing pipeline, which must drive the optics in step with gaze while holding the precise, low-latency timing that comfortable viewing requires.

Sensor Systems

AR and MR devices carry dense, heterogeneous sensor arrays. Apple Vision Pro, as one published example, integrates twelve cameras, five other sensors, and six microphones. Outward-facing cameras used for tracking are typically monochrome and global-shutter, because a rolling shutter smears image features during rapid head rotation and corrupts the pose estimate. Depth is recovered from stereo disparity, structured light, or time-of-flight sensors, each with characteristic failure modes: stereo struggles on featureless walls, structured light degrades in sunlight, and time-of-flight suffers from multipath reflections in corners.

Inertial measurement units containing accelerometers and gyroscopes sample head motion on the order of a kilohertz, far faster than any camera, and supply the short-horizon prediction that keeps imagery stable between visual updates. Their drift is corrected by the slower but absolute visual measurements, which is the essence of visual-inertial odometry. Eye-tracking subsystems illuminate the eye with infrared emitters and locate the pupil center against corneal reflections, yielding gaze for foveated rendering, interface selection, and automatic interpupillary-distance adjustment. Hand tracking runs neural networks over the same camera streams to recover joint positions, removing the need for controllers at the cost of tracking loss when the hands leave the camera frustum.

Fusing these modalities is what produces robust behavior in real rooms. When a camera is briefly occluded, saturated by a window, or confronted with a blank surface, inertial and depth data carry the pose until visual features return. Careful factory and runtime calibration matters as much as the sensors themselves, since a fraction of a degree of misalignment between a camera and the display translates directly into visible registration error.

Processing Challenges

The computational demands of AR and MR strain even advanced mobile processors. Rendering a separate image for each eye at 90 frames per second or more is a heavy graphics workload, yet a standalone headset must deliver it within a total system power budget in the single-digit to low-double-digit watts, rather than the hundreds of watts available to a desktop gaming system. The two are not directly comparable in raw rendering power; headsets close the gap through aggressive efficiency techniques rather than brute force. Simultaneous localization and mapping (SLAM) algorithms run alongside rendering, processing several sensor streams at once to build and maintain a model of the environment while tracking device motion.

Machine-learning workloads for object recognition, hand tracking, and scene understanding compete for the same resources as graphics and tracking. Because these tasks are time-critical, and even small increases in latency can break immersion or cause discomfort, the system must orchestrate its resources carefully and deliver predictable, low-jitter execution. Average throughput matters less than worst-case timing.

The industry's answer has been purpose-built silicon on leading-edge process nodes. Meta Quest 3 uses Qualcomm's Snapdragon XR2 Gen 2, built on a 4-nanometer process and pairing CPU and GPU cores with dedicated computer-vision and video engines. Apple Vision Pro splits the problem across two chips, running applications and graphics on an M2 while a companion R1 processor is dedicated to sensor ingest and passthrough compositing, which keeps the time-critical path insulated from general-purpose software. Devices that cannot fit the workload on the head move it elsewhere: Magic Leap 2 tethers to a belt-worn compute pack, and PC- and cloud-connected headsets stream compressed frames over Wi-Fi, which trades thermal headroom for encode, transmit, and decode latency.

Latency, Comfort, and Efficiency

Comfort sets some of the strictest engineering targets in consumer electronics. For immersive headsets, the widely cited goal is a motion-to-photon latency below about twenty milliseconds, paired with a refresh rate of at least 90 hertz; research and industry guidance often puts the range for convincing registration tighter still, around seven to fifteen milliseconds. Optical see-through AR is the most demanding case, because lag between head motion and the displayed image shows up directly as virtual content sliding against a real world that never lags.

No practical rendering pipeline meets those budgets by brute force, so systems predict and correct. Head pose is extrapolated forward by the expected display time, which works well over short horizons and degrades quickly over longer ones. Late-stage reprojection, also called timewarp, then takes the already-rendered frame and warps it using the freshest available pose immediately before scanout, decoupling perceived tracking latency from the application's frame rate. Related techniques handle the animation of moving objects and fill in frames when an application misses its deadline.

Efficiency is the other constant pressure. Foveated rendering exploits the fact that human acuity falls off sharply outside the fovea: the system renders the small region the eye is fixating at full shading rate and progressively reduces detail toward the periphery, usually through variable-rate shading. Fixed foveation, which assumes the user looks near the display center, requires no eye tracker but must stay conservative. Eye-tracked foveation can be far more aggressive, and vendors have reported GPU savings in the tens of percent on shipping hardware.

Thermal design closes the loop. Because a headset sits against the face, the constraint is not only junction temperature but skin-contact temperature, and nearly all wearable designs rely on passive conduction and spreading rather than noisy fans. Weight and its distribution matter as much as mass alone, since a load carried forward of the head's center of rotation produces neck strain far out of proportion to its grams. Moving silicon to more efficient process nodes helps on every axis at once, raising performance per watt while lowering the heat that must be dissipated next to the user's face.

Software Platforms and Interoperability

Hardware diversity would fragment the field without common software interfaces. OpenXR, maintained by the Khronos Group, provides a cross-vendor API for device tracking, input, and frame submission, allowing an application to target many runtimes without rewriting its device layer. WebXR extends comparable capability to the browser, so immersive content can be delivered by URL rather than through an app store. On handheld devices, Apple's ARKit and Google's ARCore supply plane detection, motion tracking, and lighting estimation to phones and tablets.

Handheld AR deserves emphasis, because it is where most people actually encounter the technology. A modern smartphone already carries the cameras, inertial sensors, and neural accelerators that AR requires, and its screen serves as a magic-window view into the augmented scene. The experience is less immersive than a headset, but the installed base is measured in billions of devices rather than millions, and it has driven much of the practical maturity in tracking and scene reconstruction that head-worn devices now inherit.

Applications and Impact

Head-mounted AR is older than the consumer industry that surrounds it. Military aviation has used helmet-mounted displays and head-up displays to overlay flight and targeting symbology on the pilot's view for decades, and those systems established much of the engineering vocabulary of registration, latency, and boresight calibration still in use today.

Industrial deployment is the clearest current success. Remote expert assistance lets a specialist see a technician's view and annotate it in place, guided assembly overlays the next step directly on the workpiece, and warehouse picking directs workers to bins while keeping their hands free. These uses share a profile that suits present hardware well: sessions are short, the environment is controlled and indoors, and the value comes from information placement rather than photorealism.

In medicine, mixed reality supports image-guided surgery, three-dimensional review of computed tomography and magnetic resonance data, anatomy education, and rehabilitation and exposure-therapy protocols. Clinical use raises the bar considerably, since any system that guides an intervention becomes a regulated medical device subject to validation, and registration accuracy becomes a patient-safety parameter rather than a comfort parameter. Design, architecture, and construction use the same registration capability to place full-scale models on site and compare as-built structures against design intent.

Consumer adoption remains more concentrated in handheld AR, gaming, media viewing, and navigation. A separate and rapidly growing category is display-light smart glasses, which forgo world-locked three-dimensional graphics in favor of a small heads-up panel, cameras, and audio in a package close to ordinary eyewear weight. These devices sidestep the hardest optical and computational problems entirely, and their commercial traction suggests that comfort and social acceptability may govern adoption more strongly than capability.

Outlook

Progress in this field is best measured by which trade-offs are loosening rather than by any single specification. Field of view, brightness, weight, battery life, and cost still pull against one another, and every shipping product resolves that tension differently. The difficulty is not merely theoretical: Microsoft ended production of HoloLens 2 in 2024 without a direct successor, a reminder that technical capability alone does not sustain a hardware platform.

The near-term trajectory points in two directions at once. Full mixed-reality headsets continue to gain resolution, passthrough fidelity, and processing efficiency while remaining relatively heavy and expensive. Lightweight glasses move in the opposite direction, shedding capability to reach a form factor people will wear all day. Whether these converge into a single general-purpose spatial computer depends on sustained progress in waveguide efficiency, low-power silicon, battery energy density, and displays that resolve the vergence-accommodation conflict. In each case, the human factors of weight, comfort, and latency remain as decisive as raw performance.

Related Topics