AR and VR Development: The Complete Spatial Computing Guide

Virtual reality replaces the user's visible surroundings with a rendered environment. Augmented reality adds digital content to a view of the physical world, whether that view comes through a phone, an optical display or headset cameras. Mixed reality usually describes experiences where digital objects appear spatially integrated with real surroundings and can respond to them, although vendors use the label differently. Extended reality, or XR, is the umbrella term covering these approaches. Spatial computing overlaps with XR but emphasises software organised around three-dimensional space, tracked movement and environmental understanding rather than a particular display technology or a single device category.

The practical AR vs VR distinction concerns what users must see and what the application can assume about their surroundings. A virtual training room can control almost every visible object, but it still needs safe movement boundaries. A maintenance overlay must cope with lighting, camera visibility, clutter and imperfect alignment with equipment. A passthrough headset can support both immersive virtual scenes and mixed reality within the same application. Choose the mode from the task, not the label: does the user need their actual workspace, a simulated environment or a controlled transition between them? That answer shapes tracking, interaction and safety requirements.

Unity vs Unreal is a question of workload, team skills and device support rather than a universal quality ranking. Unity uses C# for most application scripting and offers XR packages for interaction, device integration and mobile AR. Unreal combines Blueprints visual scripting with C++ and provides extensive rendering and profiling tools. Both can produce demanding immersive applications, but available graphics features do not guarantee acceptable headset performance. Confirm the engine version, render pipeline, required plugins and target platform together. OpenXR can reduce integration differences across supported native runtimes, but platform services and specialised capabilities may still require vendor extensions or SDKs.

WebXR exposes supported immersive device capabilities to browser applications, commonly alongside rendering libraries such as Three.js or Babylon.js. It suits experiences where opening a URL offers a useful alternative to installing an application. However, browser and device support varies, especially for AR features, hand tracking and environmental understanding. Use capability detection rather than assuming that every browser can enter an immersive session. HTTPS and user permission or activation requirements also affect implementation. Native engines generally offer deeper platform integration; WebXR offers web distribution and familiar web tooling. Prototype the most demanding required feature on the actual target devices before committing.

Start with a narrow vertical slice: one environment, one meaningful interaction and one supported device configuration. Specify whether the experience is seated, standing, room-scale or handheld, and define which inputs it requires. In Unity VR development, a typical setup uses XR Plug-in Management, a compatible provider such as OpenXR where supported, and XR Interaction Toolkit components. An XR Origin groups the tracked camera and related objects so their poses can be interpreted within the scene. In Unreal, configure the appropriate XR plugins and build from a minimal template. Avoid adding multiple overlapping input or tracking systems without understanding their responsibilities.

Keep application logic separate from device-specific integrations. A task such as selecting a component should not depend directly on one controller button if hands, gaze or touch may later perform the same action. Define input actions and interaction events, then map supported devices onto them. Use consistent scene units, sensible object scales and explicit coordinate conversions when exchanging data with external systems. Establish version control and reproducible build settings early, including package and plugin versions. Run the prototype on hardware immediately: editor simulation can validate logic, but it cannot establish real tracking quality, physical comfort or sustainable performance on the device.

Tracking estimates a device's position and orientation over time. Three degrees of freedom cover rotation; six degrees of freedom add translation, allowing users to lean and move through space. Many modern devices combine cameras and inertial sensors to estimate motion, often using techniques related to simultaneous localisation and mapping. These estimates are not infallible. Featureless surfaces, reflections, poor lighting, rapid motion and camera obstruction can reduce reliability. Applications should react to tracking state changes, pause sensitive interactions where necessary and explain recovery steps clearly. Never treat a reported pose as a guarantee of precise alignment with a real-world object throughout a session.

AR anchors associate content with tracked locations, helping the platform maintain placement as its understanding of the environment changes. They do not automatically provide permanent, universally shared coordinates. Persistence and multi-user alignment require supported platform services or additional localisation systems, with their own permissions and constraints. On phones, ARKit and ARCore provide capabilities that frameworks such as Unity AR Foundation can expose, but feature availability differs by device and provider. Plane detection supports placement, while depth or scene geometry can support occlusion where available. Design a fallback when those features are missing, and let users reposition content if alignment becomes unreliable.

Spatial interaction should communicate what users can act on, how to act and whether the action succeeded. Direct grabbing works well within comfortable reach; ray-based selection helps with distant objects and interfaces. Hand tracking removes the need for controllers in supported experiences, but hands can leave the tracking volume or occlude one another. Controller input provides physical buttons and haptics, while gaze and dwell selection can support particular accessibility needs. Avoid making any one method mandatory without a task-based reason. Provide clear focus states, forgiving target sizes and opportunities to cancel actions, especially when selection changes consequential application state or data.

Comfort depends partly on matching visual motion to physical motion. Artificial acceleration, camera shake and unexpected viewpoint changes can cause discomfort, so avoid moving the tracked camera independently without a deliberate design reason. Teleportation and snap turning are common alternatives to continuous locomotion, but user preferences vary. Offer seated and standing options where the task permits, adjustable interface placement and alternatives to sustained arm elevation. Keep text readable at its intended viewing distance and provide captions or visual equivalents for essential audio. In phone AR, account for the effort of holding a device and keep instructions visible without obscuring hazards.

VR development requires consistent frame delivery because delayed or uneven updates undermine responsiveness and comfort. The time available per frame depends on the active refresh rate: at 90 Hz, a frame interval is approximately 11.1 milliseconds. That is an illustration, not a universal headset target. CPU work, GPU work, synchronisation and runtime overhead all affect whether the application meets its timing requirements. Profile on the target hardware and inspect frame-time distributions rather than relying only on average frames per second. Reprojection techniques can help runtimes handle missed frames, but they do not remove the need to optimise the underlying application.

Identify the bottleneck before changing assets or code. CPU costs may come from physics, scripts, animation or excessive rendering submissions; GPU costs may come from shading, transparency, shadows or pixel count. Useful techniques include level-of-detail meshes, occlusion culling, batching, simpler materials and baked lighting when appropriate. Stereo rendering approaches and foveated rendering can reduce particular costs on supported configurations, but their availability and trade-offs vary. Standalone headsets and phones also face thermal and battery constraints, so test sustained sessions rather than brief demonstrations. Passthrough composition, camera processing and environmental features may add costs that an empty virtual scene does not reveal.

A useful XR test plan separates functional correctness, spatial reliability, comfort and performance. Verify selection, object manipulation, saving and task completion, then deliberately interrupt normal conditions. Remove a controller, obscure the camera, deny permissions, suspend the application and resume it in a different position. Check recentering behaviour and tracking recovery without assuming that the original coordinate frame remains unchanged. For AR, test different surface textures, lighting conditions and room layouts. For VR, test supported play styles and boundary-related transitions. Record the device model, operating system, runtime, build version and reproduction steps so failures can be investigated rather than described only anecdotally.

Include representative users rather than relying exclusively on developers who already understand the controls. Observe whether people discover interactions, read labels, recover from mistakes and complete tasks without coaching. Test reach and interface placement across different heights, mobility needs and dominant hands. Track crashes, memory usage, frame timing and thermal behaviour throughout realistic sessions. Networked experiences need additional tests for latency, disconnections, ownership conflicts and mismatched spatial alignment. Automated tests are valuable for application logic and build validation, but they cannot fully assess headset comfort or environmental tracking. Combine automation, device testing and structured usability sessions before approving a release candidate.

Deployment depends on the platform: mobile app stores, headset distribution channels, managed enterprise deployment and websites impose different requirements. Native builds may need platform signing, approved permissions, supported SDK versions and store-specific checks. WebXR deployment requires secure hosting, compatible browser behaviour and a usable fallback when immersive sessions are unavailable. Explain camera, microphone, location or spatial-data access before requesting it, and collect only what the experience needs. Environmental maps and recordings can reveal sensitive information about private spaces. Document data retention, protect network traffic and verify applicable privacy obligations rather than assuming that all tracking information remains solely on the device.

Release a versioned build with known device support, clear onboarding and a way to report problems. Monitor failures and performance using proportionate telemetry, then retest after engine, plugin, browser or operating-system updates. Keep a regression scene for critical interactions and maintain a compatibility checklist for optional capabilities. For structured coverage of these foundations, the Erudex [AR, VR and Spatial Computing Development course](/courses/ar-vr-spatial-computing) provides a relevant learning route. Use [practice tests](/practice) to check understanding, while treating a course certificate as evidence of completion rather than a substitute for tested software. A maintainable XR application ultimately depends on documented decisions and repeated validation on hardware.