Augmented reality world tracking
Maps the surroundings and places digital objects on surfaces.
In one line
Can see through the camera and map your room while the app is open
Maps the surroundings and places digital objects on surfaces.
Builds a 3D map of a person's room and tracks where they look. Cloud and geospatial anchors can send visual data to the platform vendor.
What it can and can't do
What the user does: You tap Try it in your room or open a camera based AR feature
It can
- Find floors and tables through the camera and place digital objects on them
- Track where the phone is pointing
- Use your location to pin objects to real places, if you allow it
- Upload visual feature points to Google for cloud or Geospatial features (Android)
It cannot
- Start without the camera prompt
- Use your location for geo anchors unless you allow location
- Send the room map off an iPhone by default; it stays on the device
- Run on Android without Google Play Services for AR
How it differs between platforms: iPhone keeps the world map on the device; Android cloud and Geospatial features send data to Google.
On Android and iPhone
| Android | iPhone | |
|---|---|---|
| Status | Supported | Supported |
| First version | Android 7.0 (API 24) | iOS 11.0 |
| Permission or entitlement | CAMERA; ACCESS_FINE_LOCATION for Geospatial API; Google Play Services for AR | NSCameraUsageDescription; NSLocationWhenInUseUsageDescription for geo anchoring |
| Protection level | dangerous | prompt |
| How access is granted | prompt | prompt |
| What the user sees | Camera prompt; location prompt for Geospatial | Camera prompt |
| How long it lasts | Until revoked | Until revoked |
| How to take it back | Settings > Apps > Permissions | Settings > Privacy & Security > Camera |
| Store policy | Play: AR apps declare uses-feature and ARCore as required or optional | None listed |
| API | ARCore Session, Geospatial API, Cloud Anchors (data sent to Google) | ARKit ARSession, ARWorldTrackingConfiguration, ARGeoTrackingConfiguration (iOS 14), RealityKit |
| Notes | Cloud and Geospatial features upload feature points and need ARCore Cloud ID. | World mapping stays on device; iOS 26 adds spatial widgets and Shared World Anchors on visionOS. |
Data you get
What comes back: Camera pose and anchors
Shape: stream
How often: 30 to 60 Hz
| Field | Type | Unit | Range | Precision |
|---|---|---|---|---|
pose | object | 6 DoF | unbounded | about 1 to 2 cm drift |
Technical limits
- Camera dot and permission apply on both
The story
What it is
The phone maps the room around you and keeps digital objects stuck to floors, tables and walls.
For you
You point your phone at the floor and a virtual dragon or couch appears and stays put as you walk around it. It feels magical, but the camera is on and the phone is looking at your room.
For makers
AR needs a live map of the room, which unlocks games, shopping previews and measuring tools. Cloud anchors let two people see the same object. Makers also learn the layout of homes and stores if the data leaves the phone.
Where you see it
| Type | Example | Apps |
|---|---|---|
| games | You throw Poke Balls at a Pikachu standing in your living room and the creature seems to sit on your floor. | Pokemon GO |
| shopping | You place a sofa in your lounge to see if it fits and what colour looks right. | IKEA Place, Amazon |
| games | A multiplayer AR game lets two friends see the same castle on a table and play at once. | Minecraft Earth (retired) |
| maps | A walking app paints arrows on the street through the camera view. | Google Maps Live View |
| education | A biology class lets students walk around a giant virtual heart. | None named |
| creative | A face and world filter adds animated effects to a video. | Snapchat |
| utilities | A measuring app turns your phone into a tape measure. | Measure |
Risks
- Camera sees your home
- Shared maps can leak layouts
- Battery drain
Android and iPhone in short
- Android: Cloud and geospatial features upload camera feature points to Google and need an ARCore cloud ID.
- iPhone: World mapping stays on the device. iOS 26 adds spatial widgets and shared world anchors on visionOS.
What the platform says
- Apple: ARKit. Build augmented reality that tracks the real world and places virtual objects with precision.
- Google: ARCore. Build augmented reality experiences that blend digital content with the real world.
Good uses and abuse
Legitimate uses
- Furniture preview
- Navigation
- Games
Known abuse
- Scanning private rooms
- Sending visual maps to cloud
Scores
| Score | Value | Meaning |
|---|---|---|
| Sensitivity | 4 of 5 | How revealing the data is |
| Utility | 3 of 5 | How useful it is to honest apps |
| Friction | 2 of 5 | How much the user has to do |
| Fingerprint bits | 3 | Rough identifying entropy, when known |
History
- 2017-06, iPhone iOS 11: ARKit introduced
- 2018-03, Android ARCore 1.0 (Android 7): ARCore introduced
- 2022-05, Android ARCore 1.31: Geospatial API
Related capabilities
Related
- LiDAR and depth sensing. Can measure depth and room shape through the camera, on phones with the hardware
- Camera extensions and hardware camera controls. Can use phone camera modes such as night and HDR, after you allow the camera
Same idea on the web
- WebXR immersive VR and AR sessions. Can start a full VR or AR session after you allow it
- WebXR world sensing (hit test, anchors, depth, planes, camera). Can detect surfaces and depth in your room, and even see camera frames
Sources
How well checked: Based on official documentation.