The sofa was floating. Only by about three inches, but Meera noticed it the second she opened her company's new AR feature at home. She runs a small online furniture brand, and her developers had spent two months on the "see it in your room" button. In the office, on a patterned rug under bright lights, the virtual sofa sat exactly where it should. In her living room at nine at night, with one warm lamp and a plain grey floor, it hovered, drifted, and once slid straight through the coffee table.
Nothing in the code was broken. The team had simply tested in one room, under one kind of light, on one phone, and real homes are messier than that.
That gap between a clean demo and a real living room is where most AR projects succeed or quietly fail. This guide covers ARKit development from the ground up, but it spends most of its time on the parts that feature lists skip: what happens when the phone can't see enough, when its sensors disagree, when it has a fraction of a second to decide, and when it gets hot. Founders deciding whether to fund an AR feature, developers about to open their first ARKit tutorial, and product or content teams who need to explain this to others should all come away knowing what to expect and what to ask.
What ARKit is, in plain words
ARKit is Apple's toolkit for building augmented reality into iPhone and iPad apps. Augmented reality, or AR, means placing digital objects on top of a live camera view so they look like they're part of the room. A lamp on your desk that isn't there, or an arrow on the airport floor pointing to your gate.
Apple released ARKit in 2017 alongside iOS 11. It made AR part of the operating system, and that's the main reason iOS augmented reality spread so fast: hundreds of millions of existing phones gained the ability through a software update.
The core trick is called world tracking. Your phone watches the camera feed and picks out small, distinctive details, such as the corner of a picture frame, the grain on a wooden table, or a crack in a floor tile. These are called feature points. At the same time, the motion sensors inside the phone report how it's moving. The accelerometer senses movement in a direction, and the gyroscope senses rotation. ARKit blends the two streams together. The technical name is visual-inertial odometry, which roughly means "working out where you are by combining what you see with how you feel yourself move."
From that blend, ARKit builds a rough map of the space and keeps updating where the phone sits inside it. Everything else depends on that estimate.
A few other names come up constantly, so here they are once:
• RealityKit is Apple's framework for drawing 3D content, running physics and playing spatial sound. ARKit figures out the world, and RealityKit draws things into it.
• USDZ is the file format Apple uses for 3D models. If your company already has 3D files of its products, someone will probably need to convert them.
• AR Quick Look is a viewer built into iOS that opens USDZ files straight from Safari, Messages or Mail, with no app install needed.
That last one matters more than people expect.
What the market numbers tell you, and what they don't
If you pitch AR inside a company, someone will ask about market size. The honest answer is that research firms disagree by a wide margin.
That's more than a threefold spread for the same year, mostly down to definitions. Some firms count headsets and smart glasses, others count only software. Grand View Research says hardware made up about 59% of its 2025 total, so much of that headline has nothing to do with phone apps.
Two smaller numbers are more useful for planning. Mordor Intelligence reports that gaming and entertainment held 36.26% of AR revenue in 2025, while remote assistance and maintenance was the largest single application at 29.09%. So the money is split between entertainment and very practical work: guiding technicians, fixing machines, training staff.
Use these reports to show direction, but I wouldn't bet payroll on any one of them. A better test is whether your customers already hold a phone up to something physical, like a room, a product or a machine. If they do, there's a real case for iOS augmented reality in your product.
The toolbox: what ARKit gives you before you write much code
The table below lists the ARKit features most teams actually use, along with the hardware each one needs. That last column is where planning tends to go wrong.
LiDAR is the one people misjudge most. LiDAR is a small sensor that fires pulses of light and times how long they take to bounce back, which gives the phone a direct measurement of distance. It makes almost everything better, but it only ships on Pro models. A feature that looks wonderful on the Pro phone on your designer's desk may not exist on most of your customers' phones.
So a sensible rule for ARKit development is to treat LiDAR as a bonus unless you control the hardware. A company that buys iPad Pros for its field technicians can design around LiDAR. A consumer shopping app can't.
Most successful Apple AR apps are built on the plain features in the first three rows, and then add richer behaviour for devices that support it.
Where the data runs out
ARKit can only track what it can see, and it can only see what has texture and light. Almost every "AR is buggy" complaint traces back to this.
Blank and shiny surfaces
A floor painted one flat colour gives the camera nothing to hold on to. So does a white counter, a glass table, polished marble or a room lit by one dim lamp. Meera's grey floor is a textbook case. ARKit then reports tracking as "limited" with a reason such as insufficient features. The camera image still looks fine to the user, which is what makes it confusing.
LiDAR mostly closes this gap because it measures distance directly. On other devices, the app has to ask the user to point at an edge, a rug or a brighter spot.
The first few seconds
When a session starts, ARKit knows nothing about the room. It usually takes a few seconds of gentle movement to find the first surface, which starts small and rough. ARKit then grows it and refines its height, sometimes by a centimetre or two.
Let the user place an object too early and you lock in that rough estimate, which is a common cause of floating furniture. Show a preview, and hold the final placement until tracking settles.
Depth that fades with distance
Even LiDAR has limits. Apple rates it for distances of up to about five metres, and ARKit gives each depth reading a confidence level of low, medium or high. Confidence drops at object edges, on thin things like chair legs and cables, and on dark or reflective surfaces. Treat every reading as equally true and you get jagged edges and objects clipping into furniture.
Images printed at the wrong size
Image tracking needs you to tell ARKit the physical size of the picture it's looking for. If the marketing team prints the poster at a larger size than the one entered in the app, ARKit will believe the poster is closer than it really is. Content then appears at the wrong depth and scale. The missing detail sits in the print shop's order form, where no code review will catch it.
GPS in real streets
Location anchors pin content to a real place, like a shop front. They only work in cities where Apple has detailed map data, and they rely on GPS, which can wander by several metres between tall buildings. ARKit matches camera imagery against Apple's maps to correct this, but in unsupported areas the feature won't start at all.
Light the phone can only guess at
ARKit estimates the overall brightness and colour of the light in a room, which helps virtual objects look like they belong. It doesn't know where each lamp is. A virtual vase can end up with a shadow pointing the wrong way from the real shadows around it. Users may not name the problem, but they feel it.
Pro tip: build a "bad room" test kit
Before any demo or launch, test in the worst room you can find: a plain white wall, a glass coffee table, a mirror, a dim lamp, and a floor with no pattern. Add one older non-Pro iPhone. If the experience holds up there, the office demo will be fine.
When the signals disagree
Data that contradicts itself is harder to spot than missing data, because nothing obviously fails. ARKit constantly combines inputs that don't always agree, and your app has to decide what to do when they clash.
The camera says still, the body says moving
In a moving car, train or lift, the motion sensors report acceleration while the camera sees an interior that isn't moving. ARKit can't tell which to believe, so content slides or spins. World tracking is built for still surroundings, so detect heavy drift and tell the user the experience needs solid ground.
The jump after recovery
After an interruption, ARKit may recognise the room and snap its position back into place. That's called relocalisation. The map is correct again, but every object can visibly jump by several centimetres. You choose how the user sees that change, and a short fade out and back in feels calmer than a sudden teleport.
Surfaces that merge
ARKit often detects the same floor as two separate planes and later realises they're one. When it merges them, it removes one of the plane records. A sofa attached to the removed plane can vanish. The fix is to give each placed object its own anchor, a fixed point ARKit tracks for you, at the exact spot the user tapped.
Depth versus estimate
On LiDAR devices, the room mesh might put a table edge in one place while a quick flat-surface estimate puts it slightly elsewhere. Trust the mesh when its confidence is high and fall back to the estimate when it isn't. What matters is choosing on purpose instead of letting whichever answer arrives first win.
Two phones, two maps
In shared sessions, each phone builds its own map and the two are aligned imperfectly, so one person may see a game board ten centimetres left of where their friend sees it. Apple AR apps that handle this well usually give one device authority over where shared objects sit, and let the other devices follow its lead.
The pattern is the same each time: decide in advance which source wins, and make corrections gentle. Users forgive a small, smooth adjustment. They don't forgive a lamp that jumps across the room.
Sixteen milliseconds to decide
Most AR sessions run at 60 frames per second. That gives your app roughly 16.7 milliseconds per frame to read ARKit's latest update, run its own logic and draw the scene. Miss that window regularly and virtual objects lag behind the camera image, which feels like a wobble and gets tiring. If you finish an ARKit tutorial and your app works but feels sluggish, this section is probably why.
Reading the tracking state every frame
ARKit reports its confidence on every frame. Here's how the states map to sensible behaviour:
Apple provides a ready-made coaching overlay that shows animated hints for the early states. Use it before designing your own.
Fast guesses versus slow certainty
To place an object, the app sends an invisible line from the tapped point and checks where it hits a surface. This is called a raycast. It can check against confirmed surfaces, which is reliable but may not be ready yet, or suspected ones, which is instant but rough.
Use the fast guess for a see-through preview, and commit the placement once the hit lands on a confirmed surface. Tracked raycasts keep refining the answer over the next few frames.
Keeping heavy work out of the frame loop
Running object recognition or other machine learning on every camera frame will blow the budget. Run it every third or fifth frame on a background thread. Also, if your code holds on to too many camera frames, ARKit logs a warning and stops delivering new ones, and the experience freezes. Copy what you need and let the frame go. Network requests don't belong in the frame loop either.
The edge cases that break demos
Real users do things no one planned for. These cases cause the most trouble.
• A phone call arrives mid-session. The camera stops, and ARKit reports an interruption. Your app needs a clear rule for what happens next: recover old positions, or start fresh and explain why.
• The user taps "Don't Allow" on the camera prompt. Without a fallback they see a black screen and leave. A plain 3D viewer keeps them in the app.
• Mirrors and windows. ARKit can treat a reflection as a real space and detect a floor inside a mirror.
• Kids, pets and moving crowds. Moving things confuse feature tracking, so busy shop floors are hard.
• Rooms that change. ARKit can save a room map for later, but if furniture moves, it may fail to match. Let saved maps expire and offer a quick rescan.
• Users who can't walk around freely. Offer rotate and move gestures so the object comes to them, and label controls for VoiceOver, Apple's screen reader.
• Promises about size. A 5% error on a two-metre sofa is ten centimetres, enough to decide whether it fits. Lock scale for product views.
Privacy deserves its own mention. An AR app on iPhone receives the live camera feed, which often shows someone's home and family. For most iOS augmented reality features, no frames need to leave the device, and saying so plainly in the permission text builds trust.
Pro tip: log the boring numbers
Track time to first detected surface, how long sessions spend in each tracking state, placement success rate, device model and heat warnings. These tell you more than any survey.
How the system behaves under pressure and at scale
AR is one of the hardest things a phone can do. The camera, sensors, processor and graphics chip all run hard at once.
Heat, battery and slowdown
After ten or fifteen minutes of AR, most phones get warm. iOS reports a thermal state with four levels: nominal, fair, serious and critical. Ignore it and iOS slows the processor anyway, so the app stutters on its terms instead of yours.
Plan a reduced mode and switch to it at serious: drop to 30 frames per second, turn off room scanning, remove soft shadows. Users rarely notice a simpler scene. They always notice stutter.
Memory in large spaces
A living room produces a manageable 3D mesh. A warehouse produces one that keeps growing as the user walks, and thousands of virtual labels across a factory will strain memory. Large-site apps divide the space into zones and load only nearby content.
The real cost is the 3D content
For a business, the biggest scaling problem is usually 3D models. A file made for marketing renders can hold millions of polygons, and phone AR needs a small fraction of that. Selling 5,000 products means 5,000 models that are optimised, size-checked and tested. Budget that like a production line.
Apple's Object Capture builds models from photos and cuts that cost, but it struggles with shiny, transparent and thin objects, just as ARKit does.
Device spread across your user base
Across thousands of users you'll see old phones, new phones and cracked camera lenses. Plan features in tiers, with a basic tier that runs everywhere. Serious ARKit development teams test on the oldest supported device first, because that's where most performance problems appear.
Delivery and caching
Don't bundle every model inside the app. Download on demand, cache popular ones and show a placeholder while files arrive. Shared sessions suit small groups in one room, so anything larger needs a server holding the source of truth.
A practical build path
This isn't a full ARKit tutorial, but it's the order of work that avoids the most rework.
Step 1: Check whether you need an app at all
If the goal is letting shoppers see a product in their room, AR Quick Look may be enough. Host a USDZ file, link it from your product page, and iPhone users can tap to see the item at full size in their room, with no app at all. Many retailers test demand this way before commissioning full Apple AR apps.
Step 2: Set up the project properly
In Xcode, Apple's development tool, add a camera usage description, the sentence iOS shows when asking for camera access. Without it, the app crashes when it opens the camera. Honest wording affects how many people tap "Allow".
Step 3: Start a world-tracking session
This short example turns on floor detection, uses the room for reflections, and adds room scanning only on LiDAR devices:
let config = ARWorldTrackingConfiguration()
config.planeDetection = [.horizontal]
config.environmentTexturing = .automatic
if ARWorldTrackingConfiguration.supportsSceneReconstruction(.mesh) {
config.sceneReconstruction = .mesh
}
arView.session.run(config)
The "if" block is the hardware-tier idea in code.
Step 4: Guide the user, then let them place
Add the coaching overlay and a translucent preview, and allow final placement only when tracking is normal and the raycast hits a confirmed surface.
Step 5: Handle interruptions and tracking changes
Respond to every tracking state and to calls or trips to the home screen. This step takes longer than the first four combined.
Step 6: Test on real phones in bad rooms
The iOS Simulator can't run camera-based AR, so test on physical devices in the bad room, and run sessions long enough to see heat effects.
Step 7: Measure after launch
Ship with the metrics above, then fix the devices and conditions that fail most often.
Choosing your toolset: ARKit and the alternatives
ARKit isn't the only way to build AR, and for some teams it isn't the right one.
ARKit also runs on visionOS, the Apple Vision Pro operating system, with a redesigned interface where apps don't get the raw camera feed by default. The ideas of tracking and anchors carry over, so iPhone AR is a reasonable first step toward headset work.
Who should build now? Businesses where customers judge size, fit or placement, such as furniture, eyewear and real estate. Field service and training teams. And companies that control their hardware, like a fleet of company iPads.
Who might wait? Teams whose product has no physical-world moment. If no customer ever points a phone at something real, AR will feel like a detour.
Key takeaways
▸ ARKit works by combining what the camera sees with how the phone moves, so it needs light, texture and fairly still surroundings.
▸ LiDAR improves nearly everything but only ships on Pro devices, so design a basic tier that works without it.
▸ Most "buggy AR" comes from data gaps: plain floors, glass, dim rooms and placements made in the first couple of seconds.
▸ Decide in advance which signal wins when the camera, sensors and maps disagree, and correct positions smoothly.
▸ Every frame has about 16.7 milliseconds of budget, so keep heavy processing and network work out of the frame loop.
▸ Plan a reduced mode for heat, and budget for 3D content production, which is usually the biggest cost at scale.
▸ AR Quick Look is a cheap way to test demand before building a full app.
Wrapping up
Meera's team fixed the floating sofa in about a week. They held back final placement until tracking settled, anchored each object at the tapped point, added coaching prompts for dim rooms, and ran every build through a plain-floored test room with a single lamp. None of those were new features.
ARKit gives you a lot for very little code, and a prototype can run in an afternoon. What separates a toy from a product is planning for missing data, conflicting signals, tight time limits, heat and scale. Teams that start there tend to ship iOS augmented reality features that people use more than once, which is the only number that really counts.
