How to incorporate 3D Maps in Augmented Reality using Mapbox and ARKit

I recently wanted to see if I could implement 3D maps in augmented reality for a potential installation. I’d never used ARKit, my experience with 3D was fairly limited as well, and I don’t know a thing about true native app iOS development (React Native is as close as I’ve come). Luckily, Mapbox makes an incredible plugin for handling maps in Unity, and there’s also a recent(very beta, at the time of writing) plugin (set of scripts and examples, really) for incorporating ARKit in Unity.

With a little bit of trial and error, I was able to get things working, and while I’m by no means an authority or even a good resource for help, I can share my process and the resources which helped me, as well as some of the rationale for using Unity for AR.

What we will be creating

First, quick breakdown of steps:

• Set up your environment for ARKit development on Unity
• Generate a working ARKit example in Unity / XCode
• Generate a working Mapbox Map in Unity
• Integrate ARKit into your map scene.

Setting up your environment

As it is in beta, setting up your environment is by far the most complex step, and there are issues you may come across that are specific to your unique environment, but here is what worked for me. If you are already set up with ARKit, and already set up with Mapbox in Unity, feel free to skip this section.

• Sign up for an apple developer account
• Get XCode 9 Beta (must have a developer account) for your Mac
• On your iPhone, go to the downloads page and get the provisioning profile for iOS 11 Beta. This article provides great instructions.
• Once you’ve installed the profile, you’ll be able to go to Settings > General > Software Update, and see the ios11 Beta Install. Install it.
• Install the beta version of Unity 2017

NOTE: In order to be able to see the Mapbox menu within Unity, I had to download Unity 2017.1.0f3, even though the most recent version of Unity is 2017.2.0b11 (the menu did not appear in this version). You may not have this is issue, but 2017.1.0f3 is what Mapbox was running internally, so that’s what they recommended I use and it worked.

• Sign up for a Mapbox Developer Account. This will assign you an auth token you can use to be able to use the plugin within Unity.
• Get the Mapbox Unity SDK.
• Get the Unity ARKit plugin (you can also get this from within Unity by going to Window > Asset Store and search for ARKit (link).

You should now have everything you need to get started building.

Generating a working ARKit example

• Start a new Unity project. Make sure it’s 3D, and add the Unity ARKit Plugin Asset from the menu here, or, if your project is already built, by importing it from the asset store.
• In your project navigator tab, under Assets, you should see UnityARKitPlugin , and under that, Examples. For those who do well reverse engineering, the plugin provides some cool example apps. A good place to start is inside UnityARKitScene which provides plane detection, point cloud rendering, and the ability to place 3D objects on those found planes (or anywhere else in the scene, really). This is also a good example to get the components you’ll need to render your Mapbox map in AR later.
• Open up the scene, play it to make sure you have no immediate errors (you shouldn’t, if you’ve met all of the above requirements)
• You’ll need to build to iOS so you can test it out on device. To do so, go to build settings (Shift+Cmd+B), and make sure only UnityARKitScene is selected under Scenes in Build. Select iosas your platform, and run as release, then click Build. I usually create a Builds folder at the root of my project and then save into that.
• It will take a while to compile everything. For me, I even got what I thought was the wheel of death, with Apple saying that the application was not responding. But it should build eventually, at which point you can go inside your project name in the builds folder and double click the xcodeproj file to open your buid in XCode.
• By default, your XCode build will not select a default signing team, which you’ll need to compile the app to your phone. You’ll need to select it from the menu by clicking on the icon in the project navigator (mine is called Unity-iPhone), clicking on your target (again, Unity iPhone), and going to Signing > Team . This tutorial does not cover that process, but I have previously set myself up as the Team.
• I also want to call out a minor issue I had, which was that after building a couple of the example projects, they all used the Bundle Identifier com.unity.arkitscene. This will cause errors in XCode if two projects share the same bundle identifier, so you may need to change the bundle identifier if you have multiple projects, which can be done in the same menu above, under Identity > Bundle Identifier.

XCode Settings (May want to click that “Update to Recommended Settings if you’re having trouble).

• Once you’ve done all this, plug in your device (iphone 6s or higher, running the iOS 11 beta), and select it at the top of XCode next to the play button as the device you want to build to, then hit play. XCode will compile the project, which might take some time, and eventually it should compile without error, and appear on your home screen on your device. Mine auto-launches once the build is complete, but I have heard of issues with some devices not automatically launching the app, in which case you need to just find it on your homescreen and click it to launch.
• You should see a standard camera feed, and, in the right lighting conditions with perhaps a bit of time, you will see yellow dots representing the point cloud from ARKit, as well as a checkerboard cube in the center of the scene with blue, red, and green poles representing the 3D axis lines. When your camera finds something it identifies as a plane, the yellow crosshair-like box will close in to form a complete rectangle, and you can click the screen to place a 3D cube. If you place a 3D cube without the yellow rectangle, ARKit will do its best to determine where in space to place the scene, but it will not be as accurate.
• I’ve uploaded my Unity files [here].

Congrats! Hopefully you’ve got ARKit working in Unity! If not, head to the support forums and ask or search for help.

Generating a Mapbox Map in Unity

I won’t go into much detail here, because Mapbox did a phenomenal job of creating a walkthrough of how to build a 3D map in Unity, but basically here’s the process:

• Follow this guide to install the SDK, clicking Next after each step is successfully completed.
• Complete all three phases of this tutorial (you can get by with just the first step, but the fun stuff comes in parts 2 and 3), and make sure you can successfully build a new Unity project (without ARKit), that you can click play in Unity and view.
• If you have trouble or feel you’ve found a bug, file an issue in the Github repository. Mapbox was extremely quick to respond to my own issue with using the most recent beta of Unity and not being able to see the “Mapbox” menu item.
• I’ve also uploaded my files [here] (you’ll need to make sure you add your own Mapbox access token). Maybe it will be of some help.

Putting it all together

While the documentation on how to move forward wasn’t super-intuitive to me, the Unity ARKit plugin actually gives you everything you need to integrate your own content into ARKit, and for this particular example, all we have to do is drag the right scripts to the right objects.

This will be your final Unity hierarchy

I first had some errors by trying to incorporate my MapBox map into a Unity project, so I did the reverse, took scripts from ARKit, and implemented them in my MapBox project. To get everything working:

• Import the ARKit plugin into your mapbox project.
• Make an empty game object called ARCameraManager and drag a reference to your Main Camera from the Hierarchy. Choose whether you want Get Point Cloud and Light Estimation in the script options in the Inspector. Everything else should be default.

AR Camera Manager Game Object

• If you want to generate planes so you can see when ARKit finds a surface to attach things to, copy the GeneratePlanes game object from the UnityARKitScene in the ARKit Examples into your Mapbox Scene.
• Create another game object called MapParent and drag your Mapbox Map Game Object that you created in their tutorial into the MapParent. This is useful in allowing you to scale your map. I changed my scale of the map parent to .05 a bit arbitrarily, but if you want an enormous life size AR Map, leave it at 1 (or bigger?).

Map Parent settings

• On your actual Map Game Object, attach the UnityARHitTest Example script from the ARKit > Helpers folder. For the HitTransform parameter, drag the MapParent Game Object from the Hierarchy into the Inspector field. This will make it so that you can place your map by clicking on the screen. Otherwise, it just floats in the middle of space at about eye level.

The Map Game Object setup

• On your Main Camera, attach the UnityARVideo script from the ARKit > Helpers folder, and for the Clear Material field, select the YUV Material from UnityARKit > Materials. Also, above in the inspector, make sure Depth Onlyis selected under Clear Flags.
• Finally, attach the UnityARCamera Near Far script to Main Camerafrom ARKit > Helpers folder.

The Main Camera Game Object

(I think) that’s it! You should be able to now go to build settings as you did before, and build for iOS. Then open your xcodeproj file in XCode, set your signing team again, and push play to build to your phone. Hopefully, if you did everything right, you will now be able to place your 3D Map in Augmented Reality!