How to incorporate 3D Maps in Augmented Reality using Mapbox and ARKit
I recently wanted to see if I could implement 3D maps in augmented reality for a potential installation. I’d never used ARKit, my experience with 3D was fairly limited as well, and I don’t know a thing about true native app iOS development (React Native is as close as I’ve come). Luckily, Mapbox makes an incredible plugin for handling maps in Unity, and there’s also a recent(very beta, at the time of writing) plugin (set of scripts and examples, really) for incorporating ARKit in Unity.
With a little bit of trial and error, I was able to get things working, and while I’m by no means an authority or even a good resource for help, I can share my process and the resources which helped me, as well as some of the rationale for using Unity for AR.
First, quick breakdown of steps:
- Set up your environment for ARKit development on Unity
- Generate a working ARKit example in Unity / XCode
- Generate a working Mapbox Map in Unity
- Integrate ARKit into your map scene.
Setting up your environment
As it is in beta, setting up your environment is by far the most complex step, and there are issues you may come across that are specific to your unique environment, but here is what worked for me. If you are already set up with ARKit, and already set up with Mapbox in Unity, feel free to skip this section.
- Sign up for an apple developer account
- Get XCode 9 Beta (must have a developer account) for your Mac
- On your iPhone, go to the downloads page and get the provisioning profile for iOS 11 Beta. This article provides great instructions.
- Once you’ve installed the profile, you’ll be able to go to Settings > General > Software Update, and see the ios11 Beta Install. Install it.
- Install the beta version of Unity 2017
NOTE: In order to be able to see the Mapbox menu within Unity, I had to download Unity 2017.1.0f3, even though the most recent version of Unity is 2017.2.0b11 (the menu did not appear in this version). You may not have this is issue, but 2017.1.0f3 is what Mapbox was running internally, so that’s what they recommended I use and it worked.
- Sign up for a Mapbox Developer Account. This will assign you an auth token you can use to be able to use the plugin within Unity.
- Get the Mapbox Unity SDK.
- Get the Unity ARKit plugin (you can also get this from within Unity by going to Window > Asset Store and search for ARKit (link).
You should now have everything you need to get started building.
Generating a working ARKit example
- Start a new Unity project. Make sure it’s 3D, and add the Unity ARKit Plugin Asset from the menu here, or, if your project is already built, by importing it from the asset store.
- In your project navigator tab, under Assets, you should see
UnityARKitPlugin
, and under that,Examples
. For those who do well reverse engineering, the plugin provides some cool example apps. A good place to start is insideUnityARKitScene
which provides plane detection, point cloud rendering, and the ability to place 3D objects on those found planes (or anywhere else in the scene, really). This is also a good example to get the components you’ll need to render your Mapbox map in AR later. - Open up the scene, play it to make sure you have no immediate errors (you shouldn’t, if you’ve met all of the above requirements)
- You’ll need to build to iOS so you can test it out on device. To do so, go to build settings (Shift+Cmd+B), and make sure only
UnityARKitScene
is selected underScenes in Build
. Selectios
as your platform, and run as release, then clickBuild
. I usually create aBuilds
folder at the root of my project and then save into that. - It will take a while to compile everything. For me, I even got what I thought was the wheel of death, with Apple saying that the application was not responding. But it should build eventually, at which point you can go inside your project name in the builds folder and double click the
xcodeproj
file to open your buid in XCode. - By default, your XCode build will not select a default signing team, which you’ll need to compile the app to your phone. You’ll need to select it from the menu by clicking on the icon in the project navigator (mine is called Unity-iPhone), clicking on your target (again,
Unity iPhone
), and going toSigning > Team
. This tutorial does not cover that process, but I have previously set myself up as the Team. - I also want to call out a minor issue I had, which was that after building a couple of the example projects, they all used the Bundle Identifier
com.unity.arkitscene
. This will cause errors in XCode if two projects share the same bundle identifier, so you may need to change the bundle identifier if you have multiple projects, which can be done in the same menu above, underIdentity > Bundle Identifier
.
- Once you’ve done all this, plug in your device (iphone 6s or higher, running the iOS 11 beta), and select it at the top of XCode next to the play button as the device you want to build to, then hit play. XCode will compile the project, which might take some time, and eventually it should compile without error, and appear on your home screen on your device. Mine auto-launches once the build is complete, but I have heard of issues with some devices not automatically launching the app, in which case you need to just find it on your homescreen and click it to launch.
- You should see a standard camera feed, and, in the right lighting conditions with perhaps a bit of time, you will see yellow dots representing the point cloud from ARKit, as well as a checkerboard cube in the center of the scene with blue, red, and green poles representing the 3D axis lines. When your camera finds something it identifies as a plane, the yellow crosshair-like box will close in to form a complete rectangle, and you can click the screen to place a 3D cube. If you place a 3D cube without the yellow rectangle, ARKit will do its best to determine where in space to place the scene, but it will not be as accurate.
- I’ve uploaded my Unity files [here].
Congrats! Hopefully you’ve got ARKit working in Unity! If not, head to the support forums and ask or search for help.
Generating a Mapbox Map in Unity
I won’t go into much detail here, because Mapbox did a phenomenal job of creating a walkthrough of how to build a 3D map in Unity, but basically here’s the process:
- Follow this guide to install the SDK, clicking Next after each step is successfully completed.
- Complete all three phases of this tutorial (you can get by with just the first step, but the fun stuff comes in parts 2 and 3), and make sure you can successfully build a new Unity project (without ARKit), that you can click play in Unity and view.
- If you have trouble or feel you’ve found a bug, file an issue in the Github repository. Mapbox was extremely quick to respond to my own issue with using the most recent beta of Unity and not being able to see the “Mapbox” menu item.
- I’ve also uploaded my files [here] (you’ll need to make sure you add your own Mapbox access token). Maybe it will be of some help.
Putting it all together
While the documentation on how to move forward wasn’t super-intuitive to me, the Unity ARKit plugin actually gives you everything you need to integrate your own content into ARKit, and for this particular example, all we have to do is drag the right scripts to the right objects.
I first had some errors by trying to incorporate my MapBox map into a Unity project, so I did the reverse, took scripts from ARKit, and implemented them in my MapBox project. To get everything working:
- Import the ARKit plugin into your mapbox project.
- Make an empty game object called
ARCameraManager
and drag a reference to yourMain Camera
from the Hierarchy. Choose whether you wantGet Point Cloud
andLight Estimation
in the script options in the Inspector. Everything else should be default.
- If you want to generate planes so you can see when ARKit finds a surface to attach things to, copy the
GeneratePlanes
game object from theUnityARKitScene
in the ARKit Examples into your Mapbox Scene. - Create another game object called
MapParent
and drag your MapboxMap
Game Object that you created in their tutorial into the MapParent. This is useful in allowing you to scale your map. I changed my scale of the map parent to.05
a bit arbitrarily, but if you want an enormous life size AR Map, leave it at 1 (or bigger?).
- On your actual Map Game Object, attach the
UnityARHitTest Example
script from theARKit > Helpers
folder. For theHitTransform
parameter, drag theMapParent
Game Object from the Hierarchy into the Inspector field. This will make it so that you can place your map by clicking on the screen. Otherwise, it just floats in the middle of space at about eye level.
- On your
Main Camera
, attach theUnityARVideo
script from theARKit > Helpers
folder, and for theClear Material
field, select theYUV Material
fromUnityARKit > Materials
. Also, above in the inspector, make sureDepth Only
is selected underClear Flags
. - Finally, attach the
UnityARCamera Near Far
script toMain Camera
fromARKit > Helpers
folder.
(I think) that’s it! You should be able to now go to build settings as you did before, and build for iOS. Then open your xcodeproj
file in XCode, set your signing team again, and push play to build to your phone. Hopefully, if you did everything right, you will now be able to place your 3D Map in Augmented Reality!