Make an AR Drawing App – Part 1

Make an AR Drawing App – Part 1

Introduction

Check out the other parts of this series:

One of the great things about Augmented Reality is just what its name implies: you get to add-to reality. Some developers have done this in the form of a “play basketball with a dinosaur” app or a “fight the aliens invading your living room” game. This has been the trend in the development of augmented experiences.

But what if there was a different way to augment reality? What if there was a way for you, the player, to directly augment your reality with what you create? What if you could see your own designs in your own living room? That is essentially the premise of this project.

In this tutorial series, we will create an AR “drawing app” where the player can draw directly in 3D, augmented space. This allows for another exhilarating level of augmented experience. It allows the user to dictate how and where reality will be augmented.

We will be using a tool in Unity called “AR Foundation” which is Unity’s own AR ecosystem. AR Foundation will allow us to detect flat surfaces in the real world. Because of this, we can allow the player to draw above a kitchen table or on the table. This project will be able to run on iOS and Android devices, though there will be some requirements when it comes to the hardware. Also, this tutorial series is intended for people who have some knowledge of Unity and C# scripting.

Even if you aren’t sure if this describes you, just give this tutorial series a try! You might surprise yourself!

Source Code files

You can download the tutorial source code files here.

Part 1 of this Series

This tutorial is part of a series on AR Foundation. In this tutorial, we’ll set up our AR Foundation project to run on a mobile device, create our 3D pen point for drawing, and set up the camera so that we can test our mechanic in the Unity Editor.

Creating the project

Open up Unity Hub and create a new project to run on the latest version of Unity installed. Name it “AR Drawing App.” Set it to a 3D project and hit “create.”

Unity Hub new project window for 3D project

The very first thing we’re going to do is to get the device set up so that we can test our AR project.

Setting up our device: Android

In order for AR Foundation to work on your device, it must first be compatible with ARCore for Android. Have a look at the supported devices on this page: https://developers.google.com/ar/discover/supported-devices Building to Android is really simple. If you didn’t add Android build support when you downloaded Unity, you can change that by simply going to the “Installs” portion on the Unity Hub. Click the “extra” icon on the top portion of your editor version and select the “Android Build Support” package.

Unity Hub with Installs shown

Make sure you include the “Android NDK and SDK Tools.” Unity Hub Add Modules window open with Android modules selectedClick “Done” and you are all set up. To run a project on your Android device make sure “USB debugging” is enabled (you may have to look up how to do this since it is different for each device) and plug your device into your computer. Then supply a Package Name in the Player Settings, and supply a description in the “Camera Usage Description”.

 

Unity Project settings with camera usage description highlighted

Then hit “Build and Run.”

Unity File Menu with Build And Run selected

The project will compile and download to your device. It is important to frequently test on a device when developing AR applications.

Setting up our device: iOS

Building to iOS is a bit more involved. First, you have to have a Mac running the latest version of OSX and the latest version of Xcode. Second, you must have an Apple device with at least an A9 chipset. This basically means an iPhone 6S (or iPhone SE) or later. And finally, you must have an Apple Developer account. These accounts are free to create but you won’t be able to publish to the App Store. Go to https://developer.apple.com/ to create an account and download Xcode. Then you’re going to need to hook up your developer account to Xcode.

Xcode menu open with Preferences option selected

Mac OS Accounts window with arrow pointing to plus sign

Now go to Unity and provide a Package Name and supply some description in the “Camera Usage Description”. AR Foundation is currently only compatible with “ARM64” architecture so set it to the appropriate setting.

Unity project settings window with Architecture set to ARM64

Now you can hit “Build.” It’ll ask you for a folder to store it in so go ahead and create a brand new folder.

ARProj7 folder in File viewer

Then open that same project in Xcode.

Xcode project window

Unity-iPhone.xcodeproj file selected in file hierarchy

Navigate to the build settings and set the Provisioning Team to your developer account.

Signing options window for XCode project

If it gives you a signing error then simply add a few numbers to the end of your Package Name.

Xcode Signing options with Status section circled

Going from Unity to Xcode is sometimes fiddly and so it may take a couple of Google searches before you finally can build. For example, when I opened my project in Xcode, it gave me an error that said “The Folder ‘ARReferenceImages.xcassets’ doesn’t exist”,

Mac warnings about folders now existing

I simply closed this out had no issues. Plugin your iPhone or iPad and make sure it is recognized by going to the “Devices and Simulators” window.

Window menu with Devices and Simulators selected

Finally, hit build and run.

Xcode with play button circled

The project should start up on your device once it is done.

Importing the packages

Now that our device is ready to test our AR project, let’s go ahead and import AR Foundation. Go to Window -> Package Manager

Unity Window menu with Package Manager selected

and wait for it to load the installed packages. If you can’t see AR Foundation as a package then you need to click on “Advanced” and then “Show preview packages.”

Unity Packages window with Show preview packages selected

Select AR Foundation and click “Install.”

AR Foundation package information in Unity

Once it is done installing, we need to know if we’re going to develop for Android or iOS. If for Android, then we import the ARCore package.

ARCore XR Plugin package information in Unity

If for iOS, then we import the ARKit package.

ARKit XR Plugin information within Unity

Chose the package that you need and then install it. Once it is done, we can go set up our scene.

Error Warning in Xcode

It is worth noting that you may have to downgrade your package if you’re getting “Undefined symbol” errors in Xcode. This has happened to me a couple of times and the solution I found was to downgrade the packages (AR Foundation and ARKit XR) until the error disappeared in Xcode. Like I mentioned before, exporting to a build platform is very fiddly and may take a couple of Google Searches before you find success.

Setting up the Scene

The first thing we will need is what’s called an “AR Session.” Create an empty game object called “AR Session” and add an “AR Session” component.

Unity Hierarchy with Create Empty selected from right click menu

Unity Inspector with AR Session selected from Add Component

An AR Session is what starts and stops the Augmented Reality session on the device. With this, we can start, stop, or pause any AR activities on the device. Next, add an input manager component (search “AR Input Manager”) to this game object

AR Input Manager component selected in Unity Inspector

and we’re done with the AR Session object. We have an AR Session so we can start augmented reality on the device but we need a way to take the information that’s being gathered by the device and turn it into usable 3D information that can translate over to Unity. To do this, we use another component called an “AR Session Origin.” This takes the tracked features in the real world and converts it to 3D coordinates that Unity can use. We’re going to add this into the scene in a different way than when we added the AR Session. Right-click in the hierarchy and go to “XR” -> “AR Session Origin.”

Unity Hierarchy with XR Menu, AR Session Origin selected

As you can see, this has the “AR Session Origin” component and an AR Camera parented to the object.

Unity with AR Camera and AR Session Origin script highlighted

There are a couple of components this AR Session Origin is going to need for our project. The first is an “AR Plane Manager.”

Unity Inspector with AR Session Origin component selected

This takes the tracked feature points and determines if a flat surface lies somewhere in the real world; if it does then it spawns a game object. Right now, the “plane prefab” field is empty. Create a new folder called “prefabs”Prefabs folder in Unity Assets folder

and then right-click in the hierarchy to create an “AR Default Plane.” Unity Hierarchy with XR, AR Default Plane selected from right-click menu

Make this a prefab and then assign it to the field on the “AR Plane Manager” script.

AR Default Plane object in Hierarchy and AR Plane Manage Script

Next, we’ll need an “AR Point Cloud Manager.”

AR Point Cloud Manager selected from Unity Inspector

This recognizes the existence of tracked feature points that are close to each other. Once it does recognize these points, it will spawn the prefab in the “Point Cloud Prefab” field. Create this in the same way we did the Default Plane (create it in the scene,

Unity Hierarchy with XR, AR Default Point Cloud selected

make it a prefab, and then drag it in the field on the Point Cloud Manager).AR Default Point Cloud object in Hierarchy and AR Point Cloud Manager Script

Now delete both of these objects in the scene since we no longer need them.

Unity right click menu with Delete selected

Since the AR Session Origin already has a camera we can delete the default camera.

Unity right-click menu with delete selected over main camera

We just need to make sure that the AR Camera has the tag “MainCamera.”

Unity Inspector options

We’ll be needing this later on. Having done all this, let’s go ahead and build to our device to see if everything is functioning properly.

Unity Build Settings with Build button highlighted

Smart phone with Unity AR project

Looks like everything is working, you’ll notice we can track both horizontal and vertical planes with AR Foundation. We can change it to track only horizontal or vertical planes by changing the “Detection Mode” on the AR Plane Manager.

AR Plane Manager script with Detection Mode set to Everything

Let’s leave it set to “Everything” for this project.

Drawing in 3D Space: the Paint Stroke

Now that our project can detect planes and trackable points, let’s start making the drawing mechanic. Create a new empty game object called “Stroke” and make it a prefab.

Unity Hierarchy with Stroke object selected

We are going to be using a component called a “Trail Renderer” in order to make a paint stroke. On the Stroke game object, add a Trail Renderer component.

Trail Renderer component in Unity Inspector

Drag it around and you’ll see that it leaves a trail (this is actually a new mesh) that eventually fades.

Strokes seen in Unity Scene view

We want our trail to never fade so we need to change the “Time” value in the inspector to “infinity” (type this word in the number field).

Unity Trail Renderer Component with Time set to Infinity

Next, our stroke is a very ugly pink color. This is because it currently has no material. Go to the “Materials” array in the Trail Renderer component and set the length to 1.

Trail Renderer Component with Size set to 1

Now we need to create a material for the stroke. Go to your project folder and create a new folder called “Materials”,

Materials folder selected in Unity

then create a new material in that folder called “PenColor.”

Unity menu process to create new material

We don’t need to do anything to this material yet so go ahead and just place it in the Materials field.

New Unity Material in Inspector

You’ll now see that our paint stroke has turned from pink to white!

Unity Stroke object with grey materials

Now, apply the modified component to the prefab and we’re done with our paint stroke.

Unity Added Component menu with Apply to Prefab 'Stroke' selected

Delete the Stroke game object from the scene view and we can now move onto setting up the camera for testing in Unity.

Drawing in 3D Space: the Camera

When developing a game like this, we need a way to test our drawing mechanic right in the Unity Editor as that will allow us to see if any changes need to be made. A simple way to do this is just to make a simple mouse-look script as that (for our intents and purposes) will simulate the movement of the camera during an AR session. Create a new folder named “Scripts”

Unity scripts folder highlighted

and create a new C# script called “Draw.”

Unity script being dragged onto object

Assign this to our AR Camera and then boot up Visual Studio (or your preferred script editor) to make changes to our Draw script. We’re just going to be implementing a mouse-look mechanic so this is all the code we’ll need:

As you can see, we use a public boolean to determine if we are testing in the Editor. This boolean should be set to false when you export the game so remember to do this before you export. Hit save and then go back to Unity. Enable “Mouse Look Testing” and then hit play. You should be able to tell by the way the gizmo is moving that we’ve got a mouse-look mechanic!

Unity camera object's position highlighted

Now we can move onto creating the pen point in 3D space.

Drawing in 3D Space: the Pen Point

Create a new Sphere called “Pen Point” and parent it to the AR Camera.

Unity Hierarchy with PenPoint object highlighted

This will serve as our 3D Penpoint. Make sure the position of this point is extremely close to the camera on the Z-axis (mine was less than 0.25). This is because one world unit in Unity is about one meter in the real world. This also means that we need to scale this sphere way down. If we scale it down to 0.1 (on each axis), that would be about ten centimeters in the real world which is way too large for a pen point. We have to go down to 0.05 in order to get a decent size. Your Transform component should look like this and your sphere should look like this.

Unity PenPoint Object's Transform Component circled

Alright! We’re done with the 3D pen point.

Conclusion

I like to think that we’ve now built the launch pad by which we rocket off into augmented reality worlds. We are now able to test an augmented reality app on a mobile device, we have got a paint stroke setup, and we’ve got a way to test our drawing mechanic in the editor.

In the next tutorial, we will begin to actually draw in 3D space, and we’ll build it to the device to make sure everything is working properly. Then we’ll set up the necessary components to start drawing on an AR detected plane. We will learn about how AR Foundation stores position information and how to raycast in augmented reality. Lot’s of cool things happening in the next tutorial, so come back on and let’s continue to augment our reality.