Create an augmented reality application with facial detection

Get the code View the demo Build on IBM Cloud


Augmented reality provides an enhanced version of reality by superimposing virtual objects over a user’s view of the real world. ARKit blends digital objects and information with the environment around you, taking apps far beyond the screen and freeing them to interact with the real world in entirely new ways. This code pattern combines ARKit with Watson Visual Recognition and a Cloudant database to give you a complete augmented reality experience.


The easiest way to find and connect to people around the world is through social media apps like Facebook, Twitter, and LinkedIn. However, these only provide text-based search capabilities. With the recently announced release of the iOS ARKit toolkit, you can now do a search using facial recognition. Combining iOS face recognition using the Vision API, classification using IBM Watson Visual Recognition, and person identification using classified image and data, you can build an app to search faces and identify them. One use case is to build an augmented reality-based résumé using visual recognition.

This code pattern explains how to create this type of augmented reality-based résumés with Visual Recognition. The iOS app recognizes the face and presents you with the AR view that displays a résumé of the person in the camera view. The app uses Watson Visual Recognition and Core ML to classify the image and then uses that classification to get details about the person from data stored in an IBM Cloudant NoSQL database. The images are classified offline using a deep neural network that is trained by Visual Recognition.

After completing this code pattern, you should know how to:

  • Configure ARKit
  • Use the iOS Vision module
  • Create a Swift iOS application that uses the Watson Swift SDK
  • Classify images with Watson Visual Recognition and Core ML


augmented reality application with facial detection

  1. User opens the app on their mobile device.
  2. A face is detected using the iOS Vision module.
  3. An image of the face is sent to the Watson Visual Recognition service to be classified.
  4. The information from the JSON database related to the user is placed in front of the original person’s face in the mobile camera view.


Find the detailed steps for this pattern in the README. Those steps will show you how to:

  • Clone the ar-resume-with-visual-recognition GitHub repo.
  • Log in to IBM Cloud and create a Visual Recognition service.
  • Create an IBM Cloudant NoSQL database.
  • Install the dependencies.
  • Run the app.