Recognize Your Customers with Face Recognition, Cloud AI, and Dynamics 365

Artificial Intelligence has been influencing how we live for some time. And now more than ever, the tech giants have been competing to win businesses and developers with their AI services. Microsoft is one of these, and through it’s Azure cloud it has been democratizing the power of various AI capabilities for businesses and developers to take advantage of.

In previous posts I have blogged about using voice recognition technology to create your own voice assistant to get the weather, as well as how to use Azure’s web search APIs. In this post I will show how face recognition can be used to help solve businesses challenges.

Microsoft’s Face API sits within its Cognitive Services cloud offering. Its capabilities include recognizing a face in a given image, comparing a face to different people, and determining characteristics of faces such as if the face is “happy”, the person’s age, etc.

I will take us through a sample app I built that uses face recognition technology to help a hotel check in guests, which could be fully automated (kiosk, bot, robot etc) or help hotel staff through a guided workflow. After taking a photo, faces are scanned against past guest check-ins, and their details automatically retrieved to complete the check-in process. (Note, I got inspiration to build this after seeing a screen shot of this concept at the Microsoft’s Business Applications Summit, along with the Azure documentation.)

Below is a screen shot of the finished product, and what I did and how you can build it. I myself will play the role of a “guest” in this demo, checking into a Hotel.

1. Get a Key and Create a Group in Azure Cognitive Services

The Hotel happens to use Dynamics 365 / PowerApps as the application to manage hotel reservations and guests. The hotel has signed up for a Cognitive Services key and has created an Azure Face group using the PersonGroup.CreateAsync method.

Here’s myself among a list of guests in the Dynamics 365 platform that we will be comparing faces against later:

And here is my guest profile in Dynamics 365:

2. Upload Faces and People from Hotel System to Cognitive Services

The next step is to use images in our Dynamics 365/PowerApps Hotel system and for each guest create a “person” in Azure. This can be done using PersonGroupPerson.CreateAsync, then use the AddFaceFromStreamAsync method to add a face to each person in the cloud.

3. Build Webcam App to Take Photos

Once the photos are in the cloud, Azure can verify people based on the new photo we provide when a guest checks in.

I used Microsoft’s Unified Service Desk app to integrate with my computer’s webcam and provide an automated workflow of the guest check-in experience. USD is typically used for call centers and works well here as a thick-client, extensible, multi-window app with Dynamics 365 routing.

On opening the app, we can see there is a “Webcam” control which allows a photo to be taken of a guest. Here I am checking in below:

Note: you will need to find a way to integrate with your specific webcam (there’s plenty of libraries out there).

4. Detect Face of Guest

On taking the photo, we then upload the image to Azure and try to detect a “face” in the image. You can see below, Azure has found my face within the image uploaded, and we draw a square around the detected face:

5. Verify If Guest Exists

Next, we compare the face identified in the image against faces of people in our contained Azure environment, and if there is a match, it returns the person found. This is based on a “confidence percentage”. It’s up to us what we consider a close match of a person identified. Below in the code, we’re giving this confidence a match of 95%. We use the VerifyFaceToPersonAsync method:

6. Open Guest Record in Dynamics 365

As we get a match, we can automatically open the guest’s record. Below, you can see the app has found a match and my record is shown on the right. The hotel staff can be guided through a scripted check-in on the left, or we could perform an automated check-in:

In Unified Service Desk terms, to set this up we fire an event from our control that starts a session, using the Guest returned from Cognitive Services to look up data within Dynamics 365.

7. Identify Emotions

Azure Face API can tell us several things such as emotions (including happiness and sadness), age, facial features etc. In the scenario where there is no hotel staff, e.g. an automated check-in, we may want to use the emotions of the guest to interact with the user. For example, if a user has negative emotions, you can offer them something to make them happier. Read more about using emotions here.

Summary

As you can see, it’s relatively straight forward to build apps using Azure’s Cognitive Services Face API technology. Using pieces from this demo you can build pretty interesting applications. The great thing is the hard work has been done; you simply subscribe to the service and without having to build machine learning models with data sets.

In the real world, you may want to use 2-factor authentication to identify customers as AI capabilities become more refined. AI is really exciting, and this of course comes with a disclaimer of ensuring you have permission to use your customers’ faces and address any privacy concerns. For example, if this is done at a kiosk, have each user opt in before taking their photo. You may need permission to store their photos in Dynamics 365 too.

Will this type of technology become more widely adopted or not? If yes, how fast?

For more posts like these, check out my website at carldesouza.com, and also check out my step-by-step series on building face recognition apps with Microsoft Azure Cognitive Services using C#.

ABOUT CARL DE SOUZA

Carl de Souza is a developer and architect focusing on Microsoft Dynamics 365, Power BI, Azure, and AI.

carldesouza.com | LinkedIn | Twitter | YouTube

Facebook

Twitter

Linkedin