My submission for the Xamarin.Forms + Cognitive Service Challenge

It is Challenge time! Another Xamarin.Forms Challenge was announced on Monday April 20th. Developers where given a starter travelling app with some Cognitive Services functionalities which needed to get extended in some way. The ‘exact’ assignment was: “Extend this app with whatever functionalities you want. The crazier, the better. The world is your oyster, make it fun!”. Ok, let’s go!

In the week prior to the Challenge announcement, Gerald Versluis hosted 2 live coding sessions on his Twitch channel. In this live coding sessions he started developing the starter app for this Challenge. This way we could already get some intention in which direction this challenge was going. And Gerald also shared some new functionalities like Embedded Fonts, CarouselView and some more. On Monday the 20th Gerald announced the Challenge with an award worthy video. Now it was up to us to get to work and try to earn that Xamarin Swag! 😊

Pre-challenge goals

This is always the worst least nice part of creating/extending an app without any clear instructions: what shall I develop? Any tips from Gerald or Matt: The crazier, the better. The world is your oyster, make it fun… Ok so I can do whatever I want. Let’s think of some cool new Xamarin (or Azure) techniques that I wanted to try out for some time, but didn’t get the time for so far. So here’s the list of things I wanted to try out:

• Xamarin.Forms MediaElement: how easy is it to create a video background? And how well performant will this be?
• Xamarin.Plugin.SharedTransitions: Steven showed this nice image transition/animation (f.e. when navigating to a details page) on the .NET Conf: Focus on Xamarin event last month (read more on this on my previous blog post). How easy is this Plugin to use and in which scenario’s can you use this?
• Xamarin.Forms Shell: on every Xamarin event there is a lot of talk around this ‘new‘ Xamarin.Forms Shell thing. How easy can we convert an existing app to a Xamarin.Forms Shell app? And what are the benefits of Xamarin.Forms Shell?
• Azure Cognitive Services Face-service: how well can Azure detect your emotion in an image? And what else can Azure detect from a picture of your face?

App idea/story

With all these new techniques, we need to blend this into a proper app idea. What if we can create a list of desired destinations. We get some inspiration/idea’s/imaginary of that destination and we can check via a selfie which destination make us happy the most? This way we use the Face-service. We can re-use the Bing Image Search Service. We can also add an image/transition to the details page. Now only a video background on the main page and convert the app to Xamarin.Forms Shell and we have checked all our pre-challenge goals! Ok that sounds like a plan 😊.

Implementation

First thing is to be able to create a destination and let the Bing Image Search Service find 3 images of the given destination. With a simple PromptDialog we can enter our destination and with Akavache we can store our destination to make it more persistent. We can re-use the image carousel from the main page to show the image result from Bing on a newly created details page. Nothing new here yet, but some movement of existing code and using Akavache, which feels like my second nature.

Xamarin.Forms MediaElement

Now let’s use the new Xamarin.Forms MediaElement to add a video background on our main page and replace the existing image carousel. First thing is to find a proper, and more important a license free, video file. I’ve found some options, first thing to add these files to my Android project. But where do I add these? Luckily there is a sample app for this (which can be found here)! 😊 For Android add the files to Resources/raw folder and set Build Action to AndroidResource. For iOS add the files to the Resources folder and set the Build Action to BundleResource.

Don’t forget to add the MediaElement_Experimental flag, which now can be added in your shared code via Xamarin.Forms.Device.SetFlags(new string[] { "MediaElement_Experimental" }); in our App.xaml.cs, instead of on each Platform just before the Xamarin.Forms.Forms.Init(); call.

Since it is a background video we need to be able to play this video directly, let it loop once finished and don’t show any playback buttons or whatever. Luckily for me there where properties for all this requirements: AutoPlay="True" IsLooping="True" ShowsPlaybackControls="False"

Then on to the Source property. We’ve added the video file embedded to our platform projects, but here are 3 options to play local video files:

• Embedded resource in our platform project using the ms-appx:/// URI scheme.
• Files from our local/temporary data folders using the ms-appdata:/// URI scheme.
• From the device library by using a FileStream.

Xamarin.Plugin.SharedTransitions

Then on to the image transition from the main page to our details (destination) page by using the Xamarin.Plugin.SharedTransitions package. This is what I was most exited about to try out since Steven Thewissen has mentioned this in his talk on .NET Conf: Focus on Xamarin. Which you can read more about on my previous blog post. First we need to replace the NavigationPage that is wrapped around our MainPage for the SharedTransitionNavigationPage. On the page where we are going to navigate from we only need to define a Transtion.Name. Since we have our destination image in a CarouselView (because we can have more than 1 destination added), we also need to give this image an unique identifier by adding a Transition.Group property. On the page where we are going to navigate to we also give the same image the same Transition.Name to make the connection. In the image tapped event we now only have to set the TransitionSelectedGroup to let the Plugin now which transition group is going to transition during navigation.

SharedTransitionNavigationPage.SetTransitionSelectedGroup(this, destination.Id.ToString());

Azure Cognitive Services Face-service

Next up is the Azure Cognitive Service Face-service to detect the emotion on a face image. There is a NuGet package for this, but remember to check the ‘include prerelease’ checkbox on the NuGet Package Manager in Visual Studio, because it is still in preview. At first I couldn’t find the NuGet package and this was why. More documentation about everything you need to now about the Face service can be found here.

First we need to create a Cognitive Service Face resource inside Azure and retrieve the FaceApiKey and FaceEndpoint url. Now we can create a FaceClient:

private readonly IFaceClient _faceClient = new FaceClient(new ApiKeyServiceClientCredentials(ApiKeys.FaceApiKey))
{
Endpoint = ApiKeys.FaceEndpoint
};

There are 2 options to detect or analyze a face in an image. By sending an image stream or by sending an url to an image. In our case we use the DetectWithStreamAsync(); method. Other parameters that we are going to use are the returnFaceAttributes and the recognitionModel. In the returnFaceAttributes we ask to detect the image/face on FaceAttributeType.Emotion and we need to send RecognitionModel.Recognition02 as the recognitionModel. There are 2 recognition models, number one is for detecting faces in an image and the second one is for analyzing face attributes in a face. That’s all there is to it!

Xamarin.Forms Shell

Unfortunately I didn’t got to this part. But in the near future I will definitely develop an app with Xamarin.Forms Shell to try it out and see what benefits it gives us Xamarin developer. Will get back on this and let you know my findings in a future blog with a sample app project included. To be continued…

Findings

During the development of this Challenge I’ve also faced some issue or had to do some troubleshooting to get stuff working. Here is a short list of the major findings I ran into.

iOS Statusbar transparency

The initial design looks really good, especially on iOS where the image carousel is drawn behind the statusbar, which gives the app a nice full screen experience (no edges). After adding a details page, I had to wrap the main page inside a NavigationPage. Now the main page was showing a navigation bar and the nice full screen effect was gone. Luckily we can easily hide the navigation bar with the following property: NavigationPage.HasNavigationBar="False". This caused below effect. On the main page we see the status bar in white, but on the details page it looks as we want it to. Another strange thing, when we make a Xaml change on the main page, causing a Xaml Hot Reload, made the main page look as we want it to be. Really weird and I couldn’t figure out how this is possible and how to fix this.

Xaml Hot Reload

First of all, I LOVE Xaml Hot Reload. Can’t live without it and also can’t imagine anymore how I’ve ever developed Xamarin.Forms apps without this feature! And it also almost never fails on me. Until now. While trying out some video files with the Xamarin.Forms MediaElement I’d added several video files to my Android project (approx. up to 300mB of file size). My idea was to add these video files and change the Xaml to check which video looked best in this app. But after saving, the app didn’t reload at all. My first thought was that I’ve added to much/large video files which the Xaml Hot Reload couldn’t handle properly. But in the end I tried doing the same thing again and it worked… No clue why.

While developing the details page the Xaml Hot Reload gave me the error/warning ‘1 unsupported edits’ multiple times. But after stopping the debug session, rebuilding the app and started debugging again the app worked properly. So the edit was only unsupported during the Xaml Hot Reload, because after rebuilding the change did work?! In the output I couldn’t find much more information than this:

Xaml Hot Reload no longer working. No specific error details:
[HotReload] (2020-04-26 20:50:11.9): INFO: Reloaded: 'ReloadChange in Views/DestinationPage.xaml', ignoring 1 unsupported edits:
[HotReload] (2020-04-26 20:50:11.9): INFO: In Views/DestinationPage.xaml:
[HotReload] (2020-04-26 20:50:11.9): INFO: No line info - Invalid target position

During my app development for all my clients I haven’t encountered this severe errors/problems with Xaml Hot Reload so I have no clue on why in this app I’ve encountered errors this much. Maybe the design or all the used Plugins (Xamarin.Plugin.SharedTransitions and Xamarin.Forms.PancakeView) together with the Xamarin.Forms MediaElement are not that Xaml Hot Reload friendly? Although I can’t imagine it to be so…

Xamarin.Forms MediaElement

No support for .mov file types! But luckily I could convert these video’s to .mp4 which is supported. Other than that the MediaElement worked perfectly for showing a background video loop. At first I was curious if the Main/UI thread could handle a video loop well enough to not make the app react slower or stutter. But that was not the case at all! Next time I’ll have to dive more in to the playback buttons and handling the playback by user input.

Xamarin.Plugin.SharedTransitions

In my app a destination has 3 images. On the main page the first image is shown and on the details/destination page all 3 images are shown in a CarouselView. The transition from the main to the details page doesn’t work if the image is inside a CarouselView. The transition from the details page back to the main page does work. But if the first image in the CarouselView isn’t shown or active anymore, the transitions looks weird. So the destination/details page ideally needs to show the same image without a CarouselView to transition nicely. After removing the CarouselView and only showing the first image, the transition looked superb!

But then I had a look at the iOS app and it crashed when navigating back to the main page with the image transition. After searching for known issues on GitHub if found this issue. Due to the time frame I wasn’t able to fix this. When I have some time left, I will try to dig in to this a bit deeper and try to solve this. And of course share this with you when I’ve found a fix for this. 😊

Azure Cognitive Services Face-service

It was quite easy to get this to work. A lot of documentation is around and it just works. The Azure resources creation was super easy and also the NuGet package makes it super easy to use. I was amazed by all the detection/analysis possibilities with this Face-service!

The final result

And here is the final result! In the end I’m proud of the result and I’ve learned a lot during this challenge. And more importantly, I had fun doing this! 😊 Can’t wait for the next Xamarin.Forms challenge to participate on.

Of course the code is available for you on my GitHub: https://github.com/jBijsterboschNL/TravelMonkey.

Let me know what you think of the end result and my step-to-step implementation description and findings. You can reach out to me on LinkedIn, on the Xamarin Slack channel or simply by sending me an e-mail. I would love to here from you! Also if you are struggling with some Xamarin.Forms stuff, feel free to contact me.