Integrating Cortana with Unity (Unite 2015)

The Unite Europe session is 6/25 Track 4, at 12-12:30 in WesterLiefde
The Unite Europe session is 6/25 Track 4, at 12-12:30 in WesterLiefde

IMPORTANT: This demo shows Unity Windows 10 integration. We demonstrated and shared integration at Game Jam 10, if you attended that demo you can download Unity Windows 10 beta integration setup. This example requires Unity Windows 10 integration beta.

This is the sample I’ll be presenting at Unite Europe 2015 event 6/25 (see Track 4) to demonstrate how to integrate Unity with Cortana in Windows 10, among other cool things. Fortunately, it’s very easy to do: if you are a Unity developer, integrating Unity with Cortana can certainly add flavor to your Windows version of the app. And for users it just means more fun! So let’s get started…

The Idea: Mecanim and Cortana

I’ve been thinking of combining Unity mecanim animation with Cortana to make my animated character react to human voice commands. Cortana, besides being a personal assistant and just a friendly artificial intelligence inside my Windows devices can also help with basic speech things: parsing human voice commands and as it turns out, with Unity games. Here’s what my app looks like:

Cortana integrated app built with Unity
Cortana integrated app built with Unity

I’m using a free asset from Mixamo for my animation character (in fact you can use any character with animations from Unity Asset Store). I want to activate Cortana with standard Windows “Hey, Cortana” and then make my character dance, walk, run etc, by speaking a voice command. Easy?

Dancing avatar
Dancing avatar

You can find the code and also instructions for this example at the bottom of this post. Instructions:

1. After loading the Unity project, open the MainScene scene.

2. Now let’s build a Windows specific project. Open Build Settings. Since we will use Cortana in our project, click Player Settings and make sure Microphone is checked.

3. Now, select Windows Store output, then choose Universal 10 as a target. If you do not see this target, then you don’t have a Windows 10 integrated Unity. Make sure you select XAML as an output project type.

4. Click Build…, Unity will prompt you for output folder, this is where a native Visual Studio project will be generated.

5. Make sure to make the following changes to App.xaml.cs

Make AppCallbacks public instead of private:

public AppCallbacks appCallbacks;

Change OnLaunched event as follows:

protectedasyncoverridevoid OnLaunched(LaunchActivatedEventArgs args)

{

splashScreen = args.SplashScreen;

try

{

StorageFile vcdStorageFile = awaitPackage.Current.InstalledLocation.GetFileAsync(@”CortanaCommands.xml”);

await Windows.ApplicationModel.VoiceCommands.VoiceCommandDefinitionManager.InstallCommandDefinitionsFromStorageFileAsync(vcdStorageFile);

awaitSpeechHelper.InitialiseSpeechRecognition();

}

catch (Exception ex)

{

System.Diagnostics.Debug.WriteLine(“Installing Voice Commands Failed: “ + ex.ToString());

}

InitializeUnity(args.Arguments);

}

Then, update OnActivated event to include voice command activation:

protectedoverridevoid OnActivated(IActivatedEventArgs args)

{

string appArgs = “”;

switch (args.Kind)

{

caseActivationKind.Protocol:

ProtocolActivatedEventArgs eventArgs = args asProtocolActivatedEventArgs;

splashScreen = eventArgs.SplashScreen;

appArgs += string.Format(“Uri={0}”, eventArgs.Uri.AbsoluteUri);

break;

//Add VoiceCommand start detection and use the SpeechHelper handler

caseActivationKind.VoiceCommand:

SpeechHelper.HandleSpeechCommand(args);

break;

}

InitializeUnity(appArgs);

}

6. Now, drop CortanaCommands.xml, SpeechHelper.cs, ToastNotificationHelper.cs from WindowsSpecificCode folder to your project, build and run it. For your convenience I also added App.xaml.cs file.

7. When the project starts, you can use Windows 10 Cortana to give it commands such as “Avatar dance” or “Avatar run”.

Download code

 

 

 

 

 

 

Wearables: building cross-platform wearable apps with Visual Studio and C# (Part 2)

Please see: Creating beautiful cross-platform apps for iOS, Android and Windows with Visual Studio. Part I. User Interface

(this is a multi-part series on mobile cross-platform technologies available with Visual Studio)

Microsoft Bing predicts wearables to be the hottest trend in 2015. You can check Bing Predictions technology trends right here: http://www.bing.com/trends/us/predictions/technology. According to these predictions, wearables in 2015 are even more important than personal digital assistants, home automation, 3D printing and virtual reality gaming.  This post continues the series on wonders you can do with cross-platform C# and Visual Studio tools on all kinds of devices: Android, iOS, Windows and now getting to even smaller form factors.

image16

So what are the wearables today: at least devices that are available to developers with existing or upcoming SDKs and APIs? Wearables are definitely in the IoT category: smartwatches, bands, things that we wear on our body to take body-specific measurements. The evolution of watches started a very long time ago, literally centuries ago, with the watch becoming more of a fashion item. Later, fitness bands created a new category of smart wearable devices.

image_thumb4

Let’s compare some wearable devices available today. If we attempt to classify them, they differ in sensors, functionality, price and purpose. In 2015 we will get an idea of market share each of them have, but the most interesting point: devices you see below can be accessed or programmed with C# and Visual Studio!

image_thumb7

Wearables APIs: What’s in common

From the app prospective, since wearables are so lightweight as a platform, they commonly make use of several API tiers that are more specific to small devices:

  • Actionable Notifications: wearables make an extensive use of notifications, including actionable notifications, i.e. those that call back into the host device app, like a phone. You can think of a actionable notification as a message with multiple attached actions-buttons. Android Wear SDK gives a good example of these.
  • Glance, Watchface, Tiles etc – ways to present readable information on the screen. Android SDK has CanvasWatchFaceService for example, which allows developers drawing the face. Microsoft’s concept of tiles is perfect for wearables developers.
  • Apps (optional) – some wearables may or may not have deployable apps (depending on the device and platform) . With apps come specialized controls that can present information on the small screens.
  • History (cloud APIs) – typically wearables synchronize with the phone device and that device in tern synchronizes with the cloud
  • Form factors: Microsoft Band is of band shape, with natural sequence of tiles. Android wear is round and square. Apple watch has a rectangular shape.

    Visual Studio and C# story with Wearables

    With Visual Studio and C# you can develop cross platform wearable apps that run on Android Wear, Apple WatchKit, and we hope to see Band SDK coming out soon. This makes Visual Studio a unique Wearable development platform for all devices. C# is a fantastic language, perfectly suitable for mobile and wearable development. In many ways it is easier to make apps with C# than with the native languages in each platform.

    You can start with these guides:

    Android Wear Developer Preview Now Available

    Introducing the Xamarin WatchKit Preview

    Practical examples of cross-platform apps built with C#

    Here’s a few apps I built with Visual Studio that run on wearables or are capable of interacting with wearable devices.

    Active Fitness

    Active Fitness is a cross-platform (Windows, Android, iOS) fitness, health and sport GPS tracking app and social network of more than a million users. Great for running, walking, cycling, hiking, mountain biking, skiing, snowboarding, urban sports, such as skateboarding, scooter and hundreds of other activities. Active Fitness motivates you by providing ways for sharing your activities with your buddies and friends. Active Fitness has leaderboards for any sports and activities and includes professionally designed training plans. Enjoy and stay fit!

     

    activefitness-watch4

    Winter Sports

    Winter Sports is a cross-platform (Windows, Android, iOS) app, the official app of Professional Ski Instructors of America and American Association of Snowboard Instructors (PSIA-AASI). Beautiful GPS tracking, ski run analytics, snow reports for thousands of resorts, zoomable resort maps, activity tracking, customizable goals, a social network, leaderboards and ski lessons by snow pros. The app includes many innovations, including voice guidance.

     

    ski-snowboard-watch4

  • Creating beautiful cross-platform apps for iOS, Android and Windows with Visual Studio. Part I. User Interface

    Download free Android, iOS or Windows Phone apps used in this article and GET FIT!

    See also this article on my MSDN Blog

    Cross platform apps: dream becomes reality

    With recent additions to Visual Studio 2015, it becomes possible to develop apps for all 3 major platforms: iOS, Android, Windows, and also target desktop Windows and OSX. Clearly, the world is changing and I’d like to share my experience in developing rich cross-platform apps that look great and work on iPhone, Android and Windows. Yes, you heard it right, it is now possible to create rich code that compiles to native bits on each platform, with only one code base (or one developer team, with 80-90% code reuse!), all with Visual Studio, C# and .NET. This is very powerful, and clearly developers benefit from this approach by Microsoft: it greatly increases productivity even for small app development teams and allows developers to create native apps that work across all major platforms.

    I’ll share more of my experience developing Active Fitness app (http://activefitness.co) for Android, iOS and Windows. You can download the app and see for yourself by clicking the links above and downloading the app for the device of your choice. I want to share the experience of developing these apps, because I feel that it’s a revolution or rather a natural evolutionary step in app development that changes our perception of 3 operating systems incompatibility, and instead focuses on making developers productive.

    What’s especially important, Microsoft provides Azure Mobile Services cross-platform libraries for Android, iOS, Windows Phone, Windows and Xamarin, so that developers of cross-platform apps can use cloud resources efficiently.

    Designing cross-platform user interface

    Mobile user interfaces have changed dramatically since the first iPhone. Windows has Modern Design and Google has Material Design, while Apple made significant changes to its user interface in iOS 8. What is common? Designers call it dropping skeuomorphism, or using material world objects in user interface graphics, and using clean functional design paradigms instead. I like calling a cross-platform combination of Modern Design, Material Design and Apple Design guidelines a pragmatic design, but I don’t pretend that I own the name.

    image
    From the very beginning I decided to create a common interface design that would satisfy design guidelines of Apple, Google and Microsoft. With Xamarin Forms, a recent addition to Xamarin cross platform suite, available now directly from Visual Studio, it becomes possible, with code re-use close to 80-90%, including user interface! This is a stunning achievement, given that most shops that make apps these days have very experienced developers that know iOS very well, then they need to hire developers that know Android and Windows. The cross-platform native approach taken by Xamarin Forms and other cross-platform tools changes the way developers make apps!

    The three screenshots here are from (left to right) Android, iOS and Windows Phone Active Fitness app. They are developed with the same code base, with 100% C# and XAML and naturally fit Apple, Google and Windows ecosystems. The main approach was: keep it simple: keep it beautiful! Modern lines in this design naturally blend with Google Material Design, Windows Modern Design and Apple design, because this design is clean and functional.

    Xamarin Forms introduced a set of controls: Pages, Views, Layouts and Cells that blend naturally with all three platforms: Android, iOS and Windows and provide a natural user experience. What’s even more important: one developer or a very small team of developers can now develop for all 3 major platforms (plus OSX and Windows Desktop), with one code, and one language! You can look at Xamarin Forms Gallery to get a feel of what cross-platform controls are available.

    Pages

    Pages are very natural part of Web sites, as well as native apps in iOS, Android and Windows. Xamarin Forms offer a wide selection of Pages.

    Active Fitness app takes advantage of MasterDetailPage which is very popular with Android and iPhone apps. The action bar is also very similar in Android and iPhone apps. In Windows Phone we’re more used to bottom application bar. The “sandwich” icon is becoming very popular to present a “drawer” navigation pattern.  In Xamarin Forms you can use Master Detail page to take advantage of this design. If you look at some of the latest designs for Windows Phone: Facebook app, or OneDrive app implement drawer design pattern, which is also very common in Android and iOS.

     

    clip_image008

    Layouts

    Layouts are ways to align and position controls on your user interface. Most common layouts are grids, stacks, relative or absolute layouts. It’s interesting that while Xamarin Forms use XAML as a metadata language, most layouts and controls are actually very similar to Android AXML objects. Android developers building cross-platform apps find themselves right at home by using Xamarin Forms!

     

    Views

    Views are often called UserControls in Windows apps. With the cross-platform nature of Xamarin Forms, there’re many views: for labels, entry controls, buttons, date and time pickers, progress controls, and they all look native on the platform you compile the app for:

    Cells

    Xamarin Forms also provide cells. Cells are specialized elements used for items in tables.

    Summary

    The cross-platform apps in this example are developed in XAML and C# with .NET that natively runs in iOS, Android and of course Windows. .NET has a long history of open source development, and it now becomes official with Microsoft integrating cross-platform tools with Visual Studio, open sourcing .NET. Many popular 3D games also run on multiple platforms backed by .NET and C#, with Unity. Game designs are very different from other apps, so we’ll focus on Xamarin Froms as technology suitable for cross-platform apps. Interested in this post?

    Please, let me know by sending a cheerful COMMENT or twit (@kashleytwit) and I’ll keep this series to tell you more about making beautiful cross-platform apps for Windows, Android and iOS. In the following posts I’ll show you how to make cross-platform apps work with speech, voice, storage, GPS, sensors and more!

    Download free Android, iOS or Windows Phone apps used in this article and get fit!

    Winter Sports a showcase cross-platform .NET app available for iPhone users

    Winter Sports, a comprehensive app for skiers and snowboarders is released for iPhone, powered by Microsoft cloud.

    https://itunes.apple.com/us/app/winter-sports-tracker/id927727475?ls=1&mt=8

    Winter Sports is an official app of more than 32,000 Professional Ski Instructors of America and American Association of Snowboard Instructors (PSIA-AASI), an educational organization teaching seven million ski days per season in the US alone. This cross-platform sophisticated app includes ski and snowboard-specific GPS tracker, snow reports and resorts maps from 2000+ ski resorts worldwide, as well as training, workouts and content for skiers, snowboarders, mountaineering enthusiasts and snow junkies.

    FEATURES

    GPS Tracker
    Snow Reports and Resort Maps
    Step Counter
    Friends and Social
    Training and Workouts
    Action Photos
    Ski and Snowboard Analytics
    Worldwide leaderboards
    Powered by Professional Ski and Snowboard Instructors of America

    More about the app…

    Cross-platform apps are becoming very popular with the users, who want to keep their data in sync on iPhone, Android and Windows. Also in line with Microsoft open source . NET Winter Sports became very popular on Windows Phone, grew to millions of users, and now reaches iPhone and Android markets. This app can easily be a showcase of Microsoft’s recent announcement for open-source .NET: a sophisticated native app entirely built with .NET and Xamarin.

    Ultimately, with cross-platform apps, such as Winter Sports, Microsoft wins because the app is using Microsoft’s massive cloud technology called Azure, users win because they can use their favorite mobile device with the app, and developers win because the app works on all major markets: iPhone, Android and Windows. Developers also win, because they only develop one source code, with close to 95% code reuse for Windows, iOS and Android, according to Winter Sports developers.

    Web Site: http://winter-sports.co

    Using Hyper-V and Visual Studio with Android x86 KitKat v4.4

    Androidx86 is a project hosted on http://www.android-x86.org/. The purpose of the project is porting Android open source project to x86 platform, formerly known as “patch hosting for android x86 support“. The original plan is to host different patches for android x86 support from open source community.

    Developers can actually use Hyper-V hosted Android x86 for testing and app development. Hyper-V is built into Windows 8.x and works much faster than Android emulators. In this walkthrough which requires about 10-15 minutes we’ll set up Android x86 in Hyper-V and deploy a Visual Studio Xamarin app to the virtual machine.

    Overview

    1. Deploy Android x86 in Hyper-V (3 minutes)
    2. Deploy your APK from ADB or Visual Studio

    Setting up Android x86 on HyperV

    Installing Android x86 on Hyper-V is very easy: you create a VM, download Android x86 ISO from http://www.android-x86.org/download and install Android in your VM, like a regular Linux distribution. I used the latest KitKat version: android-x86-4.4-r1-iso Below are step-by-step instructions.

    clip_image002

    1. Start Hyper-V manager (in Windows 8.x Window-Q Hyper V), then create a new VM in Hyper-V Manager (New / Virtual Machine…)
    clip_image004

    2. Download Android x86 ISO from http://www.android-x86.org/download

    3. Attach ISO to your VM (Media/DVD Drive/Insert Disk…) and select Android x86 ISO

    4. Set up Android as you would a Linux distribution: it’ll prompt you to create and format a partition, install GRUB boot loader etc.

    clip_image006

    clip_image008

    clip_image010

    clip_image012

    5. After the installation, Android x86 runs in your VM. You can now eject the ISO and let the Android boot.

    Developing apps using Android x86 in Hyper-V

    Android x86 running in Hyper-V is no different from any other Android device or emulator for ADB or Visual Studio. First we need to unlock Android x86 instance for development, then connect our Windows PC to it.

    1. In Android x86 press Alt-F1 for command prompt, then type netcfg, which will give you an IP, e.g. 192.168.1.10, press Alt-F7 to go back to graphics shell

    2. In Windows PC start ADB command prompt (provided with Android SDK) and type:

    adb connect 192.168.1.10

    3. If you use Visual Studio for development, for example for Xamarin apps, you can now select Android x86 virtual machine running in Hyper-V

    clip_image013

    4. Now you can deploy your APK from any file with adb install… or from Visual Studio directly to Android x86 emulator in Hyper-V.

    Push Notifications for Xamarin.iOS

    Configuring Notification Hubs for Xamarin.iOS apps

    Some user devices are iOS-based, some Android, some Windows Phone, and the list goes on. Each of the native platforms has its own push notification framework and backend service. Azure Mobile Services allows the developer to configure a single Notification Hub (based on Microsoft Azure Service Bus Notification Hubs) through which notifications may be sent. The notification goes to each user via the correct push service according to her device’s requirements. Broadcast to millions in consumer scenarios, to tagged groups, to individuals with the same ease.

    To use notifications hub from Azure Mobile Services with your iOS apps, you need to generate a certificate, register your iOS app for push notifications, save your certificate to a file and upload to Azure Mobile Services Management Portal, into your Notifications Hub. The whole process of generating a certificate, and configuring Azure Notification Hubs for use with iOS apps here, which is fairly extensive and has some screenshots.

    Code snippet: Xamarin.iOS app using Push Notification Hubs

    We use Xamarin Forms for best cross-platform portability: we can re-use the same UI and most of the middle tier in our Android project as well.

    1. In Xamarin Studio on your Mac, or Visual Studio on your PC, create a new Xamarin Forms project. We will need to use WindowsAzure.Messaging from GitHub. Compile WindowsAzure.Messaging.dll and reference it in your project. This assembly provides iOS bindings for Notification Hub:

    clip_image002

    2. For Xamarin.iOS project, we need to make some changes in the AppDelegate.cs file. First, start by adding:
    SBNotificationHub _hub;

    public const string _azureConnection = “<Azure connection string>”;

    public const string _notificationHubPath = “<Azure hub path>”;

    3. Next in FinishedLaunching method add this to register for remote notifications in your app (providing several types of notifications):

    UIApplication.SharedApplication.RegisterForRemoteNotificationTypes(

    UIRemoteNotificationType.Alert |

    UIRemoteNotificationType.Badge);

    4. Add the following methods to handle registration and notification processing.

    public override void RegisteredForRemoteNotifications(UIApplication application, NSData deviceToken)

    {

    _hub = new SBNotificationHub(_azureConnection, _notificationHubPath);

    _hub.UnregisterAllAsync (deviceToken, (error) => {

    if (error == null)

    {

    NSSet tags = null; // create tags if you want

    _hub.RegisterNativeAsync(deviceToken, tags, (errorCallback) =>

    {

    // handle errors here

    });

    }

    else

    {

    // handle errors here

    }

    });

    }

    public override void ReceivedRemoteNotification(UIApplication application, NSDictionary options)

    {

    if (null != options && options.ContainsKey(new NSString(“aps”)))

    {

    NSDictionary aps = options.ObjectForKey(new NSString(“aps”)) as NSDictionary;

    string alert = string.Empty;

    if (aps.ContainsKey(new NSString(“alert”)))

    alert = (aps[new NSString(“alert”)] as NSString).ToString();

    if (!string.IsNullOrEmpty(alert))

    {

    UIAlertView avAlert = new UIAlertView(“Notification”, alert, null, “OK”, null);

    avAlert.Show();

    }

    }

    }

    We can now register for hub notifications and receive notifications, displaying alerts in our Xamarin.iOS apps.

    Reference:

    Get started with Notification Hubs with Xamarin.iOS
    Xamarin.iOS: Push Notifications in iOS