[on-going project see my gitRepo for latest code. updated June2018]

0 Introduction

In January 2018, IBM and Unity announced an SDK tying IBM's Watson AI service in to Unity3D. While the SDK provides basic interface, I was also introduced to Unity's ScriptableObject as clean way to do application architecture. I very much like the Composition and Delegate approach Unity has vs more conventional OOP Inheritance, etc. It fits very well with my older approaches to Virtual World creation.

This project explores the intersection of IBM's Web Services, Unity and Composition models.

1 IBM Watson Unity SDK

While announced in Jan 2018 , the Github for the Watson SDK dates back into 2016, so they were working on it for a while. I was asked to use the SDK to integrate Watson into a team project for the Create Reality Hackathon Hackathon in March. In such pressure situations (also often in professional coding) we start with whatever demos are available and stand on the shoulders of army of midgets. Alas, there did not seem to be many complete demos available. The SDK Examples are limited, apparently, to Unit Test scenes and dont show much approaching real use. Fortunately, Ryan Anderson (aka rustyoldrake on GitHub) in collaboration with others in IBM, put out some YouTube Videos (emotional Avatar :

and Droid)

along with a loose collection of example code on github. His videos demonstrate voice commands of the Speech-Text service as well as connections between the Tone Analysis Service and both UI elements and active objects in the scene. While Ryan does not provide a complete Unity example project, I was able to hack up an interface for our hackathon team. It was not very clean, and required deep knowledge of the use of data to dispatch updates, but it worked for that venue.

2 Unity ScriptableObject

I was introduced about this time to the Unity Scriptable Object (abbreviated SO herein) as a rather powerful Data Container (etc). This looked like a very useful way to abstract the data from Watson and make it available where ever the designers (vs programmers) wanted to use it. Ryan Hipple gave a very good, highly professional talk on Scriptable Objects at Unite Austin 2017:

Ryan's code is available in a githup repository. His blog post gives some hints on the complexity of the SO usage at Schell Games that is the basis of his examples. It would be VERY interesting for someone to publish a more robust Variable, Reference, Event system built on ScriptableObjects.

Ryan referenced a 2016 talk by Richard Fine on "Overthrowing the MonoBehaviour Tyranny in a Glorious Scriptable Object Revolution" (love the title):

Richard also shared his code on a BitBucket Repository

I've also been following Jason Weimann's Unity3d.college video tutorials. He recreated some of Ryan's work in slightly different examples...

With these as motivation, I embarked on a Watson/Unity example using the ScriptableObject Data Architecture.

3 My Explorations

[this section is also exploring how to use Hackaday to document a project in process. Does it work better AddingDetails, or AddingLogs? let me know! - ji]

3.1 Setting up Watson SDK for Unity

Details of setting up the SDK in Unity are described in the GitHub

"Registering for an IBM Cloud account is a necessary step to instantiate Watson services and access your service credentials. Account registration is free and prices vary per Watson service". ( from the Unity Asset Store page)

Once you get signed up, [Create Resource] takes you to large selection of IBM services. Scroll down to the Watson collection and sign yourself up for the speech/text and tone analysis "Credentials". For initial prototyping, I recommend the free, limited service. Limits vary depending on service, eg 2500 API calls or 100 minutes. Limited credentials reduce your exposure should someone else use them, should you expose them in Git etc. Be patient with the IBM site. It was rather slow when I used it, and gave a number of timeout errors. (your dashboard of API credentials should be at https://console.bluemix.net/dashboard/apps I had some difficulties navigating to it from account login, but that link seems to get directly there)

Credentials give you an encrypted user name and key. Record these, you will need them. They seem like a perfect place to start with SO, allowing one to update the data via the Unity Editor, rather than deep in C# code.

The Watson SDK itself has a few very simple Examples of connecting to the services via the SDK. They offer a lot of services and build on a fair bit of C# infrastructure, providing their own Log, Runnable, and other supporting API chunks. I am only going to work with at the SpeechToText and ToneAnalysis services. This does however expose us to a fair bit of those enabling APIs as well.

3.2 Hipple's ScriptableObjects

Download Ryan Hipple's examples from his GitHub. Pull them into your Watson project, or pull watson sdk into this examples.

He has a number of scenes to demonstrate topics discussed in the Austin 2017 talk. Review the talk, his blog and the examples. Of primary interest are the StringVariable, FloatVariable and FloatReference. GameEvent is also of interest, but not for the initial explorations.

3.3 Initial Explorations

My GitHub for this project is public, and updated as project progresses.

Part of my code spelunking has shown the data returned by Watson (as JSON serialization) can be much richer than the simple examples indicate. The SDK's WatsonResponse data model has a lot more parts than any of the documentation or blog posts on the service talks about. There are also a lot of MonoBehavior derivatives that could be remodeled on SO. Tempting target but for now I'm going to layer on top of the provided code.

One aspect of Unity scripting is the reliance (aka tyranny) of the MonoBehavior. Just about every script derives from this Unity Object, and expects to be part of a scene (with Transform, life cycle calls Awake, Start, Update, etc). It does NOT make sense, to me, that non-embodied things (eg. gameManager, internet service connections) need a Transform, or calls to Update() every frame. The ScriptableObject lives as an Asset, outside of a scene. You reference and compose it into scene's scripts in the Unity Editor. This makes much more sense to me, allowing editorial/game staff having responsibility rather than programmers.

A main aspect of MonoBehavior, is the whole Unity Script Life Cycle (Awake, Update, Render, Destroy, etc). Lots of call back OnXX functions are invoked automagically for you on every GameObject component in the scene hierarchy. ScriptableObjects fall outside of this Execution loop, and alternatives (aka hacks) are required if you want some sort of (ir)regular callbacks. It will be interesting to see how Unity3d develops SO more widely in their system.

The WatsonSDK provides an internal class Runnable, to create a List of functions (of a MonoBehavior derived class) that operate akin to Co-Routines (i think). The ExampleStreaming code uses these to register routines that interface with the WebServices. The documentation on Runnable and related UnityObjectUtil is negligible. The one comment in code is "Helper class for running co routines without having to inherit from MonoBehavior." It might be a useful method of activating SOs, but I'm not going spelunking down that rabbit hole any further.

For first iteration, I took the IBM ExampleStreaming.cs approach and look at just using SOs for the data. See Logs for some progress updates in this area.

4.0 First Iteration

The Demo is now (2018/04/26) functional using SO credentials and response data. I rely heavily on the Hipple code, especially his Variables. I experimented with two styles of SO for Credentials. Response from the SpeechToText is a StringVariable, and the Scene's UI provides a labeled area for it. The ToneAnalysis response is tied to 5 FloatVariables for the "emotional tone" parameters, with a Scene UI display that shows a label, value and slider bar for each variable. Other than the UI panel, the scene is empty.

4.1 WatsonServices GameObject and Script

The first thing to build is the WatsonServices GameObject. To this we attach a WatsonSTT_TAService C# script, which provides us with a number of SerializeField elements in the Inspector:

Each of these can be tied to a ScriptableObject that must be created in the Assets. Note that while the script provides slots for both Emotional and Sentence Category values from TA, we are only populating the Emotional ones at this point.

[June2018: see section 5 below for updated version that splits this class into 3 parts for better modularity]

This GameObject is what connects to the Watson Services and handles response. It uses the first two SO to create the connections. Responses to Service calls are handled by Runnable.Run() routines of the script. Basically they copy values to the related SO fields. There is no triggering of updates (ala events) in this iteration. Any element in the scene that relies on the Variables must handle visual etc updates as part of its own Update() methods.

The Awake() method converts the two SO Credentials into the internal IBM Credential objects. It also initializes the IBM LogSystem, which works akin to Unity's Debug (and print).

void Awake()
{
    // Watson logging initialization
    LogSystem.InstallDefaultReactors();

    print("Speech2Text user: " + stt_credentialSO.Username);
    print("Speech2Text  pwd: " + stt_credentialSO.Password);
    print("Speech2Text  url: " + stt_credentialSO.URL);

    print("ToneAnalysis user: " + ta_credentialSO.Username);
    print("ToneAnalysis  pwd: " + ta_credentialSO.Password);
    print("ToneAnalysis  url: " + ta_credentialSO.URL);

    credentials_STT = new Credentials(
            stt_credentialSO.Username,
            stt_credentialSO.Password,
            stt_credentialSO.URL);

    credentials_TONE = new Credentials(
            ta_credentialSO.Username,
            ta_credentialSO.Password,
            ta_credentialSO.URL);
}

The Start() method initializes the Service connections and fires off the StartRecording(), which uses Runnable.Run to setup RecordingHandler() for periodic execution. The RecordingHandler() method works with the Microphone/Audio utilities to capture a sound chunk and send it off to the SpeechToText Service. This is pretty much directly from the ExampleStreaming code and I did not dig deep into the hows/whys of audio sampling. Perhaps there are alternatives, but I'm keeping to simple for now.

The SpeechToText service triggers the OnRecognize() method when a provided block has been processed. (it may call it multiple times for same audio, until the Final property becomes true.) This method iterates thru the results provided in the SpeechRecognitionEvent, extracting the text to the SO StringVariable recognizedText, passing the text to the ToneAnalysis service, and possibly extracting commands. This is pretty much what the ExampleStreaming did without the SO access.

[June2018: a WatchedStringVariable is introduced to hold text returned by Watson, with notification given to watchers - ToneAnalysis and CommandParser. This separates the Class responsibilities better.]

The OnGetToneAnalyze() method is called when the ToneAnalysis service has a response to text sent to it. This method copies the relevant values to the appropriate FloatVariables. The updateSOValue() method does the actual copy, insuring a Variable is provided and exceeds the TA_EmotionThreshold.Value, if one is provided.

Thats it on the Service side. Data has been copied from the response to ScriptableObject FloatVariables. This script needs no more knowledge of what is done with that data elsewhere. It is the responsibility of elements using the data to insure they properly update.

An enhancement here would be to implement a GameEvent type callback that would be called when a change happens to a FloatVariable. This may or may not make world building easier. Perhaps later we can investigate this.

4.2 Scriptable Objects

There are four (five?) types of Scriptable Objects used in our example. We make liberal use of Hipple's StringVariable and FloatVariable. We also utilize Hipple's FloatReference in the UI but make not changes to it. Placing a [CreateAssetMenu] tag before a class definition will show the SO as an available Asset when invoking the CreateAsset menu. We need to create instances of several such Variables and fill in values:

There are two ScriptableObjects defined for Credentials, exploring some architectural variations.

4.2.1 WatsonCredentialsSO
The first implementation for Credentials is the WatsonCredentialsSO, whichbundles three StringVariable references into one object, providing Property accessors for their values. See the GitHub repo for code

The Inspector panel for an instance of the SO in the Assets has a field for Username, Password, URL, each of which is a StringVariable. Note that we have dragged the correct StringVariable instances to these fields.

To change any of these values, we must select the StringVariable instance and change its Value via the inspector...

I was not too happy with this approach. It requires creating four SO instances, three of which are functionally identical with any other StringVariable. Setting this up and maintaining it, while providing maximum Composition in the Editor, opens up too many possible error insertion points. (eg. assign wrong StringVariable, etc)

So I tried a second approach.

4.2.2 WatsonTACredentialSO
The WatsonTACredentialSO is almost identical to the WatsonCredentialsSO except it uses a standard C# string instead of a StringVariable for each of the properties. (See code in git repo) The use of either style of Credentials by the service code is identical. They both provide the same accessors. However, the Inspector panel for an instance of WatsonTACredentialSO is where you enter the string values, rather than digging deeper to an appropriate StringVariable. (note these are not correct or complete credentials, dont bother trying to steal em)

IMHO this encapsulates the information better with less clutter in the Assets/Variables (or wherever you hide the SO instances).

4.2.3 ToneAnalysis_ResponseSO

I briefly experimented with a composite SO for the ToneAnalysis that had FloatVariable slots for each of the items. However, I quickly realized that this would create a LOT of extra code with few positive benefits. The initial code is still in the git repo but is not used.

4.3 SO Clients in UI

The ScriptableObject data architecture allows us to separate the Service setting of values from any consideration of how they are utilized in the application. GameObjects that want to access the SO Values need a script that provides a [SerializeField] for each. Our demonstration has two types of SO clients - one for the ResponseText from the SpeechToText, and a UI widget that displays the Value of a FloatVariable... The widget actually uses one of Hipple's FloatReference SO to reference the FloatVariable, but allow the designer to quickly flip to a Constant value for testing, etc.

4.3.1 STT String Variable Display

Results from the StringToText (STT) service are displayed in a UI panel with a label and text box, using a small C# script called TextPanelFiller. Drop it on the panel GameObject, drop the SO and Text fields into proper places in the Inspector, and it is hooked up.

At runtime, every frame, the Update() method updates the Text field from the SO value. It is doubtful the response will change much within several (hundred) frames so this every-frame bit is waaay overkill and not recommended for production work. Next iteration we will investigate GameEvent model to see about changing this.

using UnityEngine;
using UnityEngine.UI;

// Ryan's ScriptableObject variables
using RoboRyanTron.Unite2017.Variables;

public class TextPanelFiller : MonoBehaviour
{
    [SerializeField]
    StringVariable soText;
    [SerializeField]
    Text displayText;

    /// <summary>
    /// Update displayText based on Value of StringVariable.
    /// might be better to use event based notification
    /// </summary>
    private void Update()
    {
        if (soText == null)
            displayText.text = "No SO StringVariable";
        else
            displayText.text = soText.Value;
    }

}

4.3.2 Float Variable Display in UI: UI Meter

For each of the Tone Analysis values, I wanted a simple display with a Label, a text Value area and a graphic representing the value.

I created a prefab UI Meter using two Text boxes and the bar meter borrowed from Hipple's ImageFillSetter Health Point meter. The C# script UI_FloatVarPanel provides the functionality. Note that it uses 3 FloatReference rather than FloatVariables for the Variable (to be displayed), Min and Max.

The Inspector tab will use the CustomPropertyDrawer "FloatReferenceDrawer" to render each FloatReference. It provides a popup menu to select either Constant or Variable. If you select Use Variable, you can drop a FloatVariable on it. If you select Use Constant, it will provide a place to enter that value.

Connect in the Fill (Image) and Value Text with reference to those GO in the hierarchy. The Update() method of UI_FloatVarPanel converts value of Variable FloatReference to text for the ValueText area in UI, getting either the Constant or the FloatVariable's value. The last bit of Update() sets the size of the Fill bar.

    void Update()
    {
        if (valueText == null)
        {
            Debug.LogError("Missing valueText for UISO_FloatVarPanel");
            return;
        }
        valueText.text = Variable.Value.ToString();

        if (Image == null)
        {
            Debug.LogError("Missing Image for ImageFillSetter");
            return;
        }
        Image.fillAmount = Mathf.Clamp01(
            Mathf.InverseLerp(Min, Max, Variable));
    }

4.4 First Iteration summary

And thats about it. The system works now. The Watson Service has no idea where its data is being used. The UI clients have no idea where the data comes from. There are no hard coded links deep in code. All the usage details are left to the world designer.

Tying some of the ToneAnalysis variables to parameters of GameObjects in the scene would necessitate some C# interfacing. An example of that would be nice for the next iteration perhaps.

5.0 Second Iteration: Split Services, Watched Variable

I did not like the original Watson Service as it put the Speech-Text, CommandParser and the ToneAnalysis into one big class. So I have split these into 3 parts and use a WatchedStringVariable to share data. This can allow us to change out any of the three with an equivalent class.

The new WatsonSpeechToText class works basically the same as previous class but once it has received the text back from Watson, it puts the text into a WatchedStringVariable and goes back to waiting on input.

The WatchedStringVariable is nearly identical to Hipple's StringVariable, but it adds a "void event Action OnChanged = delegate {};" and invokes this in the set method:

using System;
using UnityEngine;

[CreateAssetMenu]
public class WatchedStringVariable : ScriptableObject
{
    [SerializeField]
    private string value = "";

    public event Action OnChanged = delegate { };

    public string Value
    {
        get { return value; }
        set { 
            this.value = value;
            OnChanged();
        }
    }
}

Clients that want to be notified when a new string is received, need to have add their action function to the OnChanged function (in Awake()) and remove it OnDestroy. Here is the very simple Command Parser:

using System;
using UnityEngine;

public class CommandParser : MonoBehaviour 
{
    // Text to be analyized - from speech to text perhaps
    [SerializeField]
    WatchedStringVariable watchedTextToParse;

    void Awake () 
    {
        if (watchedTextToParse != null) 
            watchedTextToParse.OnChanged += ParseWatchedText;
    }

    private void OnDestroy()
    {
        if (watchedTextToParse != null) 
            watchedTextToParse.OnChanged -= ParseWatchedText;
    }

    void ParseWatchedText()
    {
        if (watchedTextToParse != null) 
            ParseText(watchedTextToParse.Value);
    }

    void ParseText(string text)
    {
        // we arent really doing anything yet
        if (text.Contains("reset"))
        {
            print("Command 'reset' recognized, and ignored");
        }
    }
}

Next iteration will create a more interesting scene and modify object properties based on the various Tone values.

Project Summary:

ScriptableObjects make very useful items for passing data from Watson Service to scene components. Some useful enhancements are indicated (eg. CommandParser, Variable update events). I look forward to applying this into a real application and exercising the Watson Services to a much greater degree.

Project Details

0 Introduction

1 IBM Watson Unity SDK

2 Unity ScriptableObject

3 My Explorations