Close
0%
0%

Watson Speech AI meets Unity 3D ScriptableObject

Connecting IBM Watson AI Speech-Text and Tonal Analysis web services with Unity3d ScriptableObject Data Architecture

Public Chat
Similar projects worth following
725 views
0 followers
IBM released an SDK to integrate several Watson AI web services (notably Speech-Text and Tone Analysis) with the Unity3D game/world building engine. Unity's ScriptableObject is emerging as a more effective technique for world building than pure MonoBehaviors. This project explores how to integrate the Watson API in a more generic, pluggable world builder experience using Unity's ScriptableObject.

[on-going project see my gitRepo for latest code. updated June2018]

0 Introduction

In January 2018, IBM and Unity announced an SDK tying IBM's Watson AI service in to Unity3D.  While the SDK provides basic interface, I was also introduced to Unity's ScriptableObject as clean way to do application architecture.  I very much like the Composition and Delegate approach Unity has vs more conventional OOP Inheritance, etc. It fits very well with my older approaches to Virtual World creation. 

This project explores the intersection of IBM's Web Services, Unity and Composition models.

1 IBM Watson Unity SDK

While announced in Jan 2018 , the Github for the Watson SDK dates back into 2016, so they were working on it for a while.  I was asked to use the SDK to integrate Watson into a team project for the  Create Reality Hackathon Hackathon in March.   In such pressure situations (also often in professional coding) we start with whatever demos are available and stand on the shoulders of army of midgets.  Alas, there did not seem to be many complete demos available.  The SDK Examples are limited, apparently, to Unit Test scenes and dont show much approaching real use. Fortunately,  Ryan Anderson (aka rustyoldrake on GitHub) in collaboration with others in IBM,  put out some  YouTube Videos (emotional Avatar :

and Droid) 

along with a loose collection of example code on github.  His videos demonstrate voice commands of the Speech-Text service as well as connections between the Tone Analysis Service and both UI elements and active objects in the scene.  While Ryan does not provide a complete Unity example project, I was able to hack up an interface for our hackathon team. It was not very clean, and required deep knowledge of the use of data to dispatch updates, but it worked for that venue. 

2 Unity ScriptableObject

I was introduced about this time to the Unity Scriptable Object  (abbreviated SO herein) as a rather powerful Data Container (etc).  This looked like a very useful way to abstract the data from Watson and make it available where ever the designers (vs programmers) wanted to use it.  Ryan Hipple gave a very good, highly professional talk on Scriptable Objects at Unite Austin 2017:

Ryan's code is available in a githup repository.  His blog post gives some hints on the complexity of the SO usage at Schell Games that is the basis of his examples. It would be VERY interesting for someone to publish a more robust Variable, Reference, Event system built on ScriptableObjects.

Ryan referenced a 2016 talk by Richard Fine on "Overthrowing the MonoBehaviour Tyranny in a Glorious Scriptable Object Revolution" (love the title): 

Richard also shared his code on a BitBucket Repository

I've also been following Jason Weimann's Unity3d.college video tutorials.  He recreated some of Ryan's work in slightly different examples...

With these as motivation, I embarked on a Watson/Unity example using the ScriptableObject Data Architecture.

3 My Explorations

[this section is also exploring how to use Hackaday to document a project in process. ...

Read more »

  • April 26: UI common prefab for TA Variables

    Jerry Isdale04/27/2018 at 04:06 0 comments

    Took the Joy element from last revision and tweaked it a bit, then saved back to prefab and repositioned others.

    New style uses the Panel's Text element to display a label, and a UI.Text element to display the value within the Filled element.  New script supersedes both Hipple ImageVar slider and the previous SOUI element.

  • April 26: Working UI components Connected to live IBM Service

    Jerry Isdale04/27/2018 at 00:20 0 comments

    Filled in Credential details, bolted up the Responders and some UI components to show FloatVariables returned by ToneAnalysis

    Generally the UI element is Lable and one of Hipple's bar graphs.  The Joy element also adds a Value display in Text field.  I like this and should examine in detail for use as prefab with one script filling in the Name, Text Value, Graphic components.

    The Credentials work well.  I definitely prefer the TA approach using string properties over the other that uses StringVariables.  It keeps them close in one display, vs inspecting several components. 

  • TestSO Scene 1 Created

    Jerry Isdale04/24/2018 at 22:39 0 comments

    GitHub revised (renamed).  Created a first Test Scene with WatsonService and Text/Image UI components for the recognized text and basic tone analysis (joy, sadness, anger, fear, disgust) using image bars.

    Service should put the data into SOs when arrives. Relies on the UI component Update functions to fill in the values.  Might be better to use Event callbacks but not yet implemented.

  • First SO: WatsonCredentials (username, password, url)

    Jerry Isdale04/19/2018 at 07:12 0 comments

    Project in UnityCollab with mashup import of my older, IBM cs code, Hipple SO github. Then started playing with how to merge in ScriptableObject...

    First step is noticing that the IBM Credentials (username, password, url) are perfect candidates for ScriptableObject.  So I created two tests

    one using strings, one using StringVariables.  

    To the client code (WatsonService) the SO and SO_Simple are identical.  Using StringVariables takes more steps in Editor setting up SOs etc.  May be easier to do it _Simple way.  There are fewer steps and Assets created, unless you want to compose at that granularity.

    I’ll leave both techniques in the code to show alternatives and how it affects the project.

    Here is the basic code for now…

    public class WatsonCredentialsSO_Simple : ScriptableObject 
{
    [SerializeField]
    private string _username ;
    [SerializeField]
    private string _password ;
    [SerializeField]
    private string _url ;

    public string Username
    {
        get { return _username; }
        set { _username = value; }
    }
    public string Password
    {
        get { return _password; }
        set { _password = value; }
    }
    public string URL
    {
        get { return _url; }
        set { _url.Value = value; }
    }

}

    [CreateAssetMenu]
public class WatsonCredentialsSO : ScriptableObject 
{
    [SerializeField]
    private StringVariable _username ;
    [SerializeField]
    private StringVariable _password ;
    [SerializeField]
    private StringVariable _url ;

    public string Username
    {
        get { return _username.Value; }
        set { _username.Value = value; }
    }
    public string Password
    {
        get { return _password.Value; }
        set { _password.Value = value; }
    }
    public string URL
    {
        get { return _url.Value; }
        set { _url.Value = value; }
    }

}


    ublic class WatsonSTT_TAService : MonoBehaviour 
{

    [SerializeField]
    WatsonCredentialsSO stt_credentialSO;
    [SerializeField]
    WatsonCredentialsSO_Simple ta_credentialSO;

    private Credentials credentials_STT;
    private Credentials credentials_TA;

    void Awake (){
        print("Speech2Text user: " + stt_credentialSO.Username);
        print("Speech2Text  pwd: " + stt_credentialSO.Password);
        print("Speech2Text  url: " + stt_credentialSO.URL);

        print("ToneAnalysis user: " + ta_credentialSO.Username);
        print("ToneAnalysis  pwd: " + ta_credentialSO.Password);
        print("ToneAnalysis  url: " + ta_credentialSO.URL);

        credentials_STT = new Credentials(
            stt_credentialSO.Username,
            stt_credentialSO.Password,
            stt_credentialSO.URL);
        
        credentials_TA = new Credentials(
            ta_credentialSO.Username,
            ta_credentialSO.Password,
            ta_credentialSO.URL);

    }
    
    // Update is called once per framevoid Update () {
        
    }
}


    oh my that doesnt paste in very well at all does it? 

  • 2018-04-15 First Log

    Jerry Isdale04/16/2018 at 01:26 0 comments

    First Log.  4-15-2018

    The GitHub is up to date with as-is code.  The Hackaday documentation is basic.  Now I start on the ScriptableObject rewrite of scenes, and classes.

View all 5 project logs

  • 1
    Get Unity

    You should have a recent (2017 or 2018 beta) version of Unity on your machine.  If not Get One

  • 2
    Get IBM Login

    If you dont have an IBM web service id... Get One

  • 3
    Setup Watson SDK in Unity

    Follow instructions on the GitHub Page

View all 4 instructions

Enjoy this project?

Share

Discussions

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates