Been a long time since I updated on here, but I realized soon after the last log post (Dec 2016) that my current way of mapping text to actions was in drastic need of an upgrade.

Around Christmas I was seriously investigating systems like Google Home and Amazon Echo. I was really interested in the programming options, which Google Home was just starting to publish. After watching some of their YouTube tutorials I tried my hand at API.AI. More research into similar systems finally lead me to an open source project Rasa NLU.

It didn't take long to figure out Rasa was exactly what I was looking for. I wanted the natural language processing of API.AI but wanted more flexibility in how the intent translated to a user action; and I really wanted it to run locally without a round trip to the internet. To say that ripping out the guts of the regex based text to actions system I had was hard is a severe understatement. It took me 3 months to even get a basic workflow of what I wanted through Rasa working. Even then that was for only a handful of intents - I'm still working on training the rest into the system.

Part of what made this so daunting is that I also realized during this process that the actions were not flexible enough. The modules I'd created served well to perform individual functions, or integrate with online services. What I made the mistake of doing is having them perform "double duty" doing advanced logic or even conversational aspects of the system. This made things really messy from a programming perspective. If one module wanted to ask for information from another module this was outright impossible as modules wouldn't return results directly, but route them through the events system (async only).

Take something as simple as "when does the sun set?". As written in the original framework the Time.Sunset function would have to figure this out. Seems easy right? But what if you ask "when does the sun set on Thursday?". Well now there is a date involved, the function needs to figure out what day we went and calculate that. You also need to account for the type of date given "Thursday" could be the same as "6/29/2017". Again, this all needs to be within that one function since it can't rely on another interpreter before the information gets passed (you could build a lib that functions across all modules but for different variables this would get out of control really fast).

Enter Groovy scripts. I choose Groovy as it integrated well with the Java backend that already existed. Instead of having the Time.Sunset do all sorts of logic, instead the action is handed off to a Groovy script, which has the ability to call other Stark functions as needed. This way the Time.Sunset function can do one simple thing, calc the sunset time based on a unix timestamp. This is all pretty abstract at this point, but suffice it to say this was a major overhaul of the existing system. Let's look at how this works in practice.

Input from user: "What time is the sunset on Thursday?"

Stark takes the text and passes it to the Rasa server which returns the following:

{
  "entities": [
    {
      "end": 20, 
      "entity": "sunrise_or_sunset", 
      "extractor": "ner_crf", 
      "start": 14, 
      "value": "sunset"
    }, 
    {
      "end": 32, 
      "entity": "time", 
      "extractor": "ner_duckling", 
      "start": 21, 
      "text": "on Thursday", 
      "value": {
        "grain": "day", 
        "others": [
          {
            "grain": "day", 
            "value": "2017-06-29T00:00:00.000-05:00"
          }, 
          {
            "grain": "day", 
            "value": "2017-07-06T00:00:00.000-05:00"
          }, 
          {
            "grain": "day", 
            "value": "2017-07-13T00:00:00.000-05:00"
          }
        ], 
        "value": "2017-07-06T00:00:00.000-05:00"
      }
    }
  ], 
  "intent": {
    "confidence": 0.6721804000375875, 
    "name": "check_sun"
  }
<span class="redactor-invisible-space">}</span>

This is based on training already done that identifies that phrase as belonging to the "check_sun" intent. This is an action that will return either the sun rise or sun set of a given time. The entities returned are the arguments to the action. In this case it's an identifier either "sunrise_or_sunset" based on what was given, and then the time. The time is parsed by the excellent Duckling library bundled with Rasa NLU.

One this is found the action (ie Groovy Script) for the "check_sun" intent is loaded. The Groovy script looks like this:

//2 arg
//sunrise_or_sunset = a string either "sunrise" or "sunset"
//time = optional, datetime to check

//result is always a string response to the user
def result = ""

def now = new Date()
def dateWanted = new Date()
if(args.containsKey('time'))
{
    dateWanted = Date.parse("yyyy-MM-dd'T'HH:mm:ss.SSSXXX",args.getArgString('time'))
}

//convert to milliseconds and divide to get seconds
def unixTime = dateWanted.getTime().intdiv(1000)
def sunTime = null

if(args.getArgString('sunrise_or_sunset') == 'sunrise')
{
    def sunriseTime = callModule("Time.Sunrise",[date:unixTime])    

    //convert back to milliseconds
    sunTime = new Date((long)sunriseTime.data.time * 1000)
}
else
{
    def sunsetTime = callModule("Time.Sunset",[date:unixTime])
    
    //convert back to milliseconds
    sunTime = new Date((long)sunsetTime.data.time * 1000)
}

isPast = " was "
if(now < sunTime)
{
    isPast = " is "
}

result = args.getArgString('sunrise_or_sunset') + ' on ' + sunTime.format('MM/dd/yy') + isPast +  'at ' + sunTime.format('hh:mm a')


//must return a string
return result

This isn't super pretty code, but you can see how it is easily customized. Each Groovy script will be presented with an "args" variable that includes any arguments passed. It will also be allowed to call out to any existing module on the system and get a JSON response, notice the callModule("Time.Sunset",[date:unixTime]) function call. This allows the modules to focus on more on presenting information, with the Groovy script responsible for presenting this to the user. The resulting string is passed back to the calling client (web, Android, etc) along with a generated wave file.

I'm still fine tuning this system but will have screenshots of the new editor and updates on some of the Trigger and Monitor functions associated with this as well. Training Rasa is still taking a fair amount of time as I had lots of custom commands already completed in the old system.