Pixel 4 Recorder app with offline transcribe

A project log for Android offline speech recognition natively on PC

Porting the Android on-device speech recognition found in GBoard to TensorFlow Lite or LWTNN

biemsterbiemster 10/16/2019 at 07:584 Comments

So 15th of October Google showcased the new Recorder app for Pixel 4 devices, with real time transcription. After downloading the app and peaking inside it contains the same type of RNNT models with 2 encoders, a decoder and a joint, so I assume it's the same model. It is considerably smaller though, so I expect it to be an update.

The tflite files are also not obfuscated, and the zip contains .ascii_proto files that are human readable. It even contains shell scripts to run the models on a local machine!

This is the third full model I'm analysing, and seems to contain the most info thus far. I'll update this log is I find out more.


Jack Zhang wrote 12/28/2019 at 05:12 point

Any updates on the progress? It seems that you're really close to a working PoC.

  Are you sure? yes | no

biemster wrote 01/02/2020 at 20:43 point

that's what i've been thinking for a while now as well, but currently @theafien is the closest to a poc with using this in his own android app. I'm trying that road now too, so I have my own java code calling the tensorflow models. That way I have a better handle on what's happening, but I hit a couple road blocks atm..

  Are you sure? yes | no

Giacomo M wrote 10/29/2019 at 23:54 point

How do you run the shell scripts locally? refers to which is missing? Thanks

  Are you sure? yes | no

biemster wrote 10/30/2019 at 13:28 point

Yeah that doesn't work indeed, I posted this before I had a proper look at the scripts. The shell scripts need the greco3 application, which is a proprietary tool from google. I highly doubt I ever get my hands on that unfortunately.

  Are you sure? yes | no