Web Speech API, Phonegap and Phonegap build – A Smart Mouth App

Tweet about this on TwitterShare on LinkedIn4Share on Google+2Share on Reddit0Buffer this pageFlattr the authorEmail this to someonePrint this page

Imagine you are on a vacation to Paris and you do not know how to speak French! Wouldn’t that be a big bummer? What if, there was an App that can speak french on your behalf?

Sounds intriguing right?

I got this idea from Eric Bidelman’s demo on the Web Speech API

Note : The Web Speech API starts @ 30 minutes into the video

So, I wanted to replicate this idea as a hybrid mobile app. But because of the poor webview support for Web Speech API, I was looking around for a polyfill.

I found a polyfill named speech-synthesis which works awesome on browsers, but unfortunately it does not on webviews!

So I forked the repo and patched it with a fix, so it works for webviews as well.

The issue was with the audio tag, which was not working on the device. Tried a couple of hacks but nothing seemed to work. I added a patch so that the polyfill uses Cordova Media API instead of the HTML5 Audio tag and Bam!! it started working.

The patched repo is called speech-synthesis-cordova. And we are going to use the same in this post.

Below is quick peak of the app

Awesome right! Getting one more step closer to my robot butler Jarvis!

You can find the completed code here.

So let us get started.

Prerequisites

If you are new to PhoneGap and have not set up or used it before I would recommend going through the below

If you are new to Web Speech API, you can check out Getting Started with Web Speech API.

Understanding the polyfill

If the Web Speech API is available in your browser, the Web Speech API will be used to convert text to speech. Else, the polyfill will make a request to Google Translate and then play that media.

In our case, we will be playing the response from the Google Translate using Cordova’s Media API.

Setup Project

Make sure you have followed PhoneGap 3 CLI Setup on Mac & Windows and set up Phonegap on your machine.

Next, create a new folder named smartMouthApp. Open a new terminal/prompt here and run

phonegap create smartMouth       

This will create a new Phonegap project. Once the setup is done, we need to CD into the smartMouth folder. And then we will add the Media API plugin. Run

cd smartMouth

phonegap plugin add org.apache.cordova.media

This should take care of plugin dependency.

Next, we need to move the config.xml from the root of smartMouth folder to inside www folder. This is more of a Phonegap build dependency.

Once you move the config.xml into the www folder, open it up in your favorite editor. And update the essential details like

Do notice line 11. We have added the plugin dependency here.

Next, to manage our client side dependency, we will use bower. If you have not installed bower yet, you can do so by running

npm install bower -g

Now CD into the www folder. We will init a new bower.json file. Run

bower init 

and fill it up as

Create another file at the root of www  folder and name it .bowerrc. This file consist of the bower configs. Update .bowerrc file as below

Now, we will add the dependencies to bower.json. Add the below section to the bower.json

and then run

bower install

This will download the required dependencies and save it inside the www/libs folder.

Note : DO NOT forget to remove the unwanted files before submitting the app to Phonegap build.

Now all the dependencies are set up, we will begin the actual app development.

Begin Development

The idea of the App is simple, a user will enter text and select a language. The app will go to Google Translate and then get the translated text. Once this comes back, the polyfill will read it out loud to the user.

First, open www/index.html. We will add references to the downloaded libs. The updated index.html would be

Now, we will update the js/index.js with the app logic. Open it up in your favorite editor and update it as below

Things to notice

Line 39 : Gets triggered when the device is ready.

Line 40,41 : Init the polyfill

Line 43 : When the user clicks on translate button, we check if the text is valid on line 48 and then make an Ajax call to translate.google.com to get the translation for the text.

Line 46 : In case the text is empty we warn the user.

Line 56 : Once the data is received, we eval it, parse it and get the translated text.

Line 58 : The translated text is show on the UI

Line 60fallbackSpeechSynthesisUtterance is called passing in the translated text to be read out loud along with the lang

Line 67 : Welcomes the user when the app launches.

Bam! That is it! Our app is ready!

Test the app

You can test the app on an emulator or by adding android/ios platforms.

phonegap install android

If you are facing issues on the device, I highly recommend using chrome inspect tool.

Phonegap Build

Now that the app is tested, we will issue a Phonegap build to generate installer.

Navigate to Phonegap Build and create a new app.

Screen Shot 2014-10-28 at 4.43.19 pm

You can either upload the www folder (only www folder is required for a phonegap build) or push the www folder to a git repo and then provide the details in the above screen.

Once your code is uploaded/synced successfully, you will see a Ready to Build button. Screen Shot 2014-10-28 at 4.47.36 pm

And once you click it, the build process will start.

Once the build is completed, you should see something like

Screen Shot 2014-10-28 at 4.49.32 pmYou can either QR the image or click on install to download the installer for your device.

This app may not be using much of the Web Speech API, but this can be a start. As more and more updates keep coming in, we can run Web Speech API without any polyfills and achieve the same results.


Thanks for reading! Do comment.
@arvindr21

Tweet about this on TwitterShare on LinkedIn4Share on Google+2Share on Reddit0Buffer this pageFlattr the authorEmail this to someonePrint this page
  • Dhanis Andyanto

    hello arvind, would you like to explain why use http://www.corsproxy.com/translate.google.com/ in line 55? And is that site still working?

  • Rennan Alves

    I also play in turorial, but is not working :(

    • http://thejackalofjavascript.com/ Arvind Ravulavaru

      What error are you facing?

  • Vivek Sharma

    Hi Arvind ,

    While installing the bower.It is showing an error git not installed or path not set. I have alredy installed git. Please help me what to do. thanks

    • http://thejackalofjavascript.com/ Arvind Ravulavaru

      Which OS are you using? If windows, try restarting the machine.

  • Tiemin

    Is this possible speech to text as well? Thanks

    • http://thejackalofjavascript.com/ Arvind Ravulavaru

      Yes, but I do not think so it is supported by the web views yet.

    • Lucky

      I built a cordova app that does both…. without a webservice

      • http://thejackalofjavascript.com/ Arvind Ravulavaru

        Great! Do you mind sharing the approach? Thanks!

        • Lucky

          sure. I will upload the source to dropbox

          • http://thejackalofjavascript.com/ Arvind Ravulavaru

            Thanks! Looking forward!

          • Lucky

            I sent the link to your mail

          • Mabast Omer

            sorry i need the same if you can send to me, i will appropreat it

          • Lucky
          • Muhammad Syawaludin Hastaf

            Lucky, it seems that the speech recognition it’s not working well. but the speech to text is working fine

      • Moisesd37

        Bro, would you mind to share it with me as well? i’ve spent days trying to achieve that on ionic.

  • Bruno Seixas

    I think you are on to something 😉
    The idea is really cool and helpful!

    • http://thejackalofjavascript.com/ Arvind Ravulavaru

      Yup. The idea is awesome. Thanks to Eric Bidelman.