Web Speech API, Phonegap and Phonegap build – A Smart Mouth App

Imagine you are on a vacation to Paris and you do not know how to speak French! Wouldn’t that be a big bummer? What if, there was an App that can speak french on your behalf?

Sounds intriguing right?

I got this idea from Eric Bidelman’s demo on the Web Speech API

Note : The Web Speech API starts @ 30 minutes into the video

So, I wanted to replicate this idea as a hybrid mobile app. But because of the poor webview support for Web Speech API, I was looking around for a polyfill.

I found a polyfill named speech-synthesis which works awesome on browsers, but unfortunately it does not on webviews!

So I forked the repo and patched it with a fix, so it works for webviews as well.

The issue was with the audio tag, which was not working on the device. Tried a couple of hacks but nothing seemed to work. I added a patch so that the polyfill uses Cordova Media API instead of the HTML5 Audio tag and Bam!! it started working.

The patched repo is called speech-synthesis-cordova. And we are going to use the same in this post.

Below is quick peak of the app

Awesome right! Getting one more step closer to my robot butler Jarvis!

You can find the completed code here.

So let us get started.


If you are new to PhoneGap and have not set up or used it before I would recommend going through the below

If you are new to Web Speech API, you can check out Getting Started with Web Speech API.

Understanding the polyfill

If the Web Speech API is available in your browser, the Web Speech API will be used to convert text to speech. Else, the polyfill will make a request to Google Translate and then play that media.

In our case, we will be playing the response from the Google Translate using Cordova’s Media API.

Setup Project

Make sure you have followed PhoneGap 3 CLI Setup on Mac & Windows and set up Phonegap on your machine.

Next, create a new folder named smartMouthApp. Open a new terminal/prompt here and run

phonegap create smartMouth       

This will create a new Phonegap project. Once the setup is done, we need to CD into the smartMouth folder. And then we will add the Media API plugin. Run

cd smartMouth

phonegap plugin add org.apache.cordova.media

This should take care of plugin dependency.

Next, we need to move the config.xml from the root of smartMouth folder to inside www folder. This is more of a Phonegap build dependency.

Once you move the config.xml into the www folder, open it up in your favorite editor. And update the essential details like

Do notice line 11. We have added the plugin dependency here.

Next, to manage our client side dependency, we will use bower. If you have not installed bower yet, you can do so by running

npm install bower -g

Now CD into the www folder. We will init a new bower.json file. Run

bower init 

and fill it up as

Create another file at the root of www  folder and name it .bowerrc. This file consist of the bower configs. Update .bowerrc file as below

Now, we will add the dependencies to bower.json. Add the below section to the bower.json

and then run

bower install

This will download the required dependencies and save it inside the www/libs folder.

Note : DO NOT forget to remove the unwanted files before submitting the app to Phonegap build.

Now all the dependencies are set up, we will begin the actual app development.

Begin Development

The idea of the App is simple, a user will enter text and select a language. The app will go to Google Translate and then get the translated text. Once this comes back, the polyfill will read it out loud to the user.

First, open www/index.html. We will add references to the downloaded libs. The updated index.html would be

Now, we will update the js/index.js with the app logic. Open it up in your favorite editor and update it as below

Things to notice

Line 39 : Gets triggered when the device is ready.

Line 40,41 : Init the polyfill

Line 43 : When the user clicks on translate button, we check if the text is valid on line 48 and then make an Ajax call to translate.google.com to get the translation for the text.

Line 46 : In case the text is empty we warn the user.

Line 56 : Once the data is received, we eval it, parse it and get the translated text.

Line 58 : The translated text is show on the UI

Line 60fallbackSpeechSynthesisUtterance is called passing in the translated text to be read out loud along with the lang

Line 67 : Welcomes the user when the app launches.

Bam! That is it! Our app is ready!

Test the app

You can test the app on an emulator or by adding android/ios platforms.

phonegap install android

If you are facing issues on the device, I highly recommend using chrome inspect tool.

Phonegap Build

Now that the app is tested, we will issue a Phonegap build to generate installer.

Navigate to Phonegap Build and create a new app.

Screen Shot 2014-10-28 at 4.43.19 pm

You can either upload the www folder (only www folder is required for a phonegap build) or push the www folder to a git repo and then provide the details in the above screen.

Once your code is uploaded/synced successfully, you will see a Ready to Build button. Screen Shot 2014-10-28 at 4.47.36 pm

And once you click it, the build process will start.

Once the build is completed, you should see something like

Screen Shot 2014-10-28 at 4.49.32 pmYou can either QR the image or click on install to download the installer for your device.

This app may not be using much of the Web Speech API, but this can be a start. As more and more updates keep coming in, we can run Web Speech API without any polyfills and achieve the same results.

Thanks for reading! Do comment.

Raspberry Pi, Camera and Node.js – Live Streaming with Websockets #IoT

A few days ago Bala Kolluru has reached out to me asking if we can control a Raspberry Pi camera module using Web browser, so he can view a live stream from any HTML5 powered device. I was intrigued by this idea and wanted to give it a try.

In this post, we will see how we can implement a system that can “stream” a video from our pi to a browser. The completed system would look like

Pretty sweet right! I am able to see my aquarium from any where and check on my only gold fish. This can be extended to do anything.

For instance, you can hook the camera up pointing at the front door, as soon as someone rings the bell, you can see who is at the door by opening the video stream URL in your mobile/tablet/computer and confirm if you need to wear pants to open the door.

So let us see how we can build such an awesome multi-purpose system.

You can find the complete code for this system here.

Continue reading

Raspberry Pi, Camera, Node.js – Video & Email The Intruder

In one of my earlier posts : Raspberry pi, PIR Sensor and Node.js – An IoT Intruder Alert system, I have shown how you can detect intruders with Raspberry Pi and a PIR sensor.  And once the intruder is detected, an email will be sent with the time of incident.

This is good but not that helpful. In this post, instead of just sending an email, we will record a video of the intruder as the “act” is performed and then attach that video along with the email. Sweet right!

Below is a quick demo that show how things work

The video is a bit fast and too shaky. So Let me quickly run you through it. First we start the node program. As soon as a intruder is detected, the buzzer gets triggered. At the same time, we start the recording on the camera. We record a video for 10 seconds and then start sending the same in an email.

Pretty sweet right!

You can find the completed code here.

So let us get started building this IoT Video and Email The Intruder system!

Continue reading

Getting started with MQTT

In this post, we will take a look at a communication protocol named Message Queue Telemetry Transport a.k.a. MQTT. MQTT is a lightweight, secure, battery friendly and a machine-to-machine (M2M)/”Internet of Things” connectivity protocol.

Andy Stanford-Clark and Arlen Nipper from Cirrus Link Solutions authored the first version of the protocol in 1999.

What is MQTT?

MQTT (formerly Message Queue Telemetry Transport) is a publish-subscribe based “light weight” messaging protocol for use on top of the TCP/IP protocol. It is designed for connections with remote locations where a “small code footprint” is required and/or network bandwidth is limited. The Publish-Subscribe messaging pattern requires a message broker. The broker is responsible for distributing messages to interested clients based on the topic of a message.

The specification does not specify the meaning of “small code foot print” or the meaning of “limited network bandwidth”. Thus, the protocol’s availability for use depends on the context. In 2013 IBM submitted MQTT v3.1 to the OASIS specification body with a charter that ensured only minor changes to the specification could be accepted. MQTT-SN is a variation of the main protocol aimed at embedded devices on non-TCP/IP networks, such as ZigBee.

Historically, the ‘MQ’ in ‘MQTT’ came from IBM’s MQ message queuing product line. However, queuing per se is not required to be supported as a standard feature in all situations.

Real world applications

Facebook Messenger : Facebook has used aspects of MQTT in Facebook Messenger. However, it is unclear how much of MQTT is used or for what; Moreover it is to be noted that this is a phone application not a sensor application.

Continue reading

Raspberry pi, PIR Sensor and Node.js – An IoT Intruder Alert system

In this post, we will build an Internet of Things Intruder alert system. This system uses a PIR Sensor along with the Raspberry pi. When the PIR Sensor detects a change in a given surrounding, it will trigger the Pi to send out an email to a user with the time of intrusion. Sweet right!

The final system would look like

You can find the completed code here.

So, let us get started.

Continue reading

Node-Webkit File Player – A File player for your desktop

A few days ago, Ignacio Tartavull reached out to me mentioning that he had built a file viewer based on Node Webkit powered Dashboard application and associated the node webkit app as the default opener for.ipynb files. Which I thought was a pretty genius idea. You can check out the INotebook app here.

Inspired by that idea, I have written a file browser/player app with node webkit that can “play” basic files like txt, log, json, md, mp3, mp4, png and jpeg.

Also, mustapha has reached out to me requesting another tutorial on Node webkit. And Mitesh Agarwal reached out to me requesting a post on a node webkit file viewer. So here it goes.

The final product we are going to build will look like

You can find the completed code here.

So, let us get started.

Continue reading

Raspberry pi, a Buzzer and Node.js – An IoT Doorbell

In this post, we will build an embedded system – A Doorbell, with Raspberry pi B+, A Buzzer and Node.js. This embedded system will act as both a normal Doorbell with a button switch and the same doorbell can be controlled over internet.  Simple right?

The final output of this embedded system would look like


The completed code can be found here.

So, let us get started.

Continue reading

Image Manipulation with Node.js and L.W.I.P.

Image and Manipulation with Node.js and lwip

I have been looking around the web for quite sometime now for a Node module that can perform a simple image resize. I did find node canvas, GraphicsMagick for node, but they all have pretty heavy dependencies and I did not want to go through so much pain for a simple resize.

Then I stumbled upon LWIP.

This module provides comprehensive, fast, and simple image processing and manipulation capabilities.

There are no external runtime dependencies, which means you don’t have to install anything else on your system.

The last line caught my attention and I wanted to give it a try. I have incorporated the Lwip in my blueimp-file-upload-expressjs module to create thumbnails after upload. And it works like a charm and without any hassle.

Eyal Arubas, the author of the module wrote a neat blog post, explaining why he had to come up with one such module for node.

In this post, we will take a look at Image processing with Node.js and Light weight Image processor. By the end of this post, we will turn this image


into this one


Image credits : https://flic.kr/p/dyvdpu

You can find the completed code and images here.

So, Let us get started.

Continue reading