Getting Started with Client Side Storage

In this post, we will take a look at various client side storage solutions and how to work with them.

We are going to cover the following

  • Cookies
  • Web Storage API
    • Session Storage
    • Local Storage
    • String Compression
  • IndexedDB
  • Next Steps

You can find the completed code in the below examples here.

Cookies

There was a time when client side storage was all about saving data in cookies. Cookies were designed initially not to save data offline but rather save the established session details and send it with every request to the server.

 … a way to create a stateful session with Hypertext Transfer Protocol (HTTP) requests and responses.  It describes three new headers, Cookie, Cookie2, and Set-Cookie2, which carry state information between participating origin servers and user agents.  The method described here differs from Netscape’s Cookie proposal [Netscape], but it can interoperate with HTTP/1.0 user agents that use Netscape’s method.

You can read more : rfc2965

Back to storage with Cookies. For instance, you have a very long form to be filled in by the user. If the user refreshes the page accidentally while filling the form, you want to save the data and populate it back. And at the same time, you do not want to send this partial data to the server. This should be saved on the client side. And this is how we would implement this using cookies.

You would have a long form like

I have used only 2 form fields for simplicity. Also, I have used Twitter Bootstrap for styling. This form can be how ever long you want and how many ever form elements you want (select, radio, textarea, etc..).

And the JS code to control this would be

Things to notice

Line 5 :  A wrapper API to work with cookies. Create, Read, Delete

Line 38 : When the document is ready, we register for blur events on the form elements. And when the call back is triggered, we create a new cookie passing the cookieName, the entire form data and days to expire to  createCookie().

Line 50 : On page load, we check if there is any pre-saved data available. If yes, we read the same and populate it

Line 66 : And the form is submitted successfully, we delete the saved cookie.

And when you fill the form and “accidentally” refresh, you should the see the saved data populate. And when you inspect the saved cookies and you should see

Screen Shot 2015-04-19 at 8.41.32 am

And when submit, you should see the cookie removed.

Size Limitations :

If you want to support most browsers, then don’t exceed 50 cookies per domain, and don’t exceed 4093** bytes per domain (i.e. total size of all cookies <= 4093 bytes).

You can read more : Browser Cookie Limits.

For me the biggest problem with cookies is that, the browser sends all the cookies related to that domain with every HTTP request. Which does not sound very “Client Side Storage” for me.

Then came along HTML5 with it’s arsenal of storage solutions.

Web Storage API

The Web Storage API provides mechanisms by which browsers can securely store key/value pairs, in a much more intuitive fashion than using cookies.

The two mechanisms within Web Storage are as follows:

  • sessionStorage maintains a separate storage area for each given origin that’s available for the duration of the page session (as long as the browser is open, including page reloads and restores)
  • localStorage does the same thing, but persists even when the browser is closed and reopened.

You can read more about the Web Storage API from MDN here.

As you see from below http://caniuse.com/#feat=namevalue-storage support

Screen Shot 2015-04-19 at 9.18.22 am

there is about 94% support for Web Storage API across all browsers. Which is good as we can consider this to be a reliable solution in almost all cases.

Session Storage

Taking the above example, of saving a long form, we will implement the same using session storage API. The HTML for the page would remain the same. The JS code would look like

Things to notice

Line 1 : A wrapper around sessionStorage API.

Line 8 : We create a method to check sessionStorage support.

Line 12, 16 : Save and retrieve string based data. The Storage API saves all the values in the form of string. So any value as in a string, number, boolean can be saved and retrieved using these methods.

Line 26, 30 : These methods are used to save and retrieve Objects and Arrays. You can read more about this here : Storing Objects in HTML5 Local Storage.

Line 34 : A method to remove an item from the sessionStorage

Line 38 : A  method to clear all items from the sessionStorage

Line 47 : We check if sessionStorage is supported, else we throw an alert

Line 51 : We register a blur event on the form elements. And when the listener fires, we get the form data, convert it to an object and save it

Line 68 : We check if anything is already stored in the sessionStorage. If yes, we populate the form.

Line 76 : When the form saves the data successfully, we clear the sessionStorage.

You can inspect the Session Storage saved data and you should see.

Screen Shot 2015-04-19 at 10.01.37 amIn the above case, if the form fields are dependent on a session then the session storage is perfect. Because if the form changes for the same user for a different session, the old data will not clash with the new data.

Local Storage

What if you are looking for a way to persist the data even after the user’s session expires. The answer is simple, Local Storage API.

The transformation from Session Storage to Local Storage is as simple as replacing the word Session with Local. The updated JS file would look like

Things to notice

Line 1 : A wrapper around localStorage API.

Line 8 : We create a method to check localStorage support.

Line 12, 14 : Save and retrieve string based data. The Storage API saves all the values in the form of string. So any value as in a string, number, boolean can be saved and retrieved using these methods.

Line 26, 30 : These methods are used to save and retrieve Objects and Arrays. You can read more about this Storing Objects in HTML5 Local Storage.

Line 34 : A method to remove an item from the localStorage

Line 38 : A  method to clear all items from the localStorage

Line 47 : We check if localStorage is supported, else we throw an alert

Line 51 : We register a blur event on the form elements. And when the listener fires, we get the form data, convert it to an object and save it

Line 68 : We check if anything is already stored in the localStorage. If yes, we populate the form.

Line 76 : When the form saves the data successfully, we clear the localStorage.

You can inspect the Local Storage saved data and you should see

Screen Shot 2015-04-19 at 10.15.53 amNow, even after rebooting the machine, this data will be persisted.

Size Limitations :

The size limitations for a given domain for the Storage API is dependent on the browser and the OS. If you check out Web Storage Support Test,

Screen Shot 2015-04-19 at 10.34.15 am

The above values are for Chrome 42 64bit. Also based on a lot of solutions on the internet, the storage seems to be around 5 MB.

You can read more : Is 5MB the de facto limit for W3C Web Storage?

String Compression

If you see that your storage needs seem to exceed the 5 MB quota, you can implement lz-string compression. You can checkout the demo page and see the compression factor

Screen Shot 2015-04-19 at 10.44.28 am

The implementation is pretty simple. First we include lz-string.min.js in our page. Then we will update the storage API to compress and decompress the string. The updated Local Storage API would be

On line 13 and line 31, we compress the string before we save it. And on line 19 and line 37, we decompress the string.

Note : On Firefox and IE, localStorage cannot contain invalid UTF16 characters. You need to use  LZString.compressToUTF16() and  LZString.decompressFromUTF16() if you are targeting IE and FF as part of your app. Else you can use  LZString.compress() &  LZString.decompress()

If you observed the save data, you should see the compressed string

Screen Shot 2015-04-19 at 10.51.25 am

You can implement the same API for Session Storage as we did above for Local Storage. Do notice that compressing adds an overhead to the get and set process. So use this only if needed.

IndexedDB

IndexedDB is a way for you to persistently store data inside a user’s browser. Because it lets you create web applications with rich query abilities regardless of network availability, these applications can work both online and offline.

IndexedDB is an alternative to WebSQL Database, which the W3C deprecated on November 18, 2010. While both IndexedDB and WebSQL are solutions for storage, they do not offer the same functionalities. WebSQL Database is a relational database access system, whereas IndexedDB is an indexed table system.

Concepts of IndexedDB

  • IndexedDB databases store key-value pairs.
  • IndexedDB is built on a transactional database model.
  • The IndexedDB API is mostly asynchronous.
  • IndexedDB uses requests all over the place.
  • IndexedDB uses DOM events to notify you when results are available.
  • IndexedDB is object-oriented.
  • IndexedDB does not use Structured Query Language (SQL).
  • IndexedDB adheres to a same-origin policy.

You can read more about the above concepts here.

Before we start implementing IndexedDB, we need to go through its terminology.

  • database : A repository of information, typically comprising one or more object stores.
  • object store : The mechanism by which data is stored in the database.
  • version : When a database is first created, its version is the integer 1.
  • database connection : An operation created by opening a database.
  • transaction : An atomic and durable set of data-access and data-modification operations on a particular database.
  • request : The operation by which reading and writing on a database is done.
  • index : An index is a specialized object store for looking up records in another object store, called the referenced object store.

You can read more here.

Now that we have a basic understanding of what IndexedDB is all about, we will make use of it save our form offline. Yes, this use case is not an ideal example for IndexedDB. It can do a lot more.

If you are planning to use the IndexedDB API as is, you can do so by following this tutorial : Using IndexedDB. To keep things simple and interesting, I am going to use an awesome IndexDB wrapper named Dexie.js.

Dexie has a huge API that helps a lot while interacting with IndexedDB. You can read more about Dexie from its very well documented wiki.

Back to our example. We will be implementing offline storage of our form using IndexedDB via Dexie. The HTML for the form will remain the same. After you include dexie.js in your page, you can write your logic as follows

Things to notice

Line 3 : We create a new database named longForm

Line 5 :  We create a new auto incrementing index. You can read more about Schema definition syntax here.

Line 8 : We connect to the DB. On successful connection, we query the DB for any pre-saved data.

Line 10 : Once we get any pre-saved data, we populate our form on line 13.

Line 20 : On blur of any form fields, we collect the latest form data

Line 32 : We delete any existing data before we save a fresh copy.

Line 39 : When the form is successfully submitted, we will delete the DB.

When you inspect the data, it should look like

Screen Shot 2015-04-19 at 2.15.30 pm

Next Steps

With this, we have completed the most prominent offline storage solution available today. I recommend taking a look at the following as next steps

I will keep adding more resources as I get.


Thanks for reading! Do comment.
@arvindr21

Web Speech API, Phonegap and Phonegap build – A Smart Mouth App

Imagine you are on a vacation to Paris and you do not know how to speak French! Wouldn’t that be a big bummer? What if, there was an App that can speak french on your behalf?

Sounds intriguing right?

I got this idea from Eric Bidelman’s demo on the Web Speech API

Note : The Web Speech API starts @ 30 minutes into the video

So, I wanted to replicate this idea as a hybrid mobile app. But because of the poor webview support for Web Speech API, I was looking around for a polyfill.

I found a polyfill named speech-synthesis which works awesome on browsers, but unfortunately it does not on webviews!

So I forked the repo and patched it with a fix, so it works for webviews as well.

The issue was with the audio tag, which was not working on the device. Tried a couple of hacks but nothing seemed to work. I added a patch so that the polyfill uses Cordova Media API instead of the HTML5 Audio tag and Bam!! it started working.

The patched repo is called speech-synthesis-cordova. And we are going to use the same in this post.

Below is quick peak of the app

Awesome right! Getting one more step closer to my robot butler Jarvis!

You can find the completed code here.

So let us get started.

Prerequisites

If you are new to PhoneGap and have not set up or used it before I would recommend going through the below

If you are new to Web Speech API, you can check out Getting Started with Web Speech API.

Understanding the polyfill

If the Web Speech API is available in your browser, the Web Speech API will be used to convert text to speech. Else, the polyfill will make a request to Google Translate and then play that media.

In our case, we will be playing the response from the Google Translate using Cordova’s Media API.

Setup Project

Make sure you have followed PhoneGap 3 CLI Setup on Mac & Windows and set up Phonegap on your machine.

Next, create a new folder named smartMouthApp. Open a new terminal/prompt here and run

phonegap create smartMouth       

This will create a new Phonegap project. Once the setup is done, we need to CD into the smartMouth folder. And then we will add the Media API plugin. Run

cd smartMouth

phonegap plugin add org.apache.cordova.media

This should take care of plugin dependency.

Next, we need to move the config.xml from the root of smartMouth folder to inside www folder. This is more of a Phonegap build dependency.

Once you move the config.xml into the www folder, open it up in your favorite editor. And update the essential details like

Do notice line 11. We have added the plugin dependency here.

Next, to manage our client side dependency, we will use bower. If you have not installed bower yet, you can do so by running

npm install bower -g

Now CD into the www folder. We will init a new bower.json file. Run

bower init 

and fill it up as

Create another file at the root of www  folder and name it .bowerrc. This file consist of the bower configs. Update .bowerrc file as below

Now, we will add the dependencies to bower.json. Add the below section to the bower.json

and then run

bower install

This will download the required dependencies and save it inside the www/libs folder.

Note : DO NOT forget to remove the unwanted files before submitting the app to Phonegap build.

Now all the dependencies are set up, we will begin the actual app development.

Begin Development

The idea of the App is simple, a user will enter text and select a language. The app will go to Google Translate and then get the translated text. Once this comes back, the polyfill will read it out loud to the user.

First, open www/index.html. We will add references to the downloaded libs. The updated index.html would be

Now, we will update the js/index.js with the app logic. Open it up in your favorite editor and update it as below

Things to notice

Line 39 : Gets triggered when the device is ready.

Line 40,41 : Init the polyfill

Line 43 : When the user clicks on translate button, we check if the text is valid on line 48 and then make an Ajax call to translate.google.com to get the translation for the text.

Line 46 : In case the text is empty we warn the user.

Line 56 : Once the data is received, we eval it, parse it and get the translated text.

Line 58 : The translated text is show on the UI

Line 60fallbackSpeechSynthesisUtterance is called passing in the translated text to be read out loud along with the lang

Line 67 : Welcomes the user when the app launches.

Bam! That is it! Our app is ready!

Test the app

You can test the app on an emulator or by adding android/ios platforms.

phonegap install android

If you are facing issues on the device, I highly recommend using chrome inspect tool.

Phonegap Build

Now that the app is tested, we will issue a Phonegap build to generate installer.

Navigate to Phonegap Build and create a new app.

Screen Shot 2014-10-28 at 4.43.19 pm

You can either upload the www folder (only www folder is required for a phonegap build) or push the www folder to a git repo and then provide the details in the above screen.

Once your code is uploaded/synced successfully, you will see a Ready to Build button. Screen Shot 2014-10-28 at 4.47.36 pm

And once you click it, the build process will start.

Once the build is completed, you should see something like

Screen Shot 2014-10-28 at 4.49.32 pmYou can either QR the image or click on install to download the installer for your device.

This app may not be using much of the Web Speech API, but this can be a start. As more and more updates keep coming in, we can run Web Speech API without any polyfills and achieve the same results.


Thanks for reading! Do comment.
@arvindr21

MEAN stack – A Hands on Tutorial

MEAN stack – A Hands on Tutorial

Also checkout a lighter version of the Todo App using DiskDB instead of MongoDB : Building a Todo App with DiskDB

If you are new to MEAN, I would recommend checking out my MEAN Workshop.

Looking for a way to set up Vagrant to work with M.E.A.N. Stack applications? Take a look at M.E.A.N. Stack Development with Vagrant

If you have been working in the  Nodejs/Web development world, you must have heard about MEAN. If you did not, MEAN stands for

  • M – MongoDB
  • E – Expressjs
  • A – Angularjs
  • N – Nodejs

Sweet right! This IMO is the next gen technology stack to build scalable and efficient web applications that uses Javascript on both the server and the client.

In this post, we will be building a simple Todo App. We will be building the app from scratch, which will give you an idea as what component goes where and how to configure it. If you already have taken a stab at MEAN stack generators/frameworks, you must have noticed the complex file structures, and you are not sure which files gets wired to which other file.  We will be demystifying them but in simple a Todo app.

The final product we are going to build will look like

Screen Shot 2014-08-22 at 7.53.26 amScreen Shot 2014-08-22 at 7.53.45 amScreen Shot 2014-08-23 at 8.29.18 pm

You can find a demo here. You can find the complete code here.

So, let us get started.

Continue reading

node-webkit and Firebase – Simple and Social Authentication

node-webkit and Firebase – Simple and Social Authentication

In this post, we will take a look at the Integration between Firebase Simple Authentication and node-webkit. We are going to use slush generators to scaffold basic apps for us. Then we will create and configure a Firebase account to manage Simple Authentication i.e. Email and Password and Social Authentication using Twitter, Google and Facebook services.

The final product would look like

Screen Shot 2014-08-15 at 6.20.04 pm Screen Shot 2014-08-15 at 6.20.25 pm Screen Shot 2014-08-15 at 6.21.04 pm

We are going to implement the same using pure Javascript as well as Angularjs.  You can find the completed code here.

So, let us get started.

Continue reading

Node webkit and Angularjs – A MovieStub App

Node webkit and Angularjs – MovieStub App

In this post, we will take a look at building Node webkit based desktop app using Angular js. By the end of this post, we are going to build a Movie Booking desktop application, where a user can see a list of movies and book tickets as per availability.

The final product would look like

Screen Shot 2014-08-11 at 8.25.26 am Screen Shot 2014-08-11 at 8.25.48 am Screen Shot 2014-08-11 at 8.26.04 amYou can find a live demo of the web version here. You can find the complete code here.  So, let us get started

Continue reading

Angularjs – A Hands On tutorial

Angularjs – A Hands On tutorial

This is a hands on tutorial on Angularjs.  We will walk through most of the key features of Angular, and by the end we will build an Angular application named Movie Stub.

You can see the live demo of the app here.

Preview:

Screen Shot 2014-03-14 at 9.01.49 PM Screen Shot 2014-03-14 at 9.02.11 PM Screen Shot 2014-03-14 at 9.02.28 PM Screen Shot 2014-03-14 at 9.02.36 PMSo, lets get started.

Contents

Code for each topic is available here. And the complete application code can be found here.

Continue reading

Responsive Web Design Test Automation

Responsive Web Design Test Automation

Responsive Web Design (RWD) has been around for quite some time. Today, startups would rather invest their time in a RWD website than build a native app. Why? Here is a presentation about RWD. You can take a quick look and come back.

Contents

The present day solution for testing Responsive sites is sort of manual. You would either use tools like screenfly to resize & check the layout of the pages. Or, you would get a lot of actual devices and test your website on each of them, to see if everything is aligned properly. Or the next gen solution is to take an account from products like Browser Stack, which would take a URL and spit out images from all the devices and browsers. Nevertheless, its a manual task again to verify all the images. 
Continue reading

Test Driven Development by Example

Test Driven Development by Example

In an earlier post, we have seen how to setup a framework/workspace that can be helpful to write easily testable Javascript code. Now we will dive into writing some code with the TDD approach. So lets get started.

Table of Contents

T.D.D. – Test Driven Development

Part 1 : Understand what TDD is all about. I ran into this amazing video by Rebecca Murphy. Where she explains as how Testable javascript needs to be written

A few excerpts from her presentation

Continue reading