Electron, WordPress & Angular Material – An Offline Viewer

Tweet about this on TwitterShare on LinkedIn0Share on Google+3Share on Reddit0Buffer this pageFlattr the authorEmail this to someonePrint this page

Imagine that you are at the airport, waiting for your flight and you hear a couple of guys sitting behind talking about Nodejs, Angularjs, MongoDB and you being a full stack developer and blogger start eves dropping. And all of sudden one guy mentions.. “Dude, you should totally check out this awesome blog.. It is titled ‘The Jackal of Javascript’ & that guy writes sweet stuff about node webkit, ionic framework and full stack application development with Javascript”. And the other guy says “cool..” And with his airport wifi connection starts browsing the blog on his laptop.. With in a few moments he loses interest because the pages aren’t loading.

What if the owner of the wordpress blog had an offline viewer and the first guy showed all the awesome stuff in the blog to the other guy on his laptop without any internet connection, wouldn’t that be cool?

So that is what we are going to do in this post. Build an offline viewer for a wordpress blog.

Below is a quick video introduction to the application we are going to build. 

 

Sweet right? 

So, let us stop imagining and let us build the offline viewer.

You can find the completed code here.

Architecture

As mentioned in the video, this is a POC for creating an offline viewer for a wordpress blog. We are using Electron (a.k.a Atom shell) to build the offline viewer. We will be using Angularjs in the form of Angular Material project to build the user interface for the application.

offlineviewer

As shown above, We will be using the request node module to make HTTP request to the wordpress API to download the blog posts in JSON format and persist it locally using DiskDB.

Once all the posts are downloaded, we will sandbox the content. The word sandbox here refers to containment of user interaction & experience inside the viewer. As in how we are capturing the content and replaying it back to the user. In our case, we are sandboxing the images and links, to control the user experience when the user is online or offline. This is better illustrated in the video above.

The JSON response from the WordPress API would look like : https://public-api.wordpress.com/rest/v1.1/sites/thejackalofjavascript.com/posts?page=1.

If you replace my blog domain name from the above URL with your WordPress blog, you should see the posts from your blog. We will be working with content property of the post object.

Note : We will be using recursions at a lot of places to process collections, asynchronously. I am still not sure if this is an ideal design pattern for an application like this.

Prerequisites

The offline viewer is a bit complicated in terms of number of components used. Before you proceed, I recommend taking a look at the following posts

Getting started

We will be using a yeoman generator named generator-electron to scaffold our base project, and then we will add the remaining components as we go along.

Create a new folder named offline_viewer and open terminal/prompt there.

To setup the generator, run the following

npm install yo grunt-cli bower generator-electron

This may take a few minutes to download all the once modules. Once this is done, we will scaffold a new app inside the offline_viewer folder. Run

yo electron

You can fill in the questions after executing the above command as applicable. Once the project is scaffolded and dependencies are installed, we will add a few more modules applicable to the offline viewer. Run,

npm install cheerio diskdb request socket.io --save

  • cheerio is to manage DOM manipulation in Nodejs. This module will help us sandboxing the content.
  • diskdb for data persistence.
  • request module for contacting the wordpress blog and get the posts as well as to sandbox images.
  • socket.io for realtime communication between the locally persisted data and the UI.

For generating installer for mac & some folder maintenance, we will add the below two modules.

npm install trash appdmg --save-dev

The final package.json should look like

Do notice that I have added bunch of scripts and updated the meta information of the project. We will be using NPM itself as a task runner to run, build and release the app.

To make sure everything is working fine, run

npm run start

And you should see

Screen Shot 2015-05-27 at 6.16.23 pm

Build the Socket Server

Now, we will create the server interface for our offline viewer, that talks to the WordPress JSON API.

Create a new folder named app at the root of the project. Inside the app folder, create two folders named serverclient. This folders kind of visually demarcate the server code vs. the client code. Here the server being Nodejs & client being Angular application. Inside the Electron shell, any code can be accessed any where, but to keep the code base clean and manageable, we will be maintaining the above folder structure.

Inside the app/server create a file named server.js. This file is responsible for setting up the socket server. Open server.js and update it as below

Things to notice

Line 1: We require the getport module. This module takes care of looking for an available port on the user’s machine that we can use to start the socket server. We cannot pick 8080 or 3000 or any other static port and assume that the selected port would be available on the client. We will work on this file in a moment.

Line 2 : We require the fetcher module. This module is responsible for fetching the content from WordPress REST API, save the data to DiskDB and send the response back.

Line 3 : We require the searcher module. This module is responsible for searching the locally persisted JSON data, as part of the search feature.

Line 7 : As soon as we get a valid port from getport module, we will start a new socket server.

Line 11 : Once a client connects to the server, we will setup listeners for load event and search event.

Line 13 : This event will be fired when the client wants to get the posts from the server. Once the data arrives, we emits a loaded event with the posts

Line 19 : This event will be fired when the client send a query to be searched. Once the results arrive, we emit the results event with the found posts.

Line 27 : Once the socket server is setup, we will execute the callback and send the used port number back.

Next, create a new file named getport.js inside the app/server folder. This file will have the code to return an unused port. Update getport.js as below

Things to notice

Line 2 : Require the net module, to start a new server

Line 3 : Starting value of the port, from which we need to start checking for a available port

Line 5 : A recursive function that keeps running till it finds a free port.

Line 10 : We attempt to start a server on the port specified, if we are successful, we call the callback function with the port that worked else, if the server errors out, we call the getPort()  again. And this time, we increment the port by one & then try starting server. This goes on till the server creation succeeds.

Next, create a file named fetcher.js inside app/server folder. This file will consist of all the business logic for our application. Update app/server/fetcher.js as below

Things to notice

* Most of the iterative logic implemented here is recursive.

Line 1 : We include the request module

Line 2 : We include the sandboxer module. This module will be used to sandbox the responses that we get from the WordPress API

Line 3 : We include the imageSandBoxer module. This module consists of logic to sandbox images. As in convert the http URL to a base64 one.

Line 4: We include the connection module. This module consists of the code to initialize the DiskDB and export the db object.

Line 6 : We set a few values to be used while processing.

Line 12 : The fetcher is invoked from the server.js passing in the online/offline status of the viewer and a callback. Since we are using recursions, we pass in a third argument to  fetcher()  named skip, which will decide if we need to skip/stop making REST calls to the WordPress REST API.

Line 13, 14 : We query the DiskDB for all the posts and meta data.

Line 18 : If meta data is not present, we create a new entry with the total value set to -1. Total stores the total post that the blog has.

Line 28 : We check if at least one batch of posts are downloaded and we can skip fetching from the REST API. If it is true, we check if all the posts are downloaded. If yes, we call  sendByParts() and send 20 posts at once.

Line 104 :  sendByParts() is a recursive function that send back 20 posts per second

Line 40 : If not all posts are downloaded, we send what we have downloaded so far and then call  fetcher() passing in the online/offline status, the callback and set skip to true. Now, when  fetcher() is invoked from here, the if condition on line 28 will be false. So, it will move on and download the remaining posts. Before we call  fetcher(), we will set the page value.

Line 52 : If we are loading the posts from page 1 again, we are removing all the posts and downloading again. This is as part of the POC. Ideally, we need to implement a replace logic while saving existing posts instead of removing the file and adding it again.

Line 58 : We check if the user is online and then only start making calls to the WordPress API.

Line 59 : We create a request to fetch the first page posts.

Line 67 : If it is a success, and we have more than one post in the response, we update the total post count that is sent by the API in our meta collection.

Line 76 : Once the meta collection is updated, we invoke the sandboxer(). The sandboxer() takes in a set of posts and sandboxes the URLs and content. And the sandboxed content is sent back.

Line 80 : We save the sandboxed posts to DiskDB and return the same to the UI. After that we increment the page number and call  fetcher(). As mentioned most of logic in application runs recursively.

Line 85 : If we are done downloading all the posts, we rest the page number and invoke the  imageSandBoxer(), which reads all the saved posts from DiskDB and converts the images with http urls to base 64.

Next, we will implement the sandboxer. Create a file named sandboxer.js inside app/server folder. And update it as below

Things to notice

* Most of the iterative logic implemented here is recursive.

Line 1 : We include cheerio

Line 2 : We create a few global scoped variables for recursion.

Line 9 : When the sandboxer is called, we reset the global variables and then call the  process() recursively till all the posts in the current batch are done.

Line 15 : Inside the process(), we check if a post exists. If it does, we start the sandboxing process, if not we execute the callback on line 78

Line 16 : We will access the content property on post and run the content through  cheerio.load(). This provides us with a jQuery like wrapper to work with DOM inside Nodejs. This is quite essential for us to sandbox the content

Line 19 : We sandbox the links. We iterate through each of the links and add a ng-click attribute to it with a custom function. Since I know I am going to use Angularjs, I have attached an ng-click attribute. Apart from that I remove unwanted attributes and reset the href, not to fire links by default.

Line 37 : If the anchor tag has children, I need to process them. In my blog, all the images are wrapped around with an anchor tag because of a plugin I use. I do not want that kind of markup here, where clicking on the image takes the user to the original images. So we clean that up and add custom classes to manage the cursor. All this is sandboxing links.

Line 52 :  For syntax highlighting, I am using an Angular directive named hljs. So, I iterate over all the pre tags in my content and set attribute on them that will help me work with then hljs directive. This is a typical example of integrating a third party angular directive with the sandboxer.

Line 58 : We sandbox all the iframe urls which have youtube as their src. By default all the youtube embed URL are protocol relative. They would look like //youtube.com?watch=1234. This will not work properly in the viewer, hence we will convert them to a http URL. Also, I am adding an ng-class attribute on the iFrame, whose src has youtube in it. This is to show or hide the iFrame depending on the network status as demoed in the video.

Line 69 : We sandbox the image tag, replace any additional parameters in the URL. This is specific to my blog. I have a plugin which adds this.

Line 73 : Once the sandboxing is done, we need to update the original HTML with the sandboxed version.

Line 75 : Call  process() on the next post.

To complete the sandboxing, we will be adding a new file named imageSandBoxer.js inside app/server folder. Update imageSandBoxer.js as below

Things to notice

* Most of the iterative logic implemented here is recursive.

imageSandBoxer() will be called only after all the posts are downloaded.

Line 1 : We require the connection module. This module consists of the code to initialize the DiskDB and export the db object.

Line 2 : We require cheerio

Line 5 : We require the request module. Do notice that we are setting encoding to null. This is to make sure we download image response as binary.

Line 12 : If there is no network connection, do not make calls

Line 15 : We processPost() by passing in the post and a callback. This recursively runs till all the posts are done processing.

Line 21 : We get all the images from the post and call the  sandBoxImage() by passing in one image at a time. This recursively runs till all the images are done processing.

Line 29 : If there is a valid image, get the image data. Convert the response to a base 64 format on line 32 and update the src attribute.

Line 39 : Once all images in a given post are done processing, we update the data in DiskDB and go to the next post.

Now, we will create a file named connection.js inside app/server folder. Update it as below

For DiskDB to work, create an empty folder named db inside app/serve folder.

To complete the so called server, we will create a file named searcher.js inside app/server folder. Update searcher.js  as below

Things to notice

Line 1 : We require the connection to DB

Line 6 : We fetch all the posts

Line 9 :  We run through each post’s title and content and check if the keyword we are searching for exists. if yes, we push it to the results.

Line 19 : Finally we send back the results

This completes our server.

Build the Angularjs Client

To work with client side dependencies, we will use bower package manager. From the root of offline_viewer folder run

bower init

And fill the fields as applicable. This will create a bower.json file at the root of the project. Next, create a file named .bowerrc at the same level as bower.json. Update it as below

This will take care of downloading all the dependencies inside the lib folder.

Next, run

bower install angular-material angular-route roboto-fontface open-sans-fontface angular-highlightjs ng-mfb ionicons jquery --save

If you see a message like Unable to find a suitable version for angular, do as shown below

Screen Shot 2015-05-28 at 12.50.33 am

Quick explanation of the libraries

  • jquery : DOM manipulation.
  • angular-material : Material Design in Angular.js
  • angular-route : The ngRoute module provides routing and deeplinking services and directives for angular apps.
  • angular-highlightjs : AngularJS directive for syntax highlighting with highlight.js
  • ng-mfb : Material design floating menu with action buttons implemented as an Angularjs directive.
  • roboto-fontface : Bower package for the Roboto font-face
  • open-sans-fontface – Bower package for the open-sans font-face
  • ionicons : The premium icon font for Ionic Framework.

The final bower.json would look like

Next, we will update index.html file present at the root of  offline_viewer folder.

Things to notice

Lines 7 to 13 : We require all the CSS files.

Line 16 : We require the server.js file.

Line 17 : We invoke app(), which starts the socket server on an available port. We get the used port as the first argument in the callback. We set that value on the window object.

Line 27 : We require jQuery using the module format

Lines 30 – 42 : We require all js files needed. We will create the missing files as we go along.

Line 43 : We set up the fab button. This is the search button on the bottom right hand corner of the viewer.

Note : We have not used ng-app on any DOM element. We are going to bootstrap this Angular app manually.

Next, open index.js and update as below

Do notice line nos. 15, 32 and 36.

If you want you can delete index.css present at the root. We will be creating one more inside the client folder later.

Next, we will add the missing scripts. Create a folder named js inside app/client. Inside the js folder, create a file named app.js. Update app.js as below.

Things to notice

Lines 2 – 5 : We need to manually download the socket script file as we are not of the port that the socket server would start on.

Line 11 : Once the script is loaded, we will manually bootstrap the angular application.

Line 15 : Initialize a new Angular module named OfflineViewer and add all the dependencies.

Line 17 :  We will be configuring the default theme of Angular Material, the connection url for sockets and the routes.

Line 37 : We add listeners to the online & offline event. And when the state changes, we broadcast the appropriate event. So that the viewer can behave accordingly.

Next, create a new file named controller.js inside app/client folder. Update it as below

Things to notice

Line 1 : We create a  OfflineViewer.controllers module

Line 4 :  HeaderCtrl is the controller for our application header. Here we define the status of the connectivity and are listening to  viewer-online and  viewer-offline. And we set the status variable depending on the event fired.

Line 22 :  AppCtrl is the main controller for our application, that notifies the socket server about the status of the viewer and makes requests to fetch the posts,

Line 31 : As soon as the  AppCtrl is initialized, it will check if the viewer has access to the internet. If no, it shows a dialog with the information that the user is offline and it needs some kind of internet access to download the initial set of posts.

Line 42 : We emit the load event to our server, with the status. We will talk about the  $socket a bit letter.

Line 45 : Once the socket server receives the first set of posts, it will fire the the loaded event. And the hook here would be called with the first set of posts. Here we concat the incoming posts with the existing posts and assign it to the scope variable.

Line 49 : When a user clicks on a post links, we update the main viewer with the content of the posts.

Line 53 : This is the method we have added while sandboxing the anchor tags. When the user clicks on a link, this method would be fired and it show a popup dialog, asking the user, if s/he wants to open the link. If yes, we execute line 62.

Line 68 : When ever the user goes online, we emit the load method, indicating our socket server to start fetching the posts if it has not already done so.

Line 74 : We reset the status to false if the user goes offline. This status is used to hide/show youtube videos in the content.

Line 86 : The  SearchCtrl for managing the search feature. When the user clicks on the search icon, on bottom right corner, we show the search dialog.

Line 103 : When the user enters text and clicks on search, we emit the search event with the query text.

Line 115 : Once the results arrive, we update the scope variable with results, which displays the text.

Line 110 : When a user clicks on a post, we broadcast a showSearchPost event, which is listened to on line 80 and shows the post in the main content area.

Quite a lot of important functionality.

Next, we will create a custom directive, that takes up the content of the posts and renders it. Before we render it, we need to compile it so all the attributes we have added while sandboxing will come to life.

Create a new file named directives.js inside app/client/js folder and update it as below

All we do is parse and compile the content and render it inside a div.

Next, we will add a custom filter, all it does is takes in a content and runs a  $sce.trustAsHtml().

Create a new file named filters.js inside app/client/js folder and update it as below

And finally the service component that talks to the socket server. The code written below is taken from the an awesome blog post written by Maciej Sopyło, Tutsplus named : More Responsive Single-Page Applications With AngularJS & Socket.IO: Creating the Library.

Create a new file named services.js inside app/client/js folder and update it as below

You can read more about how the above service is written in the post linked above.

Now, we will create the three templates used in the app. The header, the app and the search.

Create a new folder named templates inside app/client. Create a new file named header.html inside the app/client/templates folder and update it as below

Next, comes the app template. Create a file named app.html inside app/client/templates folder. Update it as below

This file consist of the main application view template.

Finally the search template. Create a file named search.html inside app/client/templates.

Finally the styles. Create a folder named css. Create a file named index.css inside app/client/css. We will add application specific overrides here. Update index.css as below

And some styles for code highlighting. Create a file named highlight.css inside app/client/css folder. Update it as below

That is it!! We are all done!!

Test the Offline Viewer

To test the viewer run,

npm run start

And this should launch the viewer and start downloading all the posts. If you want to open devtools, open index.js and uncomment the below line

And run

npm run start

Screen Shot 2015-05-28 at 11.41.21 am

And you should see the above the view.

You can browse through the app, search, click on links and test it out.

If you inspect the image tag after a while, you should see that the src is replaced with base 64 format of the image.

Screen Shot 2015-05-28 at 12.07.58 pm

Build the Offline Viewer

To build and distribute the application for a given OS, we will be running the scripts which we have written in the package.json.

npm run build-mac

or

npm run build-linux

or

npm run build-win

Or if you want to build for all platforms at once run

npm run build

Do not forget to comment the mainWindow.openDevTools(); line in index.js. Else your packaged app will have the devtools enabled

Release the Offline Viewer

As of now, I have integrated appdmg module, to generate DMG files for mac. To implement this, create a new file named app-dmg.json at the root of the project. Update app-dmg.json as below

And for the resources, you can copy all the required files from this folder.

And finally run

npm run release-mac

And it should generate the installer.

So that is it! A Proof of Concept on how to replay the content whether the user is online or offline.


Thanks for reading! Do comment.
@arvindr21

Tweet about this on TwitterShare on LinkedIn0Share on Google+3Share on Reddit0Buffer this pageFlattr the authorEmail this to someonePrint this page
  • Travis Reynolds

    I also have an issue – https://uploads.disquscdn.com/images/d909cb2475a08a7d1d1265780166fd96952fe8eeae7683d24b96c47998ce2b0d.png

    what do i need to do to resolve this please?

    (running windows 10)

  • [email protected]

    [14533:1024/141541:INFO:CONSOLE(31)] “The DB Path [/Users/Utilisateur/Sites/wordpress-offline-viewer/app/server/db] does not seem to be valid. Recheck the path and try again”, source: /Users/Utilisateur/Sites/wordpress-offline-viewer/node_modules/diskdb/lib/diskdb.js

  • [email protected]

    Hello, Sorry … I have a many error … :( Your project is amazing 😀

  • Fake_of_Silva

    Hello, I’m having some issues with this project. The error log is from the project from your github:

    Failed to load resource: net::ERR_FILE_NOT_FOUND file:///C:/workspace/offlineview/app/client/lib/highlightjs/highlight.pack.js

    module.js:326 Uncaught Error: Cannot find module ‘angular’

    angular.js:68 Uncaught Error: [$injector:modulerr] Failed to instantiate module OfflineViewer due to:

    Error: [$injector:modulerr] Failed to instantiate module hljs due to:

    Error: [$injector:nomod] Module ‘hljs’ is not available! You either misspelled the module name or forgot to load it. If registering a module ensure that you specify the dependencies as the second argument.

    http://errors.angularjs.org/1.4.7/$injector/nomod?p0=hljs

    at file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:68:12

    at file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:1986:17

    at ensure (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:1910:38)

    at module (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:1984:14)

    at file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:4390:22

    at forEach (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:336:20)

    at loadModules (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:4374:5)

    at file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:4391:40

    at forEach (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:336:20)

    at loadModules (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:4374:5)

    http://errors.angularjs.org/1.4.7/$injector/modulerr?p0=hljs&p1=Error%3A%20…space%2Fofflineview%2Fapp%2Fclient%2Flib%2Fangular%2Fangular.js%3A4374%3A5)

    at file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:68:12

    at file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:4413:15

    at forEach (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:336:20)

    at loadModules (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:4374:5)

    at file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:4391:40

    at forEach (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:336:20)

    at loadModules (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:4374:5)

    at createInjector (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:4299:11)

    at doBootstrap (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:1657:20)

    at Object.bootstrap (file:///C:/workspace/offlineview/app/client/lib/angular/angular.js:1678:12)

    http://errors.angularjs.org/1.4.7/$injector/modulerr?p0=OfflineViewer&p1=Er…pace%2Fofflineview%2Fapp%2Fclient%2Flib%2Fangular%2Fangular.js%3A1678%3A12)

  • http://www.omanamazon.com Sami

    Very nice tutorial thank you very much for sharing .

  • Robert Cox

    bravo man, this is incredible as always.