Category Archives: Node.js

Automatic Deployment for Node Apps via GitHub (+submodules)

I recently went into the trouble of setting up a rather complex project related to the Internet of Things, so here are a few things that I learned about deployments.

Intro

What we want to do is to facilitate our deployments so that whenever we push on a specific branch of git, our code is installed automatically on a remote server of our choice.

Git allows for “hooks”, which are callback actions that are performed whenever specific event such as a push has occurred. There are a few setups suggested, but here I want to demonstrate a more “custom” version, where more control can added on the callbacks and the actions performed.

Environment

DigitalOcean

I used DigitalOcean, an alternative to Amazon AWS. It’s extremely easy to use and is generally considered cheaper. I will assume a Fedora 21 server (which I originally chose for its use of systemd), so be careful not to copy and paste my code on a Ubuntu distribution without modifying it first (for adding a user use this instead).

After we have an instance running and are able to login through ssh, we need to create a user that is going to run our node apps. In Linux, a process will run with the access level of the user that initialised it and we don’t want to grant un-necessary privileges to a process that is meant to be doing anything other than hacking our system 😉

Now logout and log back in as ‘node’:

Once logged in we want to generate an ssh key for pulling from GitHub.

Don’t freak out with the lack of a password! When deploying automatically, there will be no user logged in the build machine to type the password! 😉

GitHub

The code will live on GitHub where I will assume that we are not developing an open source project and that we want to keep that repo private. This is to demonstrate a few of the caveats that are only present themselves in the case of private repos as they relate to access rights. We probably want to create multiple branches on our repo as we are developing (following GitFlow or something similar) but for the moment we should at least have a master and a production branch.

On GitHub, we can add an ssh key to the account itself. This will give the owner of that key access to all repositories in the account. We don’t want that. We want to add the key to a specific repository. Go to Settings > Deploy keys and copy the :

dkeyes

Copy the key we created into the Key field.

PM2

For managing Node apps I tend to use PM2, which is a newer alternative to Forever*. Both will monitor any number of applications and restart them when they crash,  but PM2 has more robust logging as well as a Web API that can be invaluable. Back on our remote server:

*another tool that I have not used but seems good is Guvnor.

The Webhook

GitHub has the ability to add callbacks to a remote server when a specific event is performed on a repos. From your project’s repo go to Settings > Webhooks & Services. Specify a Payload URL which will be the remote address of the deployment server together with a port (like 8000 or 8080 since our node user cannot open port 80 which is restricted to the root). We also take note of the secret to embed it in our script.

hook

 

When pushing, the web-hook will send an HTTP request with a payload of a JSON object containing details on the event. This means that a server should be running on the receiving end, able to parse and respond to that request. We can write a server in Node through the github-webhook-handler module.

The Webhook Server

Inside the project’s repo we will create a folder called scripts (or whatever else we want) and a file called hook.js inside of it with the code below:

The script above will check whether the push has been on the ‘production’ branch and if this was the case then run the script called ‘deploy’.

Deployment

 The deploy script

Now for the bash script itself. Very simply create an executable called deploy inside the scripts folder and insert:

We can add as many steps as we want such as grunt tasks to build the project and jasmine tests, but for the moment let’s assume a more simplistic execution strategy.

Running the processes

Start both processes on the deployment server:

Screen Shot 2015-08-10 at 13.21.06

The output should look as above (in your bash color scheme of course ;-).

Pushing new Changes

From your local machine you can now make changes to the code base, merge into production and make a push! The code should automatically be updated on the remote branch and you will see the console logs together with whatever messages git and npm output (thanks to pm2 logs).

Pulling from Multiple Repositories

Multiple Deploy Keys

A problem arises when we need to pull from more than one repository as each deploy key gives access to a single repository. An obvious solution seems to be to create multiple ssh keys and add them to the various repositories. The problem is however that when ssh connects, it will always use the id_rsa.pub key unless told otherwise, so let’s tell it otherwise. First create a second key

Now we must add it to the ssh agent:

Then, we need to tell ssh that when calling a specific url you must use the new key instead of the default. This is achieved via the config file (either existing or being created for this occasion) on ~/.ssh/config. In there we create something similar to a virtual host:

Test what we’ve done with…

Now, we can pull from shared via:

In the git@shared above, the ‘shared’ part will be replaced with github.com as specified in the Hostname in the configuration file, while respecting all the key value pairs defined (most importantly the IdentityFile, or key, to use).

Deploying with submodules

A common reason for pulling for multiple repos is the case of submodules. Let’s assume that the project called shared is pulled as a submodule from inside the service repo. This would be recorded in the .gitmodules file as:

If our user on the server had also user access to GitHub (i.e. its ssh keys where added in the account rather than a single repository as deploy keys), this works well because we could get everything in one go with:

Once however we use deploy keys, this will fail for the reasons explained above. Following the solution we outlined we can just change the url into:

And now it will work just fine!

After we set our ssh config to support multiple hosts we need to update our deploy script to pull the submodules as well as the project.