As I mentioned in my previous post I’ve been deploying a web application to a vagrant VM using Capistrano and my initial configuration was like so:

require 'capistrano/ext/multistage'
 
set :application, "thinkingingraphs"
set :scm, :git
set :repository,  "git@bitbucket.org:markhneedham/thinkingingraphs.git"
set :scm_passphrase, ""
 
set :ssh_options, {:forward_agent => true, :paranoid => false, keys: ['~/.vagrant.d/insecure_private_key']}
set :stages, ["vagrant"]
set :default_stage, "vagrant"
 
set :user, "vagrant"
server "192.168.33.101", :app, :web, :db, :primary => true
set :deploy_to, "/var/www/thinkingingraphs"

When I ran ‘cap deploy’ I ended up with the following error:

  * executing "git clone -q git@bitbucket.org:markhneedham/thinkingingraphs.git /var/www/thinkingingraphs/releases/20130414171523 && cd /var/www/thinkingingraphs/releases/20130414171523 && git checkout -q -b deploy 6dcbf945ef5b8a5d5d39784800f4a6b7731c7d8a && (echo 6dcbf945ef5b8a5d5d39784800f4a6b7731c7d8a > /var/www/thinkingingraphs/releases/20130414171523/REVISION)"
    servers: ["192.168.33.101"]
    [192.168.33.101] executing command
 ** [192.168.33.101 :: err] Host key verification failed.
 ** [192.168.33.101 :: err] fatal: The remote end hung up unexpectedly

As far as I can tell the reason for this is that bitbucket hasn’t been verified as a host by the VM and therefore the equivalent of the following happens when it tries to clone the repository:

$ ssh git@bitbucket.org
The authenticity of host 'bitbucket.org (207.223.240.182)' can't be established.
RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.
Are you sure you want to continue connecting (yes/no)?

Since we aren’t answering ‘yes’ to that question and bitbucket isn’t in our ~/.ssh/known_hosts file it’s not able to continue.

One solution to this problem is to run the ssh command above and then answer ‘yes’ to the question which will add bitbucket to our known_hosts file and we can then run ‘cap deploy’ again.

It’s a bit annoying to have that manual step though so another way is to set cap to use pty by putting the following line in our config file:

set :default_run_options, {:pty => true}

Now when we run ‘cap deploy’ we can see that bitbucket automatically gets added to the known_hosts file:

    servers: ["192.168.33.101"]
    [192.168.33.101] executing command
 ** [192.168.33.101 :: out] The authenticity of host 'bitbucket.org (207.223.240.181)' can't be established.
 ** RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.
 ** Are you sure you want to continue connecting (yes/no)?
 ** [192.168.33.101 :: out] yes
 ** [192.168.33.101 :: out] Warning: Permanently added 'bitbucket.org,207.223.240.181' (RSA) to the list of known hosts.

As far as I can tell this runs the command using a pseudo terminal and then automatically adds bitbucket into the known_hosts file but I’m not entirely sure how that works. My google skillz have also failed me so if anyone can explain it to me that’d be cool

I’ve been working on a tutorial around thinking through problems in graphs using my football graph and I wanted to deploy it on a local vagrant VM as a stepping stone to deploying it in a live environment.

My Vagrant file for the VM looks like this:

# -*- mode: ruby -*-
# vi: set ft=ruby :
 
Vagrant::Config.run do |config|
  config.vm.box = "precise64"
 
  config.vm.define :neo01 do |neo|
    neo.vm.network :hostonly, "192.168.33.101"
    neo.vm.host_name = 'neo01.local'
    neo.vm.forward_port 7474, 57474
    neo.vm.forward_port 80, 50080
  end
 
  config.vm.box_url = "http://files.vagrantup.com/precise64.box"
 
  config.vm.provision :puppet do |puppet|
    puppet.manifests_path = "puppet/manifests"
    puppet.manifest_file  = "site.pp"
    puppet.module_path = "puppet/modules"
  end
end

I’m port forwarding ports 80 and 7474 to 50080 and 57474 respectively so that I can access the web app and neo4j console from my browser.

There is a bunch of puppet code to configure the machine in the location specified.

Since the web app is written in Ruby/Sinatra the easiest deployment tool to use is probably capistrano and I found the tutorial on the beanstalk website really helpful for getting me setup.

My config/deploy.rb file which I’ve got Capistrano setup to read looks like this:

require 'capistrano/ext/multistage'
 
set :application, "thinkingingraphs"
set :scm, :git
set :repository,  "git@bitbucket.org:markhneedham/thinkingingraphs.git"
set :scm_passphrase, ""
 
set :ssh_options, {:forward_agent => true}
set :default_run_options, {:pty => true}
set :stages, ["vagrant"]
set :default_stage, "vagrant"

In my config/deploy/vagrant.rb file I have the following:

set :user, "vagrant"
server "192.168.33.101", :app, :web, :db, :primary => true
set :deploy_to, "/var/www/thinkingingraphs"

So that IP there is the same one that I assigned in Vagrantfile. If you didn’t do that then you’d need to use ‘vagrant ssh’ to go onto the VM and then ‘ifconfig’ to grab the IP instead.

I figured there was probably another step required to tell Capistrano where it should get the vagrant public key from but I thought I’d try and deploy anyway just to see what would happen.

$ bundle exec cap deploy

It asked me to enter the vagrant user’s password which is ‘vagrant’ by default and I eventually found a post on StackOverflow which suggested changing the ‘ssh_options’ to the following:

set :ssh_options, {:forward_agent => true, keys: ['~/.vagrant.d/insecure_private_key']}

And with that the deployment worked flawlessly! Happy days.

Although I know this problem is already solved by collectd and New Relic I wanted to write a little shell script that showed me the memory usage on a bunch of VMs by parsing the output of free.

The output I was playing with looks like this:

$ free -m
             total       used       free     shared    buffers     cached
Mem:           365        360          5          0         59         97
-/+ buffers/cache:        203        161
Swap:          767         13        754

I wanted to find out what % of the memory on the machine was being used and as I understand it the numbers that we would use to calculate this are the ‘total’ value on the ‘Mem’ line and the ‘used’ value on the ‘buffers/cache’ line.

I initially thought that the ‘used’ value I was interested in should be the one on the ‘Mem’ line but this number includes memory that Linux has borrowed for disk caching so it isn’t the true number.

There’s another quite interesting article showing some experiments you can do to prove this.

So what I wanted to do was get the result of the calculation ’203/365′ which I wasn’t sure how to do until I realised you can match multiple regular expressions with awk like so:

$ free -m | awk '/Mem:/ { print $2 } /buffers\/cache/ { print $3 }'                                                        
365
203

We’ve now filtered the output down to just our two numbers but another neat thing you can do with awk is change what it uses as its field and record separator.

In this case we want to change the field separator to be the new line character and we’ll set the record separator to be nothing because otherwise it defaults to the new line character which will mess with our field separator.

Those two values are set by using the ‘RS’ and ‘FS’ variables:

$ free -m | 
  awk '/Mem:/ { print $2 } /buffers\/cache/ { print $3 }' | 
  awk 'BEGIN { RS = "" ; FS = "\n" } { print $2 / $1 }'
0.556164

This is still sub optimal because we’re using two awk commands rather than one! We can get around that by storing the two memory values in variables and printing them out in an END block:

$ free -m | 
  awk '/Mem:/ { total=$2 } /buffers\/cache/ { used=$3 } END { print used/total}'
0.556164

I’ve been playing around with some code to spin up AWS instances using Fabric and Boto and one thing that I wanted to do was define a bunch of default properties in a JSON file and then load this into a script.

I found it harder to work out how to do this than I expected to so I thought I’d document it for future me!

My JSON file looks like this:

config/defaults.json

{
	"region" : "eu-west-1",
	"instanceType": "m1.small"
}

To read that file we can do the following:

>>> open('config/defaults.json').read()
'{\n\t"region" : "eu-west-1",\n\t"instanceType": "m1.small"\n}'

We can then use the json.loads function to convert that from a string into a Python object:

>>> import json
>>> config = json.loads(open('config/defaults.json').read())
>>> config
{u'region': u'eu-west-1', u'instanceType': u'm1.small'}

We’d write the following code to get the region:

>>> config["region"]
u'eu-west-1'

I guess we might want to use a different approach that didn’t load the whole string into memory if we had a large JSON file but for my purposes this will do!

Although I didn’t go to Dev Ops Days London earlier in the year I was following the hash tag on twitter and one of my favourites things that I read was the following:

“Treating servers as cattle, not as pets” #DevOpsDays

I think this is particularly applicable now that a lot of the time we’re using virtualised production environments via AWS, Rackspace or <insert-cloud-provider-here>.

At uSwitch we use AWS and over the last week Sid and I spent some time investigating a memory leak by running our applications against two different versions of Ruby.

One of them was from the Brightbox repository and the other was custom built but they had annoyingly different puppet configurations so we decided to treat them as separate machine types.

We spun up one of the custom built Ruby nodes and put it in the load balancer alongside 11 of the other node types and left it for the day serving traffic.

The next day we had look at the New Relic memory consumption for both node types and it was clear that the custom built one’s memory usage was climbing much more slowly than the other one.

Instead of trying to work out how to change the Ruby version of the 11 existing nodes we realised it would probably be quicker to just spin up 11 new ones with the custom built Ruby and swap them with the existing ones.

This was pretty much as easy as removing the existing nodes from the load balancer and putting the new ones in although we do have one ‘special’ machine which runs some background jobs.

We needed to make sure there weren’t any jobs on its queue that hadn’t been processed and then make sure that we tagged one of the new machines so that they could take over that role.

One thing that made it particularly easy for us to do this is that spin up of new VMs is extremely quick and completely automated including the installation and start up of applications.

The only manual step we have is to put the new nodes into the load balancer which I think works ok as a manual step because it gives us a chance to quickly scan the box and check everything spun up correctly.

We install all packages/configuration on nodes using puppet headless which makes spin up easier than if you use server/client mode where you have to coordinate node registration with the master on spin up.

I do like this philosophy to machines and although I’m sure it doesn’t apply to all situations we’re almost at the point where if something breaks on a node we might as well spin up a new one while we’re investigating and see which finishes first!

After coming across this post about why you should use Sublime Text I decided to try using it a bit more and one of the things that I missed from Textmate was the way you can select the current file on the sidebar.

In Textmate the shortcut to do that is ‘Cmd + Ctrl + R’ so I wanted to be able to do something similar or configure Sublime so it responded to the same shortcut.

The option to reveal a file in the side bar is accessible from the context menu by right clicking on the contents of a file after it’s opening and selecting ‘Reveal in Side Bar’ which is a good start.

To map that to a key binding we need to go ‘Preferences > Key Bindings (User)’ and put the following into that file:

[
	{ "keys": ["ctrl+super+r"], "command": "reveal_in_side_bar" }
]

Of course if we already have other custom key bindings then we can just add it after those instead.

We can work out what the name of commands are by turning on command logging in the Sublime console.

We need to first open the console with ‘Ctrl + `” and then paste the following:

sublime.log_commands(True)

Any commands that we run will now have their name printed in the console window. e.g.

>>> sublime.log_commands(True)
command: context_menu {"event": {"button": 1, "x": 390.21484375, "y": 329.66796875}}
command: reveal_in_side_bar
command: rename_path {"paths": ["/Users/markhneedham/code/thinkingingraphs/public/js/bootstrap.js"]}
no command for selector: noop:
command: show_panel {"panel": "console", "toggle": true}

We can then setup appropriate key bindings for whichever commands we like.

I part time administrate a football forum that I used to run when I was at university and one problem we had recently was that some of the tables/indices had got corrupted when MySQL crashed due to a lack of disc space.

We weren’t seeing any visible sign of a problem in any of the logs but whenever you tried to query one of the topics it wasn’t returning any posts.

I eventually came across a useful article which explained how to check whether some of the tables in a MySQL database had been corrupted and how to fix them.

I first shutdown the database using the following command:

mysqladmin shutdown

And then I ran this command to check on the status of each of the tables:

for path in `ls /var/lib/mysql/forum/*.MYI`; do echo $path; myisamchk $path; done

This gave an output like the following for each table:

Checking MyISAM file: /var/lib/mysql/forum/forum.MYI
Data records:     217   Deleted blocks:       4
myisamchk: warning: 1 client is using or hasn't closed the table properly
- check file-size
- check record delete-chain
- check key delete-chain
- check index reference
- check data record references index: 1
- check record links
MyISAM-table '/var/lib/mysql/forum/forum.MYI' is usable but should be fixed

If you pass the ‘–recover’ flag to myisamchk it will attempt to fix any problems it finds. I therefore ran the following command:

for path in `ls /var/lib/mysql/forum/*.MYI`; do echo $path; myisamchk --recover $path; done

After I’d run that it seemed to fix most of the problems we’d been experiencing. There are still a couple of edge cases left but at least the majority of the forum is now in a usable state.

I think we could just as easily run myisamchk by passing a wildcard selection of files for it to run against but I didn’t realise that until afterwards!

The following would therefore work just as well:

myisamchk --recover /var/lib/mysql/forum/*.MYI

Despite the fact that I’ve been working full time in software for almost 8 years now every now and then I still need a reminder of how useful reading logs can be in helping solve problems.

I had a couple of such instances recently which I thought I’d document.

The first was a couple of weeks ago when Tim and I were pairing on moving some applications from Passenger to Unicorn and were testing whether or not we’d done so successfully.

We were doing this by creating an /etc/hosts entry from our top level domain to an nginx proxy node which was to forward on the request to the application server.

Request -> nginx on proxy node -> nginx on app server node -> unicorn on app server node

This didn’t work and we got a 404 response code so I logged onto the server hosting the application server and started writing our a cURL command to simulate what the proxy should be doing to see if the problem was there.

After watching me do this a couple of times Tim suggested that we might be more successful if we opened a bunch of tabs on the shell tailing the various log files that the request should pass through.

We set up tail commands against the following files:

• nginx access log on proxy node
• nginx error log on proxy node
• nginx access log on the app server node
• unicorn log on the app server node

Immediately it became clear that we actually had a problem on the proxy node because we’d configured one of the nginx directives incorrectly.

Once we’d fixed this the request flowed through smoothly.

We extended this tailing of files idea when testing multiple nodes through a load balancer except this time we made use of Fabric to make things a bit easier.

The second was earlier this week when Baris and I were trying to configure puppet so that we could install different Ruby versions on different machines.

We were having trouble figuring out why the wrong version was getting installed so eventually we chatted to Andy who amongst other things had a look at the apt history log @ /var/log/apt/history.log and was able to figure out how this was happening.

Lesson of the last two weeks: embrace the logs!

I’ve done a bit of modelling with years and months in neo4j graphs that I’ve worked on previously but I haven’t ever done anything with time so I thought it’d be interesting to have a go with my football graph.

I came across this StackOverflow post on my travels which suggested that indexing nodes by time would be helpful and since I have a bunch of football matches with associated times I thought I’d try it out.

I created the key of the index by running code similar to the following:

> DateTime.now.strftime("%H%M")
=> "2200"

We can then write a query to show all the games at a certain time of day:

START game=node:times('time:*')
RETURN game.time, COUNT(game)
ORDER BY game.time

+-------------------------+
| game.time | COUNT(game) |
+-------------------------+
| 1245      | 21          |
| 1330      | 21          |
| 1500      | 163         |
| 1600      | 29          |
| 1730      | 22          |
| 1945      | 21          |
| 2000      | 19          |
+-------------------------+
7 rows

To be fair any index that referenced all the matches would allow us to do this. e.g.

START game=node:matches('match_id:*')
RETURN game.time, COUNT(game)
ORDER BY game.time

The time based indexing becomes more interesting when we use Lucene’s numeric range query syntax to only select matches which happened between certain times of day:

START game=node:times('time:[1600 TO 2000]')
RETURN game.time, COUNT(game)
ORDER BY game.time

+-------------------------+
| game.time | COUNT(game) |
+-------------------------+
| 1600      | 29          |
| 1730      | 22          |
| 1945      | 21          |
| 2000      | 19          |
+-------------------------+
4 rows

I couldn’t see a way to set an open ended value either side of the ‘TO’ so if we want to do that we just need to set a really high maximum value or really low minimum value.

For example if we want to find all the evening matches we could use this query:

START game=node:times('time:[1730 TO 2359]')
RETURN game.time, COUNT(game)
ORDER BY game.time

+-------------------------+
| game.time | COUNT(game) |
+-------------------------+
| 1730      | 22          |
| 1945      | 21          |
| 2000      | 19          |
+-------------------------+
3 rows

I also indexed each match by its full timestamp so we could find all the evening games this year if we wanted as well:

> Time.new(2013,1,1).to_i
=> 1356998400

START game=node:times('time:[1730 TO 2359] AND date: [1356998400 TO 9999999999]')
RETURN game.time, game.name, game.friendly_date

+------------------------------------------------------------------------------------+
| game.time | game.name                                | game.friendly_date          |
+------------------------------------------------------------------------------------+
| 1730      | "Wigan Athletic vs Liverpool"            | "2013-03-02 17:30:00 +0000" |
| 2000      | "Aston Villa vs Manchester City"         | "2013-03-04 20:00:00 +0000" |
| 2000      | "West Ham United vs Tottenham Hotspur"   | "2013-02-25 20:00:00 +0000" |
| 2000      | "Liverpool vs West Bromwich Albion"      | "2013-02-11 20:00:00 +0000" |
| 1730      | "Fulham vs Manchester United"            | "2013-02-02 17:30:00 +0000" |
...
| 1945      | "Chelsea vs Southampton"                 | "2013-01-16 19:45:00 +0000" |
+------------------------------------------------------------------------------------+
25 rows

A couple of weeks ago Tim and I were spinning up a new service on a machine which wasn’t quite working so we were manually making changes to the /etc/nginx/nginx.conf file and restarting nginx to try and sort it out.

This process is generally not that interesting – you open the file in vi, make some changes, close it, then restart nginx and see if it works. If not then you open the file again and repeat.

Except Tim had a slight variation on this workflow which is an improvement that I don’t want to forget!

Once we’d finished making the changes to the file in vi Tim hit ‘Ctrl + Z‘ which suspended the vi process and put us back at the shell prompt.

We could then restart nginx or do whatever else we needed to do and then type ‘fg‘ to go back into vi again.

Not only is this workflow quicker, it also keeps the history of the changes that we’ve made to the file so if one of our changes really screws things up we can easily undo it. Previously we’d have to remember what changes we’d made and do that manually.

In summary this workflow is a pretty simple idea but nevertheless one I had never thought about or seen anyone else do and I’ll be using it in future.