Deploying Rails 3.1 applications with Capistrano

By Matthew Savage

I’m currently working on an application using the latest RC of Rails 3.1 (rc5, which is actually quite stable). In Rails 3.1 there is a new Asset Pipeline which now handles images, stylesheets, javascripts etc differently, especially when it comes to the filename.

In a Rails 3.0 or Rails 2.3 app, by default, a cache busting parameter is appended to the resource URL, for example: http://www.something.com/images/monkeys.jpg?138767865 – this number appended to the end if the mtime of the file from that server. When this was first put into place this was great, it worked as prescribed and was easy to manage, however there are some behaviours which this technique brings along with it:

  • Some caching servers will not cache a URL with a query string
  • The mtime of the file would be different across a web cluster, so you would have conflicting URLs and the cache wouldn’t work as expected
With the Rails 3.1 Asset Pipeline the files are now named with a hash in their file name, so monkeys.jpg becomes monkeys-bec6c752b4e7fbfa9ee4b99b562a03ca.png.
This, like the previous technique, comes for free in terms of configuration in the application, how you reference an image etc*, however when you deploy there is a rake task you should run to generate these files, which is rake assets:precompile.
So, how do you make this happen all by magic? Well I use Capistrano for deployment, so all I really need to do it make it call the rake tasks after deploying. You can easily do the same by adding the following code to the bottom of your deploy.rb file:
namespace :assets do
  task :precompile, :roles => :web do
    run "cd #{current_path} && RAILS_ENV=production bundle exec rake assets:precompile"
  end

  task :cleanup, :roles => :web do
    run "cd #{current_path} && RAILS_ENV=production bundle exec rake assets:clean"
  end
end

after :deploy, "assets:precompile"

There is also a cleanup task included to get rid of any hanging around assets which aren’t being used any more which you can run by calling cap assets:cleanup.

Hope this helps.

* There is one caveat, and that is with images referenced in CSS – if you are using SASS you can use image-url(...) which will output a url(...) directive with the correct URL, however in plain old CSS you should use url('<%= asset_path(...) %>'); and name the file with the .erb prefix to be preprocessed.

Posted in Development, General on August 11, 2011 – 5:41 pm | Comments (0)
Tagged , , , , |

Problems logging in to Snow Leopard domain after install OS X Lion

By Matthew Savage

TL;DR

Just installed Lion and can’t log in? Boot into recovery, open terminal, enter resetpassword, reset root password, reboot, login as root, rebind to Open Directory, reboot, profit :)

The Long Story

I received a phone call on Friday from someone who made the decision to update to OS X Lion, and ran into a huge problem – unable to log into the computer… at all. In normal circumstances this wouldn’t be an issue, I would just have the user log in with a local account, however as this was a system which was bound to an Open Directory server on an OS X Snow Leopard server, and during the setup a local-only account was not setup.

When I looked into the issue there was some errors popping up in the Open Directory password service, mainly:

'username' DIGEST-MD5 authentication failed, SASL error -13

Now, this to me normally indicates there’s an issue between the client computer and the server with the Kerberos authentication, i.e. possibly an issue with the shared secret or similar – especially considering that the users password hadn’t changed, and the user was able to log into another machine without any issues.

Now, the solution, was actually somewhat simple. OS X Lion now sets up a Recovery Partition which you can access by booting while holding Command + R, or holding down the Option key and choosing the Recovery HD from the options.

Once you have booted into the Recovery Partition there is a menu titled Utilities - click that and choose Terminal, which will bring up the lovely world of the OS X Terminal, and thankfully there is one single command you need to enter: resetpassword

Entering this magical command will bring up a dialog asking you to choose which account you want to reset the password for – choose the root account, and then enter a password of your choosing, then simply reboot your computer.

Once you have rebooted you can then login as the root user, by selecting ‘other user’ and typing in the username and password – now you are logged in as an administrator account which will allow for you to recovery your login (hint: rename your account, including the login, maybe back up your data, then kind of create a new account and move the data back).

If your issue relates to the binding with Open Directory the solution is quite straight forward – in System Preferences, under Users & Accounts/Login Options edit the Open Directory bindings and remove your current one, then re-join the directory (you may want to remove the machine account from the directory in between these two steps). Once done simply reboot and you should be able to log in to your account again – woo hoo!

Good luck!

Footnote: this probably doesn’t need to be said, but don’t forget to back up your system before installing Lion – it will save you a world of trouble and panic.

Posted in Development, General, IT Consulting, User Experience on July 24, 2011 – 9:20 pm | Comments (1)
Tagged , , , |

On procrastination

By Matthew Savage

I’ve traveled for work a fair bit over the past three months – my job has seen me ending up in New Zealand (Auckland & Gisbourne), Scotland (Thurso, via Edinburgh, and then back to London for 2 days off) and in Sydney for a week around Dee Why. While I’ve been away for work I was hoping for some time to multitask – while my main requirement on site has been satisfied (i.e. everything is running smoothly, as expected etc) I had expected to be able to work on other projects, perhaps one specific project which is a little over-due.

Unfortunately this wasn’t how it played out…

I suppose our best laid plans don’t always happen as we expect them to, and in my case it was no different. However part of this was myself allowing minor hurdles stopping me from doing the work I had originally intended to do.

A bit of background

Let me paint the picture for you a bit – I am, as my main vocation, a software developer. Currently I work full-time for a company based in Sydney, but from my cosy apartment in Canberra. This company is based deeply in the surfing industry, and as a result we’re somewhat in events that pop up, mainly within the broadcasting side of things.

Remembering what I do daily – my position in these events is the broadcast technician – at the end of the day a fancy pants word for:

  • Setting up the webcast encoder
  • Setting up the webcast internet connection (and this varies from ADSL connections, 3G connections, a satellite connection – whatever it is, its my domain)
  • Setting up the audio mixer and tuning the levels on the Microphones and other inputs
  • Setting up the production mixing desk, video playout system, camera sources etc
  • Making sure everything plays nicely together
  • Ensuring those who need it can get access to the internet (this, believe me, is a huge problem up in Thurso…)
  • then when all that is up and running and no one is panicking… panicking over the things which need to be fixed and educating the runners
  • when required, be a runner – at a last resort

I’m not complaining about this though – working on the events is such an amazing experience. Normally I am not a morning person but when you get into these things where you’re up at 5am, finishing off at 9pm, rinse and repeat… its challenging but amazing.

But…

My expectations on how often I would be required, interrupted, etc – completely underestimated. Here I was imagining that I would get everything up and running, then be able to sit down and bash out some code until the next situation – and perhaps I could have.

Unfortunately I fell into the trap of laziness and things would pop up which I allowed to make it ‘hard’ to work on things. And these weren’t necessarily small things either – something like “oh the internet connection is unstable, so I can’t drop stories into Pivotal” or “I am not able to find a comfortable place where I won’t be interrupted, so….”. In hindsight its relatively pathetic.

The moral?

I’m (hopefully) not just blubbering uselessly about my trials and tribulations, I promise – there is a ‘moral to this story’. Recently I sat down with a business partner and discussed how I “didn’t have time to work on this project” – which was not only disappointing to my business partner, because he’s depending on me to deliver something, but it was also disappointing to me, because I was letting my friend down.

During our discussion we decided to setup a bit of a mind trick to get some work churned out. Originally I would say something like “Oh let me work on this tonight and I’ll come back to you in a day or two with it…” and lo and behold, I never delivered… When I was looking back on some things I had read by the guys at 37signals (in their book Getting Real – not an affiliate link, but the book, it rocks!) I remembered what DHH had said about his limited time while working on Basecamp – 10 hours per week, plus there was a time difference from the office being in Chicago and DHH being in Denmark.

With this in mind I decided to try something out – I had already had good experience working in sprints of 25 to 30 minutes, then taking a break, then starting again, so with my business partner we agreed to try something similar. Each day, for 1 hour, we sit down together and bash out some work. It doesn’t have to be perfect, in reality it could change the very next day, but the main thing is that we were making progress – taking that step forward required to gain momentum.

Has it worked?

So far, yes. There have been days, due to appointments etc, where we haven’t been able to meet as agreed, however most of the time we accomplish what we have aimed for and the progress is a good feeling. I still have issues with the procrastination bug at times, but at the end of the day there really is a reason why we procrastinate.

Looking back at the times I was away I could have easily done the ‘work’ I had intended to do while I was away. The reasons seem pitiful:

  • I can’t get a stable internet connection – just pick something to work on and churn out some code. If it isn’t right you can go and change it later but at least you have a baseline to work on – a clean slate is just intimidating and makes the problem worse.
  • There is nowhere comfortable to sit – Screw comfort – work until your butt goes numb, then take a break – or, perhaps find somewhere to sit down – there were heaps of cars around, even chairs – I’m sure you could have borrowed one.
  • If I sit in the car the satellite will lose its alignment – (This one is quite pathetic) The satellite’s by your computer, so it in the car, check the signal strength, and adjust the elevation for the difference your weight in the car makes.

I really wish I could go back and change them. I think that the reason I was procrastinating on this stuff is that I saw it as such as massive task at hand and couldn’t even get the momentum to break it up into smaller chunks, and see each iteration worked as an achievement, but now things are looking a bit different.

So next time you’re stuck perhaps try to keep this in mind. Any momentum is good momentum, and try not to get stuck looking at the big picture. If you are a software developer there is always going to be things you are going to have to look at, review and change, later in the future – so stop being such a perfectionist and just do the damn work.

Posted in General, Motivation on May 28, 2011 – 11:05 pm | Comments (0)

Quick Note: Crontabs on OpenWRT Devices (mainly: RobinMesh)

By Matthew Savage

seiko stopwatch

I have a few Ubiquiti Wireless Mesh AP devices which I look after for my local cafe’s (they keep me supplied with coffee, I keep them online… its a great deal). Recently I wanted to add a cron entry to one of these units to purge users from the captive portal at a certain time each day (we were finding that sessions weren’t being reset properly, and returning users had no luck getting online).

In a normal system this would simply entail adding an entry to the crontab at /etc/crontabs/root (yes, the cron processes run as root – unfortunately this is part of the firmware), however using the RobinMesh firmware I noticed that the crontabs were being reloaded from scratch each time a reboot ran.

Initially I thought the cron entries were coming out of UCI, however after a bit of digging I noticed that there are a few calls to ‘/lib/robin/setcron.sh’ from the init script at /etc/rc.d.

Looking into this script it seems that the cron entries are being loaded each time from this script, so adding my entry was as easy as adding in a new line to the exec block at the end, which is then piped into the crontab executable.

Simple. Hopefully this will save someone else from a headache. And myself, if I forget in the future.

Posted in General on March 14, 2011 – 2:15 pm | Comments (0)
Tagged , , , , |

Blocked numbers for business….. really?

By Matthew Savage

This is subjective, but I feel that blocked or restricted numbers should not be an option for businesses. Why should businesses have the right to hide behind a hidden number when contacting their customers or other businesses?

I can understand that in situations you don’t want customers having the DID number of the person they were talking to regarding an inquiry etc, but this is 2010, we can send out text messages with special names instead of numbers, Steve Jobs can make his name come up on your iPhone when he calls you… why can’t the teleco send the ‘calling from’ number as the main switch or IVR menu number?

It sure would make things a lot easier when someone leaves you voice mail, or you have a missed call – instead of scrabbling for a pen and paper or trying to remember that 1300 number, which options to press, the reference number, you can just hit ‘call back’ and you return to where you are supposed to be.

As I mentioned above, this is subjective, completely my opinion, but when companies wonder why so many people don’t answer their calls (banks or credit agencies, for instance), and they have a blocked number, you begin to think that common sense is lost once again.

The invention of Caller ID may have been controversial, questionable, but welcomed none the less, but at least don’t let businesses abuse the privilege of restricting their number – there really is no need for it.

Posted in Business, General, Gripes on December 3, 2010 – 1:47 pm | Comments (0)
Tagged , |

Params Filtering in Rails the custom way – masking the content

By Matthew Savage

For an app I have been working on credit card numbers are being submitted. Naturally a credit card number should be filtered out of the application logs – after all, you don’t want someone jumping onto you server and copying your logs to get hold of all those juicy credit card numbers? Nor do you want or need to go to the trouble of PCI-DSS Compliance.

I could have easily just setup my param filters as follows:

  # config/application.rb
  config.filter_parameters += [:password, :card_number, :cvn]

However, I want to have the masked card number available in part to be able to trace back any issues via the logs. So, to avoid the issues with the full card being presented I went about figuring out how to change a card number like 4444333322221111 into 4444••••••••1111.

It turns out that while a fair bit has changed with how you define which params should be filtered, the functionality is still available to pass in a proc to handle the param filtering. So, here’s how to do it:

Important: If you are going to use the • character in your filter you will need to add # encoding: utf-8 to the top of you application.rb file

  # config/application.rb
  card_number_filter = Proc.new do |key, value|
    if key.to_s =~ /card_number/i and value.present?
      value.replace("#{value[0,4]}#{"•" * (value.length - 8)}#{value[-4,4]}")
    end
  end
  config.filter_parameters += [:password, :cvn, card_number_filter]

See, it’s simple. One quick thing to note though – you need to replace the value object its self, not re-assign to it – you cant use assignment, such as value = "new_value", you need to use something which modifies the object in place, such as sub!, gsub!, downcase! or replace

So, a handy little nugget there – you can apply this to whatever you want to do with string manipulation – or more ;) .

Posted in Development, General on November 23, 2010 – 9:13 pm | Comments (0)
Tagged , , , , |

The Giant Automagical ImageMagick and Fonts install script

By Matthew Savage

Recently I’ve run into issues with inconsistent ImageMagick installations, which mainly manifest in the form of fonts for annotations etc not working properly, or missing delegates.

Finally I got fed up with repeating this drama every time, so in addition to updating my stack scripts I have put together a script which does all the ImageMagick stuff in one hit, for those one-off installs.

This script installs:

  • Ghostscript Fonts 8.11
  • Freetype 2.4.1
  • Fontconfig 2.8.0
  • Ghostscript 8.71
  • JPEG Lib 8b
  • Zlib 1.2.5
  • libTiff 3.8.2
  • libpng 1.4.3
  • and finally, ImageMagick 6.6.3-1

Download it here.

To run (with extra automagical):

wget -O- http://shares.amasses.net/image_magick_installer | sudo sh

To run (normally…):

chmod +x image_magick_installer
sudo ./image_magick_installer

And then to test the install:

convert -v

I have found that occasionally there are errors like:

error while loading shared libraries: libMagickCore.so.4: cannot open shared object file: No such file or directory

Try running the following, and if it works, put it into a .sh file in /etc/profile.d (e.g. /etc/profile.d/image_magick_ldd.sh):

#!/bin/sh

export LDFLAGS="-L/usr/local/lib -Wl,-rpath,/usr/local/lib"
export LD_LIBRARY_PATH="/usr/local/lib"
ldd /usr/local/bin/convert > /dev/null

Enjoy

Posted in Development, General on July 31, 2010 – 8:52 pm | Comments (11)
Tagged , , |

Is your build environment ‘sane’…?

By Matthew Savage

I’ve been testing a Varnish reverse-proxy/cache setup this weekend to try to resolve the ’80%’ of the problems I’ve been dealing with across our websites.

Because I seem to have an aversion to spending extra moneys to run up test servers (ok look, we’re a small business, and while EC2 is affordable I seem to have a problem where I forget to shut down our instances when I’m done with them – this can become expensive), so I’ve been running Ubuntu 9.10 and 10.04 in both Parallels and VMWare Fusion to test this varnish setup, and at the same time (I’ve been sidetracked halfway through) upgrading our production environments to Ubuntu 10.04.

In Parallels things were more or less fine, though I was running into some unexpected issues with running the CPU high (100%) while doing some cache-less bench testing. For whatever reason though I decided to swap over to using VMWare with 10.04 (who says we need consistency?).

The install of Ubuntu 10.04 is pretty much straight forward, but where I did run into problems was running my auto-build scripts which sets up our stack (REE, Passenger, nginx et al). Regardless of what I tried REE would always die during its install process, right on the ‘checking whether build environment is sane‘ test – in fact the exact error is:

checking whether build environment is sane… configure: error: newly created file is older than distributed files!
Check your system clock

Yes, this is a really really helpful error message. Especially when you check the time and verify that its correct, and synced up with NTP.

So lets look into this a little bit. The ‘sane’ check is doing a few things. It writes out the latest timestamp to a file, then it runs ls and checks the timestamp of the file – if the times are different or something breaks then the environment is not considered ‘sane’, this could be an issue with your coretools or your system clock or what ever… pick an option, then pull out your hair.

But, in my case – running on VMWare – my coretools were completely fine, but the system clock was a little… funky… even after kicking off an ntpupdate there was still some issues – a very very tiny miniscule offset, but enough to make my environment ‘insane’.

You can confirm this yourself actually, by running the following:

for i in `seq 1 60`
do
date
done

Take not of what the first time printed out is… then, 1 hour later, check the final time – it should be exactly the same, but one hour later. In my case I had 20:39:22, and at the end it was 21:39:23 – yes, its a whole 1 second, but in this case it made a difference.

So, whats the whole point to this? Well, I do have a fix, its actually really simple. Install the VMWare tools. No really, its that simple.

  1. From the VMware tools Virtual Machine menu, choose Install VMware Tools
  2. Run the following commands:
sudo -s
mount /dev/dvd /cdrom
cd /cdrom
tar zxvf VMwareTools-8.2.3-204229.tar.gz -C /tmp
cd /tmp/vmware-tools-distrib
./vmware-install.pl # (and follow the prompts)

Hopefully this will help anyone else also bashing their head against this brick wall, if not, m’eh, I don’t mind the writing ;)

Posted in General, IT Consulting on May 16, 2010 – 6:57 pm | Comments (0)
Tagged , , , |

‘Did you mean….’ style searching in your database

By Matthew Savage

I bet this has happened to everyone, you are in a rush, you mis-type something and instead of writing ‘polka-dotted socks‘ you’ve come up with ‘polka-dotted sokcs‘ – legitimate mistake, right?

Thankfully Google doesn’t rub it in our faces by showing 0 results or just plain laughing, and they offer the humble pie with a link saying ‘did you mean polka-dotted socks?’… so how do they do this?

There’s a few different options (phonetic algorithms) when it comes to working with text and normalizing down to a searchable index. The most common one is Soundex, which reduces a word like Coffee to C100, or Chimmichunga to C525 – these values can then be used as an index to search on – more or less using it as an equality comparison (e.g. Chimmychungah also reduces to C525, there for they sound alike).

If you’re working in your database there are a few options you can choose from – for MySQL you can use SOUNDEX("Coffee") to calculate the value on the server, or you can use a SOUNDS LIKE operator in your where clause. PostgreSQL offers similar options, with SOUNDEX("Chimmichunga") and also DIFFERENCE("Coffee", "Cafe") to find terms within a specific range. However (and this is merely speculation, I haven’t benchmarked to confirm this) there could be performance issues for large datasets compared to using a precomputed index column — the reasoning for this is that the database is doing a table-scan, hitting each row, computing the Soundex value and then comparing — not an efficient way to lookup records in a large dataset.

Of course, this is useful if you are using MySQL or Postgres, but what if you want something a bit more generic, or you have a large dataset? Well, why not setup precomputed indexes?

Take an example web application, you have a form to lookup a location but you don’t quite know how to spell it… is it Wollongong or Woolongong? If we assume an existing application the first step is to add a new column for the Soundex value to live in, and index it. For example, with a Rails application using Active Record I would do the following:

require 'text' # Provides access to the algorithms so we don't have to do the hard work (and possibly stuff it up!)

class AddSoundexIndexColumnToLocations < ActiveRecord::Migration
   def self.up
        add_column :locations, :soundex, :string
        add_index :locations, :soundex
         Locations.reset_column_information
         Locations.each do |location|
             location.update_attributes(:soundex => Text::Soundex.soundex(location.name))
        end
    end

    def self.down
        remove_index :locations, :soundex
        remove_column :locations, :soundex
    end
end

Once this is completed we now have a Locations table populated with the Name, other existing attributes, and now, the Soundex column. Then, searching is quite simple:

require 'text'

# Assuming that this is a postback and you have a form field or query param of name
@locations = Location.find(:all,
                                            :conditions => ["name = ? or soundex = ?", params[:name], Text::Soundex.soundex(params[:name])])

This is a very basic example, but it can be used with a number of ORMs or data storage options, i.e. you can use MongoMapper with an indexed key to lookup the values, or even load a YAML file into memory and scan through that (or XML, or JSON or… you get the idea). In actuality it should actually be 100% agnostic, at the end of the day it’s just checking equality.

Of course, the solution isn’t perfect, but its a beginning.

We have used something like this (albeit using the MySQL SOUNDS LIKE match instead of pre-filled indexes – currently we only have ~50 records to lookup) for our location searches and to help with hacked-urls (i.e. mis-spelt locations in the URL, such as /reports/balina instead of /reports/ballina).

In the case where we return a Soundex result instead of an exact match we also provide the visitor with some text explaining what has happened (e.g. ‘We couldn’t find Woolongong, but did you mean Wollongong?’) – but I’ll leave that up to you and what your user experience requires.

Enjoy, and feel free to shoot me a question if you have any issues making this stuff play nice, I’ll see what I can do to help.

Posted in Business, Development, General, User Experience on May 11, 2010 – 7:40 pm | Comments (0)

The 'hidden' costs of a Rails website

By Matthew Savage

I need to get something out on the ‘internets’ – running a website isn’t cheap.

Don’t get me wrong – you can get away with running a site on almost zero dollars, and we’ve all heard about those sites running on someone’s PC under their desk… but what I’m touching on here is a ‘real’ website.

I’m the only developer on a website which is somewhat large. Its no where near the level of the big news sites, or the social sites etc, but we get a fair amount of traffic and the codebase is big enough to keep me busy all the time. And to be honest my life would be much easier if I could just push out the latest version of the site and ignore everything else.

But lets be honest – large websites are complicated, there is always the change (more often then not) that there might be some bugs laying around, and on top of this performance can be an issue. And don’t even get me started on email and advertising. So what I now hope to accomplish is an inventory of all the ‘extras’ which seem to get over looked, but are quite important, for a production website.

Read More »

Posted in Business, Development, IT Consulting on April 29, 2010 – 9:38 am | Comments (0)
Tagged , , , , , , , |