In a Rails 3.0 or Rails 2.3 app, by default, a cache busting parameter is appended to the resource URL, for example: http://www.something.com/images/monkeys.jpg?138767865 – this number appended to the end if the mtime of the file from that server. When this was first put into place this was great, it worked as prescribed and was easy to manage, however there are some behaviours which this technique brings along with it:

• Some caching servers will not cache a URL with a query string
• The mtime of the file would be different across a web cluster, so you would have conflicting URLs and the cache wouldn’t work as expected

namespace :assets do
  task :precompile, :roles => :web do
    run "cd #{current_path} && RAILS_ENV=production bundle exec rake assets:precompile"
  end

  task :cleanup, :roles => :web do
    run "cd #{current_path} && RAILS_ENV=production bundle exec rake assets:clean"
  end
end

after :deploy, "assets:precompile"

There is also a cleanup task included to get rid of any hanging around assets which aren’t being used any more which you can run by calling cap assets:cleanup.

Hope this helps.

* There is one caveat, and that is with images referenced in CSS – if you are using SASS you can use image-url(...) which will output a url(...) directive with the correct URL, however in plain old CSS you should use url('<%= asset_path(...) %>'); and name the file with the .erb prefix to be preprocessed.

Just installed Lion and can’t log in? Boot into recovery, open terminal, enter resetpassword, reset root password, reboot, login as root, rebind to Open Directory, reboot, profit

The Long Story

I received a phone call on Friday from someone who made the decision to update to OS X Lion, and ran into a huge problem – unable to log into the computer… at all. In normal circumstances this wouldn’t be an issue, I would just have the user log in with a local account, however as this was a system which was bound to an Open Directory server on an OS X Snow Leopard server, and during the setup a local-only account was not setup.

When I looked into the issue there was some errors popping up in the Open Directory password service, mainly:

'username' DIGEST-MD5 authentication failed, SASL error -13

Now, this to me normally indicates there’s an issue between the client computer and the server with the Kerberos authentication, i.e. possibly an issue with the shared secret or similar – especially considering that the users password hadn’t changed, and the user was able to log into another machine without any issues.

Now, the solution, was actually somewhat simple. OS X Lion now sets up a Recovery Partition which you can access by booting while holding Command + R, or holding down the Option key and choosing the Recovery HD from the options.

Once you have booted into the Recovery Partition there is a menu titled Utilities - click that and choose Terminal, which will bring up the lovely world of the OS X Terminal, and thankfully there is one single command you need to enter: resetpassword

Entering this magical command will bring up a dialog asking you to choose which account you want to reset the password for – choose the root account, and then enter a password of your choosing, then simply reboot your computer.

Once you have rebooted you can then login as the root user, by selecting ‘other user’ and typing in the username and password – now you are logged in as an administrator account which will allow for you to recovery your login (hint: rename your account, including the login, maybe back up your data, then kind of create a new account and move the data back).

If your issue relates to the binding with Open Directory the solution is quite straight forward – in System Preferences, under Users & Accounts/Login Options edit the Open Directory bindings and remove your current one, then re-join the directory (you may want to remove the machine account from the directory in between these two steps). Once done simply reboot and you should be able to log in to your account again – woo hoo!

Good luck!

Footnote: this probably doesn’t need to be said, but don’t forget to back up your system before installing Lion – it will save you a world of trouble and panic.

Unfortunately this wasn’t how it played out…

I suppose our best laid plans don’t always happen as we expect them to, and in my case it was no different. However part of this was myself allowing minor hurdles stopping me from doing the work I had originally intended to do.

A bit of background

Let me paint the picture for you a bit – I am, as my main vocation, a software developer. Currently I work full-time for a company based in Sydney, but from my cosy apartment in Canberra. This company is based deeply in the surfing industry, and as a result we’re somewhat in events that pop up, mainly within the broadcasting side of things.

Remembering what I do daily – my position in these events is the broadcast technician – at the end of the day a fancy pants word for:

• Setting up the webcast encoder
• Setting up the webcast internet connection (and this varies from ADSL connections, 3G connections, a satellite connection – whatever it is, its my domain)
• Setting up the audio mixer and tuning the levels on the Microphones and other inputs
• Setting up the production mixing desk, video playout system, camera sources etc
• Making sure everything plays nicely together
• Ensuring those who need it can get access to the internet (this, believe me, is a huge problem up in Thurso…)
• then when all that is up and running and no one is panicking… panicking over the things which need to be fixed and educating the runners
• when required, be a runner – at a last resort

I’m not complaining about this though – working on the events is such an amazing experience. Normally I am not a morning person but when you get into these things where you’re up at 5am, finishing off at 9pm, rinse and repeat… its challenging but amazing.

But…

My expectations on how often I would be required, interrupted, etc – completely underestimated. Here I was imagining that I would get everything up and running, then be able to sit down and bash out some code until the next situation – and perhaps I could have.

Unfortunately I fell into the trap of laziness and things would pop up which I allowed to make it ‘hard’ to work on things. And these weren’t necessarily small things either – something like “oh the internet connection is unstable, so I can’t drop stories into Pivotal” or “I am not able to find a comfortable place where I won’t be interrupted, so….”. In hindsight its relatively pathetic.

The moral?

I’m (hopefully) not just blubbering uselessly about my trials and tribulations, I promise – there is a ‘moral to this story’. Recently I sat down with a business partner and discussed how I “didn’t have time to work on this project” – which was not only disappointing to my business partner, because he’s depending on me to deliver something, but it was also disappointing to me, because I was letting my friend down.

During our discussion we decided to setup a bit of a mind trick to get some work churned out. Originally I would say something like “Oh let me work on this tonight and I’ll come back to you in a day or two with it…” and lo and behold, I never delivered… When I was looking back on some things I had read by the guys at 37signals (in their book Getting Real – not an affiliate link, but the book, it rocks!) I remembered what DHH had said about his limited time while working on Basecamp – 10 hours per week, plus there was a time difference from the office being in Chicago and DHH being in Denmark.

With this in mind I decided to try something out – I had already had good experience working in sprints of 25 to 30 minutes, then taking a break, then starting again, so with my business partner we agreed to try something similar. Each day, for 1 hour, we sit down together and bash out some work. It doesn’t have to be perfect, in reality it could change the very next day, but the main thing is that we were making progress – taking that step forward required to gain momentum.

Has it worked?

So far, yes. There have been days, due to appointments etc, where we haven’t been able to meet as agreed, however most of the time we accomplish what we have aimed for and the progress is a good feeling. I still have issues with the procrastination bug at times, but at the end of the day there really is a reason why we procrastinate.

Looking back at the times I was away I could have easily done the ‘work’ I had intended to do while I was away. The reasons seem pitiful:

• I can’t get a stable internet connection – just pick something to work on and churn out some code. If it isn’t right you can go and change it later but at least you have a baseline to work on – a clean slate is just intimidating and makes the problem worse.
• There is nowhere comfortable to sit – Screw comfort – work until your butt goes numb, then take a break – or, perhaps find somewhere to sit down – there were heaps of cars around, even chairs – I’m sure you could have borrowed one.
• If I sit in the car the satellite will lose its alignment – (This one is quite pathetic) The satellite’s by your computer, so it in the car, check the signal strength, and adjust the elevation for the difference your weight in the car makes.

I really wish I could go back and change them. I think that the reason I was procrastinating on this stuff is that I saw it as such as massive task at hand and couldn’t even get the momentum to break it up into smaller chunks, and see each iteration worked as an achievement, but now things are looking a bit different.

So next time you’re stuck perhaps try to keep this in mind. Any momentum is good momentum, and try not to get stuck looking at the big picture. If you are a software developer there is always going to be things you are going to have to look at, review and change, later in the future – so stop being such a perfectionist and just do the damn work.

I have a few Ubiquiti Wireless Mesh AP devices which I look after for my local cafe’s (they keep me supplied with coffee, I keep them online… its a great deal). Recently I wanted to add a cron entry to one of these units to purge users from the captive portal at a certain time each day (we were finding that sessions weren’t being reset properly, and returning users had no luck getting online).

In a normal system this would simply entail adding an entry to the crontab at /etc/crontabs/root (yes, the cron processes run as root – unfortunately this is part of the firmware), however using the RobinMesh firmware I noticed that the crontabs were being reloaded from scratch each time a reboot ran.

Initially I thought the cron entries were coming out of UCI, however after a bit of digging I noticed that there are a few calls to ‘/lib/robin/setcron.sh’ from the init script at /etc/rc.d.

Looking into this script it seems that the cron entries are being loaded each time from this script, so adding my entry was as easy as adding in a new line to the exec block at the end, which is then piped into the crontab executable.

Simple. Hopefully this will save someone else from a headache. And myself, if I forget in the future.

I can understand that in situations you don’t want customers having the DID number of the person they were talking to regarding an inquiry etc, but this is 2010, we can send out text messages with special names instead of numbers, Steve Jobs can make his name come up on your iPhone when he calls you… why can’t the teleco send the ‘calling from’ number as the main switch or IVR menu number?

It sure would make things a lot easier when someone leaves you voice mail, or you have a missed call – instead of scrabbling for a pen and paper or trying to remember that 1300 number, which options to press, the reference number, you can just hit ‘call back’ and you return to where you are supposed to be.

As I mentioned above, this is subjective, completely my opinion, but when companies wonder why so many people don’t answer their calls (banks or credit agencies, for instance), and they have a blocked number, you begin to think that common sense is lost once again.

The invention of Caller ID may have been controversial, questionable, but welcomed none the less, but at least don’t let businesses abuse the privilege of restricting their number – there really is no need for it.

I could have easily just setup my param filters as follows:

  # config/application.rb
  config.filter_parameters += [:password, :card_number, :cvn]

However, I want to have the masked card number available in part to be able to trace back any issues via the logs. So, to avoid the issues with the full card being presented I went about figuring out how to change a card number like 4444333322221111 into 4444••••••••1111.

It turns out that while a fair bit has changed with how you define which params should be filtered, the functionality is still available to pass in a proc to handle the param filtering. So, here’s how to do it:

Important: If you are going to use the • character in your filter you will need to add # encoding: utf-8 to the top of you application.rb file

  # config/application.rb
  card_number_filter = Proc.new do |key, value|
    if key.to_s =~ /card_number/i and value.present?
      value.replace("#{value[0,4]}#{"•" * (value.length - 8)}#{value[-4,4]}")
    end
  end
  config.filter_parameters += [:password, :cvn, card_number_filter]

See, it’s simple. One quick thing to note though – you need to replace the value object its self, not re-assign to it – you cant use assignment, such as value = "new_value", you need to use something which modifies the object in place, such as sub!, gsub!, downcase! or replace

So, a handy little nugget there – you can apply this to whatever you want to do with string manipulation – or more .

Finally I got fed up with repeating this drama every time, so in addition to updating my stack scripts I have put together a script which does all the ImageMagick stuff in one hit, for those one-off installs.

This script installs:

• Ghostscript Fonts 8.11
• Freetype 2.4.1
• Fontconfig 2.8.0
• Ghostscript 8.71
• JPEG Lib 8b
• Zlib 1.2.5
• libTiff 3.8.2
• libpng 1.4.3
• and finally, ImageMagick 6.6.3-1

Download it here.

To run (with extra automagical):

wget -O- http://shares.amasses.net/image_magick_installer | sudo sh

To run (normally…):

chmod +x image_magick_installer
sudo ./image_magick_installer

And then to test the install:

convert -v

I have found that occasionally there are errors like:

error while loading shared libraries: libMagickCore.so.4: cannot open shared object file: No such file or directory

Try running the following, and if it works, put it into a .sh file in /etc/profile.d (e.g. /etc/profile.d/image_magick_ldd.sh):

#!/bin/sh

export LDFLAGS="-L/usr/local/lib -Wl,-rpath,/usr/local/lib"
export LD_LIBRARY_PATH="/usr/local/lib"
ldd /usr/local/bin/convert > /dev/null

Enjoy

Because I seem to have an aversion to spending extra moneys to run up test servers (ok look, we’re a small business, and while EC2 is affordable I seem to have a problem where I forget to shut down our instances when I’m done with them – this can become expensive), so I’ve been running Ubuntu 9.10 and 10.04 in both Parallels and VMWare Fusion to test this varnish setup, and at the same time (I’ve been sidetracked halfway through) upgrading our production environments to Ubuntu 10.04.

In Parallels things were more or less fine, though I was running into some unexpected issues with running the CPU high (100%) while doing some cache-less bench testing. For whatever reason though I decided to swap over to using VMWare with 10.04 (who says we need consistency?).

The install of Ubuntu 10.04 is pretty much straight forward, but where I did run into problems was running my auto-build scripts which sets up our stack (REE, Passenger, nginx et al). Regardless of what I tried REE would always die during its install process, right on the ‘checking whether build environment is sane‘ test – in fact the exact error is:

checking whether build environment is sane… configure: error: newly created file is older than distributed files!
Check your system clock

Yes, this is a really really helpful error message. Especially when you check the time and verify that its correct, and synced up with NTP.

So lets look into this a little bit. The ‘sane’ check is doing a few things. It writes out the latest timestamp to a file, then it runs ls and checks the timestamp of the file – if the times are different or something breaks then the environment is not considered ‘sane’, this could be an issue with your coretools or your system clock or what ever… pick an option, then pull out your hair.

But, in my case – running on VMWare – my coretools were completely fine, but the system clock was a little… funky… even after kicking off an ntpupdate there was still some issues – a very very tiny miniscule offset, but enough to make my environment ‘insane’.

You can confirm this yourself actually, by running the following:

for i in `seq 1 60`
do
date
done

Take not of what the first time printed out is… then, 1 hour later, check the final time – it should be exactly the same, but one hour later. In my case I had 20:39:22, and at the end it was 21:39:23 – yes, its a whole 1 second, but in this case it made a difference.

So, whats the whole point to this? Well, I do have a fix, its actually really simple. Install the VMWare tools. No really, its that simple.

1. From the VMware tools Virtual Machine menu, choose Install VMware Tools
2. Run the following commands:

sudo -s
mount /dev/dvd /cdrom
cd /cdrom
tar zxvf VMwareTools-8.2.3-204229.tar.gz -C /tmp
cd /tmp/vmware-tools-distrib
./vmware-install.pl # (and follow the prompts)

Hopefully this will help anyone else also bashing their head against this brick wall, if not, m’eh, I don’t mind the writing

Thankfully Google doesn’t rub it in our faces by showing 0 results or just plain laughing, and they offer the humble pie with a link saying ‘did you mean polka-dotted socks?’… so how do they do this?

There’s a few different options (phonetic algorithms) when it comes to working with text and normalizing down to a searchable index. The most common one is Soundex, which reduces a word like Coffee to C100, or Chimmichunga to C525 – these values can then be used as an index to search on – more or less using it as an equality comparison (e.g. Chimmychungah also reduces to C525, there for they sound alike).

If you’re working in your database there are a few options you can choose from – for MySQL you can use SOUNDEX("Coffee") to calculate the value on the server, or you can use a SOUNDS LIKE operator in your where clause. PostgreSQL offers similar options, with SOUNDEX("Chimmichunga") and also DIFFERENCE("Coffee", "Cafe") to find terms within a specific range. However (and this is merely speculation, I haven’t benchmarked to confirm this) there could be performance issues for large datasets compared to using a precomputed index column — the reasoning for this is that the database is doing a table-scan, hitting each row, computing the Soundex value and then comparing — not an efficient way to lookup records in a large dataset.

Of course, this is useful if you are using MySQL or Postgres, but what if you want something a bit more generic, or you have a large dataset? Well, why not setup precomputed indexes?

Take an example web application, you have a form to lookup a location but you don’t quite know how to spell it… is it Wollongong or Woolongong? If we assume an existing application the first step is to add a new column for the Soundex value to live in, and index it. For example, with a Rails application using Active Record I would do the following:

require 'text' # Provides access to the algorithms so we don't have to do the hard work (and possibly stuff it up!)

class AddSoundexIndexColumnToLocations < ActiveRecord::Migration
   def self.up
        add_column :locations, :soundex, :string
        add_index :locations, :soundex
         Locations.reset_column_information
         Locations.each do |location|
             location.update_attributes(:soundex => Text::Soundex.soundex(location.name))
        end
    end

    def self.down
        remove_index :locations, :soundex
        remove_column :locations, :soundex
    end
end

Once this is completed we now have a Locations table populated with the Name, other existing attributes, and now, the Soundex column. Then, searching is quite simple:

require 'text'

# Assuming that this is a postback and you have a form field or query param of name
@locations = Location.find(:all,
                                            :conditions => ["name = ? or soundex = ?", params[:name], Text::Soundex.soundex(params[:name])])

This is a very basic example, but it can be used with a number of ORMs or data storage options, i.e. you can use MongoMapper with an indexed key to lookup the values, or even load a YAML file into memory and scan through that (or XML, or JSON or… you get the idea). In actuality it should actually be 100% agnostic, at the end of the day it’s just checking equality.

Of course, the solution isn’t perfect, but its a beginning.

We have used something like this (albeit using the MySQL SOUNDS LIKE match instead of pre-filled indexes – currently we only have ~50 records to lookup) for our location searches and to help with hacked-urls (i.e. mis-spelt locations in the URL, such as /reports/balina instead of /reports/ballina).

In the case where we return a Soundex result instead of an exact match we also provide the visitor with some text explaining what has happened (e.g. ‘We couldn’t find Woolongong, but did you mean Wollongong?’) – but I’ll leave that up to you and what your user experience requires.

Enjoy, and feel free to shoot me a question if you have any issues making this stuff play nice, I’ll see what I can do to help.

Don’t get me wrong – you can get away with running a site on almost zero dollars, and we’ve all heard about those sites running on someone’s PC under their desk… but what I’m touching on here is a ‘real’ website.

I’m the only developer on a website which is somewhat large. Its no where near the level of the big news sites, or the social sites etc, but we get a fair amount of traffic and the codebase is big enough to keep me busy all the time. And to be honest my life would be much easier if I could just push out the latest version of the site and ignore everything else.

But lets be honest – large websites are complicated, there is always the change (more often then not) that there might be some bugs laying around, and on top of this performance can be an issue. And don’t even get me started on email and advertising. So what I now hope to accomplish is an inventory of all the ‘extras’ which seem to get over looked, but are quite important, for a production website.

Exception Handling

No one likes to see error messages, but they’re unfortunately one of those things we have to live with – lets be honest, no one is perfect (I’m certainly not) – but even though we cant guarantee that errors won’t be a problem we should be ready to handle them properly and log them, so when the time comes and you have angry users complaining that your website isn’t working you can be rest-assured that you have any error details available to you without the hassle of asking the user what the error message was, what they were doing, what color t-shirt they had on… you get the picture.

There are a couple of options here, but the one I use for everything is HopToad from Thoughtbot – this is an extremely simple tool to use. Once your signed up online they give you some quick instructions which includes installing a plugin into your Rails application, then running a generator with an api key for your project, and then thats it – once you go live and if an error occurs Hoptoad sends you a nice email and makes all the details, including a backtrace, the URL the error happened at, the referring page, the environment variables etc, online and within reach.

There are, of course, free alternatives around, and if you really wanted you could just have your app email you when an error occurs, but I’m sure most people can relate to the situation where your inbox becomes flooded because you forgot to check for a nil? Yeah, Hoptoad prevents this exact problem.

And the cost? $15/month, for 16 projects – very affordable for what it provides, and there’s some extra bonuses you’ll stumble upon which make you think ‘wow this is worth it’.

Application Performance

So, errors we’ve covered. Next question – do you know how well your website is running? How do you know which areas are the slowest to run, and do you know what the user experience is like?

On this front as well there are a few options – you could quite easily run Request Log Analyzer and pass it your latest production log file, but thats not really a real time view, and if you want to look through historical periods its somewhat difficult, involving splitting out your logs based on time stamps – not my idea of fun.

RPM to the rescue – NewRelic RPM to be exact. I suppose that this is probably the best known ‘Rails’ application around. Mind you, its not cheap, but for what it provides its actually invaluable for popular websites.  And what does it provide? Real time metrics, drill down trouble shooting, pretty graphs and more. Really, I’d just recommend that you give their free version a go – I’d wager that you will want to upgrade to get the extra features once you get a taste.

And on top of providing a great product they also sponsored the Scaling Rails screen cast series with Gregg Pollack – I’d highly recommend everyone checks out this series.

So how much will this set you back? It varies based on pricing level, but currently we pay about $200/month – and its worth every cent.

Server Monitoring

Now we have errors covered off and application performance is taken care of what about your hosting infrastructure? If your using a Rails hosting specialist like Engine Yard or Rails Machine or similar then they’ve got this covered, skip on to the next point, but if you’ve rolled your own, be it with your own hardware, or on cloud infrastructure like EC2, it would be very wize to start collecting some metrics about how your machines are actually running.

Of course, there is a myriad of options here, so let me just focus on what we use, which is Server Density – these guys make monitoring, alerting and metric collection as simple as making a sandwich (well, as long as it isn’t some fancy sandwich which requires some weird condiment you can only get by milking deep sea predator fish…). Sign up for an account, setup a server in their system, and once you’ve installed the Server Density agent, drop in the config details and wait.

Its extremely easy to setup alerts based on thresholds (such as memory free, CPU load, number of request for nginx or apache, even connections to MySQL or MongoDB), and they support notifications via Email, SMS and iPhone push as well (though to be honest, if you publish something controversial which generates some traffic the alerts can become a bit irritating). And if you want to monitor something they don’t support, and your a bit of a dab hand at Python, they have a brand spanking new plugin framework for you to use.

This has saved me a few times, especially being able to look back in time, see a spike, check out what was happening on my system at that time, and the simplicity is just awesome. Plus, they’ve been public adopters of MongoDB for their data collection and feedback to the community with their experiences.

Server Density – $10 (per server), being able to tell your boss that the servers aren’t really on fire… priceless (yeah thats kind of lame, sorry).

Email

I’ve unfortunately discovered, more often then not, that sending emails from web applications is a bit of a thorn in the side (sure, Rails makes this a lot easier with ActionMailer providing views etc). Its not actually the sending thats the issue – anyone can put together some content and use a library like tmail, Pony, even ActionMailer to shoot off a message – the real issue is dealing with delivery problems.

Running on Amazon EC2, while amazingly flexible and cost effective, has a slight issue – the same things that make EC2 appealing to me also make EC2 appealing to spammers. And as a result there are a lot of RBLs, block lists etc, which don’t like emails from the EC2 address space. Plus, PTR records, you have no control over them.

So why risk the emails not being sent? And what about bounces – soft and hard, plus do you know what happened with the email after it was received?

I was initially introduced to SendGrid through Heroku (that is another topic all together) – before that I was hacking about with smtp-tls and Google Apps SMTP to get my emails out. For the site that I work on now there were two options – use the SMTP server provided by the hosting company (before we moved to EC2), setup our own MTA, or use an external service. The decision time there was less then a second – straight away I was on the SendGrid page and registering for the company.

SendGrid makes things simple – I tell my app which SMTP settings to use, which authenticate, of course, and then I get everything I need from SendGrid straight away – I can turn on open and click tracking, so I know if users are ‘doing things’ with the emails we send out, we have domains keys enabled automatically, plus we also get to see if there are any bounces of emails, and compare that to our outgoing amount (more shiny graphs).

And the total cost for this is $9 per month - perfect.

To summarize this hasn’t covered everything we pay for externally – there are tools like Basecamp (for the higher level project management and collaboration), Unfuddle (for the development project management), Zendesk (for helpdesk.. those pesky users) and MailChimp (for email campaigns) we use, but its a case of ‘different strokes for different folks’ – but what I’ve mentioned above are just the basics.

The real take away message here is that there are always other things you need to factor in when your considering when you figure out how much running a website is actually going to cost you. And if your wondering why you don’t just build this stuff your self.. well, sure, I’m not stopping you, I just feel that most times its smarter to pay for something already available rather then spending time reinventing the wheel, only to realize what you finished up with is missing a whole lot of stuff.