Flash 10 On 64bit Debian

2010-05-22T00:00:00+10:00

Debian recently removed flash from their AMD64 to security issues and Adobe dropping official support.

I dislike flash as much as the next self respecting web developer, but some site (like google analytics) are pretty useless without it.

If you need a temporary solution to getting it up and running, manually download and install the old flash9 package from backports

    wget http://backports.mithril-linux.org/pool/contrib/f/flashplugin-nonfree/flashplugin-nonfree_1.4~bpo40+1_amd64.deb
    sudo dpkg -i flashplugin-nonfree_1.4~bpo40+1_amd64.deb

For further reading, check out the backports page for flashplugin-nonfree.

It’s ye olde, but it works.

New Mysql Ruby Bindings

2010-05-20T00:00:00+10:00

There’s been some recent discussions about the way ruby 1.9 handles encodings and the exceptions people are running into, particularly when running rails.

A lot of the time, the exceptions are caused by database drivers passing back multi-byte strings incorrectly tagged as ASCII-8BIT instead of UTF-8.

The work around was ususally to do something like the monkey patch described here. I’ve used that particular patch in production for a few months and it works fine, but monkey patches make me squirm.

This week I was finally able to replace the patch with an alternative mysql gem that retrieves all strings from the database as utf-8 and marks them correctly.

Brian Mario has released mysql2. The basic API is different to the standard mysql gem, but he’s included an ActiveRecord adapter that acts as a drop-in replacement for the mysql one.

To give mysql2 a go in rails install the gem:

    gem install mysql2

Then update your database.yml:

    development:
      adapter: mysql2
      database: myapp_dev
      username: foo
      password: bar
      host: localhost

Make sure you get mysql2 version 0.1.6 or higher, in lower versions the ActiveRecord adaptor behaved slightly differently from the default Mysql one.

Brian is incredibly responsive to bug reports, so if you have any issues make sure you report them on the github project.

Ethical Superannuation

2010-01-16T00:00:00+11:00

For all of my (admittedly short) professional life, my superannuation has been in Australian Super’s Sustainable Balanced fund.

I was more or less happy with this. Conventional wisdom says that non-profit industry super funds tend to perform better than commercial funds due to lower fees, and the sustainable option meant my money wasn’t being invested in too many companies I disagreed with.

However, I recently received a letter from Australian Super informing me of an upcoming change in their sustainable investment policies. The key paragraph was:

In the past our investment manager excluded companies that it believed to
receive a significant proportion of their revenue (at least 5%) from the
manufacture of sale of alcohol or tobacco, the operation of gaming
facilities or the manufacture of gambling equipment, uranium extraction or
the manufacture of weapons or armaments. This automatic exclusion will no
longer take place.

Hah. Imagine that. Presumably they were sick of missing investment returns from companies like BHP that are involved in uranium.

Whatever their reasoning, chances are my money is now being invested in places I’d prefer it not to be, so I’ve been researching my options.

The obvious alternative is Australian Ethical Super. Their investment policies gel strongly with my preferences and my non-super investments with them have been performing well.

Being a commercial fund, a comparison of fees was in order. The results were surprising and a little disappointing.

Australian Super’s Fees:

$78/year administration costs
0.77% of account balance
0% of contributions

Australian Ethical Fees:

$41/year administration costs
2.15% of account balance, calculated weekly
4.1% of contributions

For the 08-09 financial year, Australian Ethical fees for my super balance would’ve been a little more than triple what I actually paid Australian Super. Bugger.

All this is ignoring historical returns for each fund, a topic that would demand an entirely separate blog post. The short of it is that over the previous 5 years, Australian Ethical seems to have performed worse on comparable funds.

So do I move? Are there other ethical super options out there? What premium am I willing to cop in return for having my money invested in ethical companies?

Intro To The Pdf File Structure

2009-11-12T00:00:00+11:00

For a couple of years I’ve been involved in various PDF related Ruby projects - most recently in prawn, a pure ruby PDF generation library.

Prawn is a long way from implementing all parts of the spec and there has been no shortage of people who desire support for various features. Many of these users are the kind who will submit a patch to an open source project when they find a missing feature, yet this has only happened to prawn in a small number of cases.

The PDF spec is epic (version 1.7 is over 1300 pages), yet the basic structure is less complicated than it first appears. Don’t be put off! We’d love more contributors to prawn, so this post is intended to be a basic introduction to reading a PDF file - the first step in understanding how prawn does the reverse process. I’ve created a simple PDF to use as an example.

The best way to think of the file is a set of objects (strings, hashes, symbols, integers, arrays, etc) in an object tree that descends from a single “root” hash.

To start navigating the example file by eye, open it in a text editor and browse to the last line. Just above the end of the file token (%%EOF) is a number that indicates a byte offset in the file.

That byte offset points to the “xref” token on line 46. Everything between here and the “trailer” token several lines below is called the Cross Reference Table (xref table to its friends). “0 6” indicates that this file has 5 objects in it numbered 1-5 (plus a number 0 null object). The following lines indicate object 1 is at byte offset 15, object 2 at offset 71, etc.

After the byte offsets you’ll see the following lines:

    << /Info 1 0 R
    /Size 6
    /Root 3 0 R
    >>

This is called the trailer and it is a serialised hashmap. In Ruby syntax, this would look something like:

    {
      :Info => "1 0 R",
      :Size => 6,
      :Root => "3 0 R"
    }

This is the top of our object tree. “1 0 R” is the syntax for pointing to object #1. Using our xref table we can see object 1 is at byte 17 and it looks like this:

    << /Creator (Prawn)
    /Producer (Prawn)
    >>

It’s another hash that looks like this in Ruby:

    { :Creator => "Prawn", :Producer => "Prawn" }

Merge this with our trailer hash and our object tree now looks like:

    {
      :Info => { :Creator => "Prawn", :Producer => "Prawn" },
      :Size => 6,
      :Root => "3 0 R"
    }

Repeat this process for every time you see “n 0 R” (called an indirect object) and you will have a complete PDF object graph.

Prawn’s job is to do the reverse - build an object graph that conforms to the spec and serialise it to a file in the correct format. Prawn has all the tools for serialising standard ruby objects into their PDF representation, so the trick is to find the correct place to add your new information (an extra font, an advanced colour definition, whatever) and insert it as a ruby object. Usually this means something like adding an entry to an array or hash.

Once that’s done, prawn will handle the serialisation when a user calls render.

For homework, download a copy of the PDF spec and read chapter 7 (particularly sections 7.3, 7.5, 7.7, 7.8 and 7.9). Also check out lib/prawn/pdf_object.rb to see how prawn converts each Ruby object into PDF equivalents.

Partition Resizing

2009-11-05T00:00:00+11:00

This is for my own reference as much as anyone elses.

For years I’ve used an old copy of Partition Magic when I needed to resize live partitions, usually as a step towards making a computer dual boot Windows and Linux.

Last week I purchased a new laptop, only to find partition magic failed to even load. I suspect the 320Gb SATA drive had new geometry that freaked Partition Magic out with your standard cryptic error message (“error 117”).

Not wanting to pay for a tool I use so irregularly, I hunted around for a free alternative and found one that worked nicely - Easeus Partition Master. The home edition is free for personal use.

Dell E4300 And Linux

2009-11-05T00:00:00+11:00

After 3.5 years of trusty service, it was time to retire my Dell Latitude D620. It was out of warranty, wouldn’t charge batteries and with a NVidia graphics card it didn’t play nice with Linux.

After considering my options (mac? lenovo?) I decided to stick with Dell. I’ve used their Latitude line for some time and enjoy the ruggedness, backed by the multi-year, next business day, onsite warranty. As a freelance web guy, it’s hard to beat having a tech visit you with 24 hours to fix a broken machine.

Wanting something slightly smaller than the D620, I settled on the Latitude E4300.

I’m a Debian guy, and had no trouble installing the testing distrubution from the netinstall CD, then dist-upgrading to unstable.

Hardware compatability is excellent - here’s what lspci -nn tells me:

    [jh@gaz Desktop]$ lspci -nn
    00:00.0 Host bridge [0600]: Intel Corporation Mobile 4 Series Chipset Memory Controller Hub [8086:2a40] (rev 07)
    00:02.0 VGA compatible controller [0300]: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller [8086:2a42] (rev 07)
    00:02.1 Display controller [0380]: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller [8086:2a43] (rev 07)
    00:19.0 Ethernet controller [0200]: Intel Corporation 82567LM Gigabit Network Connection [8086:10f5] (rev 03)
    00:1a.0 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] (rev 03)
    00:1a.1 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] (rev 03)
    00:1a.2 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 [8086:2939] (rev 03)
    00:1a.7 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] (rev 03)
    00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03)
    00:1c.0 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 [8086:2940] (rev 03)
    00:1c.1 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 2 [8086:2942] (rev 03)
    00:1c.3 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 4 [8086:2946] (rev 03)
    00:1d.0 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 03)
    00:1d.1 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] (rev 03)
    00:1d.2 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] (rev 03)
    00:1d.7 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03)
    00:1e.0 PCI bridge [0604]: Intel Corporation 82801 Mobile PCI Bridge [8086:2448] (rev 93)
    00:1f.0 ISA bridge [0601]: Intel Corporation ICH9M-E LPC Interface Controller [8086:2917] (rev 03)
    00:1f.2 RAID bus controller [0104]: Intel Corporation Mobile 82801 SATA RAID Controller [8086:282a] (rev 03)
    00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 03)
    02:01.0 FireWire (IEEE 1394) [0c00]: Ricoh Co Ltd R5C832 IEEE 1394 Controller [1180:0832] (rev 05)
    02:01.1 SD Host controller [0805]: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter [1180:0822] (rev 22)
    02:01.2 System peripheral [0880]: Ricoh Co Ltd R5C843 MMC Host Controller [1180:0843] (rev ff)
    0c:00.0 Network controller [0280]: Intel Corporation PRO/Wireless 5300 AGN [Shiloh] Network Connection [8086:4235]

The following is a summary of major components and how they performed, along with any work I had to perform to get them working. I was absolutely blown away by how everything just worked. Linux compatability for Linux has come a long way in the last 3 years.

Display/Graphics

The Intel GS45 Express graphics card works fine in xorg with the xserver-xorg-video-intel package. No /etc/X11/xorg.conf file is necesary.

Thanks to the open source intel drivers, xrandr 1.2 works really well. Using a package like arandr, you can add or remove external monitors and change resolutions, all without restarting X.

Wired Ethernet

The 82567LM Intel gigabit network card works with the e1000 kernel driver.

Wireless Network

The 5300 Intel wireless card works once the firmware-iwlwifi package has been installed. This is a non-free package, make sure you have added non-free to your apt sources.

Trackpad

The trackpad works under X with no customisations.

Power

Suspend, resume and hibernate all work fine. A refreshing change after 3 years of having no support for these at all. I suspect the binary NVidia drivers were to blame, so I’m more than happy with my decision to get Intel hardware.

Audio

Alsa audio worked with no customisations. I did need to unmute the default channel.

Firewire

The firewire port worked with no customisations.

Kernel

I upgraded to the current kernel in Sid (2.6.31+22) with no issues.

Prawn And X Accel Redirect

2009-05-30T00:00:00+10:00

Prawn is a pure Ruby PDF generation library. PrawnTo is a rails plugin that makes it dead easy to add PDF views to standard rails actions.

I needed to generate a significant number of PDF reports in an app I support and although Prawnto makes things simple I had a few requirements that meant it wasn’t a good fit:

I wanted to share my reports across several projects
I wanted to be able to generate PDFs in situations that weren’t responses to a HTTP request. For example, for attaching to an email or as part of a background job
I didn’t want downloads of large PDFs to tie up one of my rails processes

With these in mind I came up with an alternative (and slightly more complex) technique for using Prawn in my app. The key features are storing the report definitions in their own directory (to allow sharing between projects using git sub modules) and using nginx’s X-Accel-Redirect feature to let nginx handle streaming the generated file to the client. You can read a little about this feature in the nginx wiki. Apache has a similar feature called X-Sendfile that is roughly comparable.

There are a number of steps involved in getting this technique running, so I will assume you are familiar with both Prawn, nginx and Rails. I have code snippets inline, or you can view a full sample application at github.

Start by editing your config/enviroment.rb file to add the following three lines. The first tells Rails about the new directory we’ll use to store our reports and the next 2 load prawn.

    config.load_paths += %W( #{RAILS_ROOT}/app/reports )
    config.gem "prawn", :version => "0.4.1", :lib => "prawn"
    config.gem "prawn-format", :version => "0.1.1", :lib => "prawn/format"

Next, edit or create config/initializers/mime_types.rb and add a line to register the PDF mime type:

    Mime::Type.register "application/pdf", :pdf

Now, create a new class in app/reports/application_report.rb. This class will be a superclass for all the reports you will be generating.

    require 'pathname'

    class ApplicationReport

      attr_reader :path

      def render
        self.__send__(:render_header) if self.respond_to?(:render_header)
        self.__send__(:render_body)   if self.respond_to?(:render_body)
        self.__send__(:render_footer) if self.respond_to?(:render_footer)
        File.open(tmp_file, "w") { |f| f.write pdf.render }
        self
      end

      private

      def pdf(opts = {})
        @path ||= tmp_file
        @pdf ||= ::Prawn::Document.new(opts)
      end

      # choose a file to render our PDF to before sending it to the user
      def tmp_file
        counter = 1
        path = nil
        dir = RAILS_ROOT + "/tmp/pdfs"
        FileUtils.mkdir_p(dir)
        dir = Pathname.new(dir).realpath
        while path.nil? || File.file?(path)
          path = "#{dir}/pdf-#{counter}"
          counter += 1
        end
        path
      end
    end

Now, create a subclass of ApplicationReport in app/reports/product_report.rb. This will be your first real report. I’ve created a sample one here called ProductReport to display the basic information from my Product model. You will need to modify it to suit your application.

    class ProductReport < ApplicationReport
      def initialize(product)
        @product = product
      end

      def render_header
        pdf.font_size(16) do
          pdf.text "<b>#{@product.description}</b>"
        end
      end

      def render_body
        pdf.font_size(12) do
          pdf.text "<b>Code: </b>#{@product.code}"
          pdf.text "<b>RRP: </b>#{@product.rrp}"
        end
      end
    end

Next, we need to setup nginx to use the X-Accel-Redirect feature. Add the following four lines to your nginx config file, changing the path to point to your own rails app directory. For more information on this step, consult google. There’s plenty of relevant information around.

location /srv/rails/mypp/tmp/pdfs {
  internal;
  root /;
}

To simplify using X-Accel-Redirect, add two helper methods to your ApplicationController:

    def x_accel_pdf(path, filename)
      x_accel_redirect(path, :type => "application/pdf", :filename => filename)
    end

    def x_accel_redirect(path, opts ={})
      if opts[:type]
        response.headers['Content-Type'] = opts[:type]
      else
        response.headers['Content-Type'] = "application/octet-stream"
      end
      response.headers['Content-Disposition'] = "attachment;"
      if opts[:filename]
        response.headers['Content-Disposition'] << " filename= \"#{opts[:filename]}\""
      end
      response.headers["X-Accel-Redirect"] = path

      Rails.logger.info "#{path} sent to client using X-Accel-Redirect"

      render :nothing => true
    end

Finally, add the rendering and X-Accel-Redirect instructions to the relevant controller action. Since my sample report earlier was to display a single product, I’ve added it to the show action on my ProductsController.

    class ProductsController < ApplicationController

      ...

      def show
        @product = Product.find(params[:id])

        respond_to do |format|
          format.html
          format.pdf do
            report = ProductReport.new(@product)
            report.render
            x_accel_pdf(report.path, "product-#{@product.id}.pdf")
          end
        end
      end

      ...

    end

The key lines for generating a report are:

    report = ProductReport.new(@product)
    report.render

I can call these lines any time I need to generate my ProductReport, whether it be in a controller action like here, a ActionMailer email, or a rake task. The output will be rendered to disk and I can retrieve the path by calling report.path.

The key line for using X-Accel-Redirect is:

    x_accel_pdf(report.path, "product-#{@product.id}.pdf")

I can call x_accel_pdf() in any controller action and the filename I give it will be streamed to the client by nginx instead of my rails app.

The setup to all this is a little involved, but once it’s in your app adding new reports is dead simple. A worthwhile tradeoff in my opinion.

The most likely problem you’re likely to run into is misconfiguring nginx. For hints on what might be wrong, make sure you check out the nginx error log.

Santising Activerecord String Attributes

2009-05-22T00:00:00+10:00

For the vast majority of string attributes on my ActiveRecord models, I want a blank string submitted by the user to be saved to the database as NULL. I started writing a small Rails plugin to achieve my goal via a before_validation callback, but then found an existing plugin, Ryan McGeary’s StripAttributes that does the job just fine. As an added bonus it removes leading and trailing whitespace as well.

More recently I was processing a large volume of CSV data and creating new AR records. It turned out the data had some odd ASCII control characters in it (like 0x0B - vertical tab) that were causing grief elsewhere in the app.

In this day and age it’s rare to require the use of ASCII control characters, so I built a new plugin to strip them out before a model is saved. The plugin is essentially a fork of StripAttributes so I won’t claim it’s innovative, but it’s useful.

StripControlChars can be found on github and installed as a gem.

    gem install strip_control_chars

Load the gem from your environment.rb file:

    config.gem 'strip_control_chars'

And add the macro to the models you want to use it with:

    class Product < ActiveRecord::Base
      strip_control_chars!
    end

    p = Product.create!(:name => "Some\x0BWidget")
    puts p.name
    => Some Widget

The following bytes will be replaced with a standard space (0x20): 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0B, 0x0C, 0x0E, 0x0F, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1A, 0x1B, 0x1C, 0x1D, 0x1E, 0x1F

Whilst I was making plugins based on StripAttributes, I made another that replaces various “smart quotes” with their ASCII equivalents. It’s somewhat more controversial than StripControlChars and will probably get me in trouble with typographers, but them’s the breaks. It operates in the same way as StripControlChars, so doesn’t really deserve its own blog post. Check it out on github: DumbQuotes

Eventmachine Based Ftp Server

2009-03-21T00:00:00+11:00

The short version

I’ve released a demo of an FTP server for ruby, built on the EventMachine library.

Check it out here: http://github.com/yob/em-ftpd/tree

The full version

Inside our little web 2.0 bubble we all get very excited about shiny things like REST, web services and open JSON APIs, so it’s easy to forget in the business world with many trading partners and legacy systems, a technology that works tends to hang around for an awfully long time.

In the supply chain world, the most common way I’ve encountered for electronic trading between businesses (submitting orders, receiving invoices, etc) is over good ‘ole FTP.

Yes, it’s old and insecure. However it’s also simple, well understood, has plenty of library support and is now embedded in inventory management software all over the place. You work with what you’ve got.

For a few years I’ve maintained a custom FTP server for a client. It accepts uploaded orders from their customers, translates them to a format the clients system can understand and saves a copy to a outbox folder. Every few hours the clients inventory software connects and downloads the orders directly into their system. It’s hardly a high traffic site, maybe 20 transfers a day, all small text files.

The server was a small ruby daemon, based on the ftpd.rb script published by Chris Wanstrath, and it performed admirably. Recently I was asked to add some new features, and I realised over the years the code had turned into 800 lines of spaghetti. A rewrite was in order.

At the same time a few fellow Australian Rubyist’s had been playing around with EventMachine, a library that claims to help produce highly scalable network code. Scalability wasn’t my issue, but the examples I could see looked alarmingly clean and simple for network socket based applications. Best of all, there was no need to involve threads, the arch nemesis of maintainable code.

Clean and simple was the goal of my rewrite. Bingo.

It took a little while to come to a working solution, mainly because I found few public EventMachine samples that indicated how to open multiple sockets and share data between them. It turned out to be possible and my rewritten server went into production a few days ago.

There was no point asking this client to release their new server as open source, it’s fairly specific to their needs. However I was keen to provide a public demo of an EventMachine based FTP server, so I’ve published a new demo script on github: http://github.com/yob/em-ftpd/tree

It’s not particularly useful as a real FTP server - it hard codes the authentication tokens and provides a simulated directory structure. Hopefully it proves interesting or useful to someone out there though - as a basis for their own custom FTP server or some other multi-socket protocol.

Outdated Gems Under Rails The Second

2009-02-27T00:00:00+11:00

Last year I shared a short initialiser fragment for rails that warns you if you’ve loaded an out of date gem.

It’s a non fatal warning, just a reminder that it might be worthwhile upgrading.

I still use the fragment, but have updated it to support rails 2.2+. To use it, just drop the code into a file in the config/initializers directory of your app.

    # *************************************
    # A handy initiliser that logs when the loaded version of
    # Rails or a gem dependency is out of date. The notice is
    # non-fatal (often we want it to be out of date). I often
    # forget which version of a gem my apps are using, and 
    # don't notice when there is a newer version available.
    #
    # Only really makes sense on Rails >= 2.1, where initialisers
    # and gem dependencies first appeared. Drop this file in
    # config/initializers/
    #
    # James Healy
    # 2nd Feb 2009
    # *************************************

    outdated = []

    # *************************************
    # check the current version of Rails to see if it's the latest
    # *************************************
    max_rails_gem = Gem.cache.find_name('rails').map(&:version).map(&:version).max

    if max_rails_gem && (Rails::VERSION::STRING < max_rails_gem)
      outdated << {:name => "rails", :loaded => Rails::VERSION::STRING, :max => max_rails_gem}
    end

    # *************************************
    # check the current version of all required gems to see if they're the latest
    # *************************************
    Rails.configuration.gems.each do |gem|
      name = gem.name
      if Rails::VERSION::STRING >= "2.2.0"
        loaded_version = gem.specification.version.to_s
      else
        loaded_version = gem.version.to_s
      end
      max_gem_version = Gem.cache.find_name(name).map(&:version).map(&:version).max

      if max_gem_version && (loaded_version < max_gem_version)
        outdated << {:name => name, :loaded => loaded_version, :max => max_gem_version}
      end
    end

    # *************************************
    # print notices 
    # *************************************
    unless outdated.empty?
      logger = RAILS_DEFAULT_LOGGER
      logger.info 
      logger.info "*******************************"
      outdated.each do |w|
        logger.info "NOTICE: #{w[:name]} version #{w[:loaded]} is not the most recent version of #{w[:name]} available on the system (#{w[:max]})"
      end
      logger.info "*******************************"
      logger.info 
    end