I just released Couch Potato 0.2.14 and amongst other things it has a new feature i think is pretty neat: you can unit test your (JavaScript) views using RSpec and Ruby.

You can declare a view in Couch Potato like this:

class Comment
  property :post_id
  view :by_post_id, :key => :post_id
end

This will generate a pair of map/reduce function and push them to Couch Potato. The map function looks something like this:

function(doc) {
  if(doc.ruby_class == 'Comment') {
    emit(doc.post_id, 1);
  }
}

And here’s a unit test for that function:

describe Comment, 'by_post_id' do
  it "should map to the post_id" do
    Comment.by_post_id.should map({:ruby_class => 'Comment', :post_id => 3}).to([3, null])
  end
end

As you can see all you have to do is pass a Ruby Hash that looks like your CouchDB document and the expected results. You can also pass in multiple results if you expect your map function to emit multiple key/value pairs.

Testing a reduce function works the same way:

describe Comment, 'by_post_id' do
  it "should reduce to the number of comments" do
    Comment.by_post_id.should reduce([], [1, 1, 1]).to(3)
  end
end

For testing re-reducing you simply call .should rereduce(...).to(...).

How it works

So how come you can test JavaScript functions in pure Ruby? Well, by stealing other people’s tricks. I recently contributed a few patches to mustache.js which is a new templating library ported to JavaScript by @janl. It has a test runner currently implemented in Ruby which generates JavaScript code on the fly, runs it using Spidermonkey and reads back the results. I have added a few steps to this process:

1. A custom RSpec matcher collects the map function, a Ruby Hash representing the input and the expected output
2. The input is converted to JavaScript using the JSON gem
3. JavaScript code is generated that runs the document through the map function and prints the resulting JSON.
4. The Ruby code runs spidermonkey, collects the output and parses it back to Ruby using again the JSON gem
5. The results are compared to the expected values.

You can see how it works by looking at the code for the RSpec matchers.

I think this addition lowers the barrier to test your views quite a bit. Although I’m usually not a big fan of “one language to rule them all” and love writing JavaScript, being able to write all the necessary tests in Ruby when working on a Ruby project makes things way easier.

Yesterday I added a new feature to Couch Potato, my persistence layer for CouchDB that I call Dynamically Inherited Methods™ and found the implementation I came up with interesting enough to share. Here’s the problem:

class User

  def self.string_attr_reader(name)
    define_method name do
      instance_variable_get("@#{name}").to_s
    end
  end

  string_attr_reader :nickname
  attr_writer :nickname
end

user = User.new
user.nickname = 123
user.nickname # => '123'

This code adds a macro to the class user called string_attr_reader. When you call this macro on the class and pass it a name it generates a method that returns the instance of the same name converted to a string. This all works perfectly until you want to override the generated method to add some behavior:

class User
  def nickname
    super.gsub('6', '7')
  end
end

The enhanced method now replaces all occurrences of 6 with a 7 – except it doesn’t. When you run the above code you will get an exception where Ruby complains there is no super to call. That’s because we defined the nickname method in the User class so we didn’t inherit it, hence no super.

The one way out of this is to use alias_method, or alias_method_chain when you are using Rails:

class User
  def nickname_with_six_replaced
    nickname_without_six_replaced.gsub('6', '7')
  end
  alias_method_chain :nickname, :six_replaced
end

First of all this isn’t very beautiful anymore, and secondly using alias_method_chain when you could use standard object oriented metaphors (like inheritance) doesn’t make a lot of sense – except that you can call yourself a metaprogramming programmer, yay.

Anyway, there is another way – actually involving much funkier meta programming – to solve the problem:

class User
  def self.string_attr_reader(name)
    accessor_module.module_eval do
      define_method name do
        instance_variable_get("@#{name}").to_s
      end
    end
  end

  private

  def self.accessor_module
    unless const_defined?('AccessorMethods')
      accessor_module = const_set('AccessorMethods', Module.new)
      include accessor_module
    end
    const_get('AccessorMethods')
  end
end

The updated code now dynamically creates a Module called User::AccessorMethods and includes it into the User class. All methods generated by string_attr_reader are now added to that new module instead of the class. The result is that when overriding those methods you can now call super because they’re are inherited from the new module.

class User
  attr_writer :nickname
  string_attr_reader :nickname

  def nickname
    super.gsub('6', '7')
  end
end

user = User.new
user.nickname = 678
user.nickname # => '778'

While this code involves some fairly crazy metaprogramming which would be too much for a simple example like the above, I think libraries like CouchPotato can still benefit, as the application code can become cleaner by not having to do any metaprogramming but resort to standard object oriented ways of programming.

After my Ruby sittin’ on the Couch talk about Ruby Frameworks and CouchDB at Scotland on Rails last week a bit of a debate started. In the talk I did compare the available Ruby Frameworks and discussed how well they fit the CouchDB way of doing things. In my conclusion I recommended using either CouchRest or RelaxDB for development and at the same time urged people not to use one of the ActiveRecord like libraries like CouchFoo.

As it seems to me the intention of my talk hasn’t reached all of its audience yet I’ll try to make my point again using this blog post.

The frameworks I looked at could be divided into two classes: the ones using CouchDB semantics (e.g. CouchRest/RelaxDB) and the ones trying to provide an ActiveRecord like interface for the applications (e.g. CouchFoo). The reasons I recommended not to use ActiveRecord semantics are:

CouchDB is not about relations

In ActiveRecord we have to model our domain models so they fit into a relational schema. That means flat tables and relationships between two tables, through a third and fourth table etc. To get results from the database we usually join a few tables and get back the resulting rows, nicely converted into Ruby objects for us. That is (sort of) fine for a relational database because all it has are those tables but it doesn’t work at all with CouchDB.

To use CouchDB to its full potential you need to understand and use its views

CouchDB doesn’t have a concept of tables at all. And the way you pull your data from CouchDB is fundamentally different. Instead of joining data from different tables via an SQL query you procedurally build up an index of data by providing the map and reduce functions which you then query.

In order to fully use CouchDB you have to write custom map/reduce functions and abstracting that into an API that was designed for generating SQL queries doesn’t allow you that.

I could go into more details but that really is my main point. I know from my experience that changing the wiring in your head after too many years of ActiveRecord is hard so I don’t expect anyone to immediately agree with me but I do believe that the only viable way of using CouchDB is through the interface that was designed for it and not by an abstractions that just happened to be there first.

I’m looking forward to a lively discussion, either in the comments or on other blogs.

Ruby on Rails 2.3 is almost there and brings a bunch of pretty cool new features. One of them is called templates and it allows you to customize the bootstrapping of new applications in order to automate the initial setup of your new app and hence get up to speed faster.

I find the naming a bit confusing: since we already have view templates in Rails I will refer to this new feature as application templates from now on.

So what does an application template do? Whenever you start a new rails application you are already using the Rails default template by issuing the rails command. It creates the usual app, config etc. folders and generates a bunch of scripts and configuration files for you. Application templates simply extend this process by allowing you to add your own setup steps. This is done by writing your own template file (or grabbing someone else’s).

Pratik Naik has already written an excellent tutorial on this so I’m not going to repeat what’s already written. Basically Rails offers you a few convenience methods for adding for example routes, plugins, gems or files to your new application.

So this blog post is more about showing off my very own template that I wrote today. (For the impatient: here it is).

Of course I’m not the first and only one who has written such a template so before I started I found this collection of templates which I immediately started to steal borrow from. As you probably know every programmer has its own style, uses his own unique set of tools in his (or her) own way. Which is why I found all of the templates I saw didn’t really fit my very own needs. The ones I found were pretty basic (to me) so when I rolled my own I took what I needed from what was there and added a whole bunch of my own stuff:

• setup of configuration files for Rspec, Cucumber and Culerity (Culerity is my plugin for driving Celerity – think Selenium but withour having to use a real browser – with Cucumber)
• a working user registration/login process using authlogic, including all controllers and views
• an XHTML application layout with jQuery (+ a few plugins) and blueprint CSS set up
• a simple capistrano deployment script
• German localization for all the built-in helpers
• a Thinking Sphinx configuration file

I pretty much need all of the above for 99% of my Rails projects and it costs me at least a day every time to set this all up. Well, not anymore Now I can run rails my_new_app -m http://gist.github.com/75038 and immediately after that start working on the distinct features.

I’m already planning to add a few extra. First of all the user authentication needs cucumber features so I don’t break anything when extending that. I also want to add configuration and deployment for a staging server. And maybe make the whole script a bit more configurable, i.e. install thinking sphinx or not etc.

The script is already pretty large so I’m not sure how much more I would want to put into it. Since you can apply templates to exisitng apps it will probably make sense to split the whole thing up at some point, so I could put the authentication part into its own template.

If you want to get started with application templates now I suggest you simply start with mine and/or the others and grab whatever you need. Just make sure you re-publish what you added so we can all build upon everyone’s work. Thanks.

On a recent Rails project we have been working a lot with PDF documents. The application generates PDF invoices and account statements, places markers and comments into existing documents and also merges multiple single page PDFs into one. We do all this with a few PDF tools:

• the PDF::Writer library for Ruby to generate PDFs
• pdftk – to merge multiple PDFs into one or overlay them
• pdftotext part of the xpdf library, to extract texts from PDFs

On OSX all of these can be installed via MacPorts. Debian has packages as well.

While PDF::Writer is a Ruby library pdftk is a command line tool. We simple call it using Kernel.system and check the return code via the $? variable.

unless system (“pdftk #{source_pdf} … output #{target_pdf}”)
raise PdfError(“pdftk returned error code #{$?}”)
end

With all this PDF processing the need for testing the contents of the generated documents arose. We have factored out all the PDF processing into a bunch of extra classes which we simply unit test with RSpec: make sure the parameters are passed to the command line correctly, that the right exception is thrown for each return code etc.

In addition to unit testing we also write customer driven acceptance tests with Cucumber, where we assert on a high level the outcome of certain actions. With HTML pages we can simply use the built-in steps that in turn use Webrat to parse the HTML like this:

Given a purchase over 200 EUR
And an invoice
When I go to the start page
And I follow “Invoice”
Then I should see “200 EUR”

Now in our case the invoice link links to a PDF but we still want to know what’s inside the document. The solution we came up with looks like this:

Given a purchase over 200 EUR
And an invoice
When I go to the start page
And I follow the PDF link “Invoice”
Then I should see “200 EUR”

What this does in the background is follow the link as usual, write the response into a temporary file, convert that to text using pdftotext and write the result back into the response. This way we can make assertions about the contents of the PDF almost as if it were an HTML page (except for tags of course). Here is the implementation:

When ‘I follow the PDF link “$label”‘ do |label|
click_link(label)
temp_pdf = Tempfile.new(‘pdf’)
temp_pdf << response.body
temp_pdf.close
temp_txt = Tempfile.new(‘txt’)
temp_txt.close
`pdftotext -q #{temp_pdf.path} #{temp_txt.path}`
response.body = File.read temp_txt.path
end

Since the day we started upstream testing was our obsession. When we began all we had was the Rails built-in unit, functional and integration tests. When we learned about RSpec the world became a brighter place. When the RSpec team released their StoryRunner we were thrilled. We started writing stories in English and were able to run them. Whoa. It soon turned out the first version had some serious issues with staying DRY. More versions were released, we rewrote our stories, step by step StoryRunner got better and we got better, too.

Then Cucumber came out and it was a huge leap ahead again. With the integration of Webrat we could write much shorter stories, but more importantly we could test that our links and forms were working as well. Everything was working perfectly and the world was a happy place – as long as we didn’t use any JavaScript/AJAX…

Welcome to Culerity. Culerity is a small gem that integrates Cucumber with Celerity, a wrapper around HtmlUnit which in turn is a Java library that parses HTML and runs the embedded JavaScript code. And unlike Watir or Selenium this all happens in headless mode: without hijacking your browser.

The problem with Celerity has been that it only works in a JRuby environment, which means you either had to run your application in JRuby as well (which might not even work with certain libraries and plugins) or somehow work your way around it by running your tests in JRuby and your application in whatever Ruby you wanted to use and somehow glue it together. Culerity now fills this gap. After installing it you run your Cucumber features as usual. Celerity now spawns a Java process in the background, sends all the Celerity calls to this process and evaluates the results back in the original Ruby environment — everything works (almost) as if you were running just in your single Ruby process. For an easier start Culerity comes with the same set of Cucumber step definitions that are provided by a default Cucumber/Webrat installation. This means you can reuse your step definitions written for Webrat.

To get started you can either clone the GitHub repository or simply install the gem (langalex-culerity). Then follow the instructions in the README. If you find something is not working for you or could be improved please feel free to fork the source code and show it some love. The codebase is really small right now and fully spec’d.

—

Does your company need help with software testing? We can help: check out our new product Scene Investigation.

When Hoptoad came out we were really excited. No more hundreds or even thousands of emails in case of some server failure or *gasp* even a bug deployed to the production server. Instead a nice looking website with a clean list. Nice.

Within days we had switched all projects from exception_notifier to Hoptoad – and then that was it. Occasionally Hoptoad would send an email with an exception but most of the time it would be some crawler requesting an invalid URL so we read and forgot about it. The problem was that with Hoptoad to forget about those errors had become be really easy – too easy. So last week a client told me the signup form on our project was showing an error page and nobody had been able to sign up for days.

Wait. We are monitoring the app and get notified about every exception. Of course we did but over the months we had accumulated more than 50 exceptions in Hoptoad and nobody was really paying attention to that big pile of false alarms caused by some crawler anymore. So yes, we did miss that positive alarm.

As a consequence we spent the same day going through all our exceptions in all projects and killed them all.

A big part of the exception came from crawlers requesting non-existent actions and stuff like that. With Rails 2.2 this is relatively easy to fix. In our routes.rb we have declared most of our routes as restful resources:

map.resources :posts

Now the PostsController does not implement the index action but we do get requests on /posts from time to time which result in an Unknown Action exception. The solution is to remove the index route:

map.resources :posts, :except => :index

In other cases we had to improve our guards on parameters a little. Passing an empty string for the page parameter to a paginate call seems to be a favorite pastime for the google crawler.

So, lessons learned: keep your exception inbox clean. Even if it’s just a bunch of bots.

Apple’s Application Framework Cocoa gained popularity recently, also due to the iPhone. Like many others we started to learn Objective-C to play around with the iPhone a couple of months ago. The basic syntax is fairly easy to learn, also because lots of sources exist to get you up and running. But in my opinion the real challenge is to understand how the interface builder works

The past

But as a Rubyist the C like syntax of Objective-C was quite a pain. So I didn’t enjoy working with Objective-C. The existing Ruby bridge RubyCocoa has its own drawbacks. Especially moving parameter names into the method name to mimic Objective-C’s keyed arguments doesn’t appear future proof to me. This all kept me from getting deeper into the subject.

The future

In March this year MacRuby made its appearance as a successor for RubyCocoa. MacRuby is a port of Ruby 1.9 on top of the Objective-C runtime and garbage collector, which aims to fix all of the RubyCocoa drawbacks and bring the full power of Ruby to the Cocoa framework. With the 0.3 release a couple of weeks ago, MacRuby now supports the Interface Builder, thus making it a full fledged member of the OS X Development tool chain. Since then MacRuby gained serious popularity. An official Apple Developer Connection Tutorial for “Developing Cocoa Applications Using MacRuby” was published recently. More good tutorials can be found on the web, so I just link to them instead of writing another.

During the weekend I went through the first chapters of “Cocoa Programming for Mac OSX” with MacRuby instead of Objective-C to get familiar with the Cocoa library and Interface Builder. It went super smooth so far, I really had fun. Here is just a short example of the very first app out of the book to see how the code compares to objective-c.

class Foo
attr_writer :text_field

def seed(sender)
@text_field.StringValue = “Generator doesn’t need to be seeded ”
end

def generate(sender)
@text_field.StringValue = (rand(100) + 1)
end
end

^ Ruby vs. Objective-C v

@interface Foo : NSObject
{
IBOutlet NSTextField *textField;
}

- (IBAction)gerneate:(id)sender;
- (IBAction)seed:(id)sender;
@end

@impelemtation Foo

- (IBAction)generate:(id)sender
{
int generated = (random() % 100) +1;
[textField setIntValue:generated];
}

- (IBAction)seed:(id)sender
{
srandom(time(NULL));
[textField setStringValue:@"Generator seeded"];
}
@end

As you can see you don’t need a header file and much less special characters. It even gets better, because you don’t need to call [[NSMutableArray alloc] init] but Array.new or just [] to initialize an empty Array. Isn’t Ruby beautiful? And with MacRuby you can have it for your Cocoa Apps too. So jump on board.

Update: the gem is now available, see the installation instructions below.

After several weeks of incubating on my computer it’s finally time to get real: I have just open sourced Couch Potato under the MIT license. You can get Couch Potato on github now. For an introduction to CouchDB and ruby please read my previous blog post A CouchDB primer for an ActiveRecord mindset. The following is a very short introduction into using Couch Potato. If you want to know more you can start with the README.

The goal of Couch Potato is to create a migration path for users of ActiveRecord and other object relational mappers to port their applications to CouchDB. It therefore offers a basic set of the functionality provided by most ORMs and adds functionality unique to CouchDB on top.

Installation

Couch Potato is available as a gem from http://gems.github.com, so you can just do

$ sudo gem source –add http://gems.github.com # if you haven’t alread
$ sudo gem install langalex-couch_potato

$ irb
>> require ‘rubygems’
>> gem ‘couch_potato’
>> require ‘couch_potato’
>> CouchPotato::Config.database_name = ‘name of the db’

Alternatively you can download the sources from github. If you are using rails just copy the files into vendor/plugins, create a RAILS_ROOT/config/couchdb.yml file (see the README for the format) and you are ready to go. For other applications you will have to require the lib/couch_potato.rb file and then set the database name by calling CouchPotato::Config.database_name = 'name of the db'.

As Couch Potato is still very young you can expect its feature set to grow quite a bit in the near future. What you can download now is the very core together with a few features giving you a glimpse of what is about to come:

Persistence

Create a new class and make its instances persistable by including the Persistence module. As there is no schema in a CouchDB you have to declare the properties you want to persist:

class User
include CouchPotato::Persistence

property :name
end

Now you can save your objects:

user = User.new :name => ‘joe’
user.save # or save!

Properties:

user.name # => ‘joe’
user.name = {:first => ['joe', 'joey'], :last => ‘doe’, :middle => ‘J’} # you can set any ruby object that responds_to :to_json
user._id # => “02097f33a0046123f1ebc0ebb6937269″
user.created_at # => Fri Oct 24 19:05:54 +0200 2008

You can of course also retrieve your instance:

User.get “02097f33a0046123f1ebc0ebb6937269″ # => < #User 0x3075>

Associations

As of now has_many and belongs_to are supported. By default the associated objects are stored in separate documents linked via foreign keys just like in relational databases.

class User
has_many :addresses, :dependent => :destroy
end

class Address
belongs_to :user
property :street
end

user = User.new
user.addresses.build :street => ‘potato way’
user.addresses.first # => < #Address 0x987>
user.addresses.create! # raises an exception as street is blank
user.addresses.first.user == user # => true

When saving an object all associated objects are automatically saved as well. All these save operations are sent to CouchDB in one operation which means the whole process is atomic across all objects saved, plus only one database roundtrip is required making it much faster.

As CouchDB can not only store flat structures you also store associations inline:

class User
has_many :addresses, :stored => :inline
end

This will store the addresses of the user as an array within your CouchDB document.

(more…)

That couch was our intro picture at the workshop.
(picture from gizmodo.com)

Monday was the first event at our new upstream office, starring @janl and me presenting an introduction to CouchDB – including a new hands on examples part – and afterwards an overview of what can be done with CouchDB in Ruby so far. The talks were followed by a discussion that gave (hopefully not only) me a couple of new insights I want to share here – after summarizing the evening for the people who couldn’t make it. There will also be a video recording with synchronized slides of both talks be available in the next days (thanks @klimpong).

What is CouchDB

CouchDB ist the new cool kid on the block. It’s a document oriented database that has replication built in, can scale massively and uses an HTTP REST interface to query it. Documents are stored as JSON constructs and can be queried with views that are built using Map-Reduce (a smaller company called google has had a bit of success with that recently). Oh and it’s written in Erlang. Jan has given a number of talks on numerous events already, so there are already a couple of videos and slides available – not from the hands on part though For that you should watch his blog I guess.
(more…)