I just released Couch Potato 0.2.14 and amongst other things it has a new feature i think is pretty neat: you can unit test your (JavaScript) views using RSpec and Ruby.

You can declare a view in Couch Potato like this:

class Comment
  property :post_id
  view :by_post_id, :key => :post_id
end

This will generate a pair of map/reduce function and push them to Couch Potato. The map function looks something like this:

function(doc) {
  if(doc.ruby_class == 'Comment') {
    emit(doc.post_id, 1);
  }
}

And here’s a unit test for that function:

describe Comment, 'by_post_id' do
  it "should map to the post_id" do
    Comment.by_post_id.should map({:ruby_class => 'Comment', :post_id => 3}).to([3, null])
  end
end

As you can see all you have to do is pass a Ruby Hash that looks like your CouchDB document and the expected results. You can also pass in multiple results if you expect your map function to emit multiple key/value pairs.

Testing a reduce function works the same way:

describe Comment, 'by_post_id' do
  it "should reduce to the number of comments" do
    Comment.by_post_id.should reduce([], [1, 1, 1]).to(3)
  end
end

For testing re-reducing you simply call .should rereduce(...).to(...).

How it works

So how come you can test JavaScript functions in pure Ruby? Well, by stealing other people’s tricks. I recently contributed a few patches to mustache.js which is a new templating library ported to JavaScript by @janl. It has a test runner currently implemented in Ruby which generates JavaScript code on the fly, runs it using Spidermonkey and reads back the results. I have added a few steps to this process:

1. A custom RSpec matcher collects the map function, a Ruby Hash representing the input and the expected output
2. The input is converted to JavaScript using the JSON gem
3. JavaScript code is generated that runs the document through the map function and prints the resulting JSON.
4. The Ruby code runs spidermonkey, collects the output and parses it back to Ruby using again the JSON gem
5. The results are compared to the expected values.

You can see how it works by looking at the code for the RSpec matchers.

I think this addition lowers the barrier to test your views quite a bit. Although I’m usually not a big fan of “one language to rule them all” and love writing JavaScript, being able to write all the necessary tests in Ruby when working on a Ruby project makes things way easier.

On last week’s RailsCamp UK I started hacking on a new CouchApp called HejHej. Its purpose is to help me learn Swedish, but what’s more important here: I wrote this app BDD style using Cucumber, the famous BDD tool and Culerity, my humble addition that allows me to test any webapp (including client side JavaScript) with it.

The whole thing is available on Github so you can check out the features and steps there. In the following post I will show the necessary steps to test your own CouchApps with Cucumber.
(more…)

For doing Red/Green TDD it is essential to have fast running tests. But no matter how fast your tests are, there are always these seconds when you and your pair are sitting in front of the screen waiting that the test run actually starts. That is the time when Rails, your code, gems and plugins are loaded. In a rather small project, which I use as an example here, this cumulates into a start up time of around 5s before EACH test run. This becomes even longer when your project grows.

…..

Finished in 0.127571 seconds

5 examples, 0 failures

real 0m6.456s
user 0m4.682s
sys 0m1.681s

Although 5s, as in my example, aren’t such a long time, it sums up to a significant amount when doing Red/Green TDD. So everything that helps you to cut down test startup times, increases your productivity. AutoSpec or Rspactor approach this on the user site by running your test as soon as you change something. But they won’t cut down the technical implied start up time. Spec-server tried to approach the load time issue first by loading the Rails stack including the libraries of your environment upfront in an extra process. It uses Drb to get your test code to the pre-loaded testing process. Although the idea behind spec-server is the right one, it never worked for me in day to day usage. I came in situations in which tests failed or code wasn’t reloaded properly at all. But recently I discovered spork thanx to Rany (@purzelrakete). Spork takes the idea from spec-server and learned from its mistakes. E.g. it has its own class loading mechanism that takes care of not preloading your models at startup, instead of relying on rails auto loading. Beside working more reliable, this is also an option for other frameworks but Rails. It just works! Running the same test code than before with spork (spec paht_to_spec --drb), cuts the startup time into half.

…..

Finished in 0.066628 seconds

5 examples, 0 failures

real 0m3.588s
user 0m0.469s
sys 0m0.108s

When doing 200 test startups a day, a number which is not unrealistic, consider doing it Red/Green, this a 10 min time win You probably won’t need more than 10 min to set up spork.

1. Install spork

gem install spork

2. Bootstrap your Rails app


cd /path/to/project/root
spork --bootstrap


This command adds some instructions and code to spec/spec_helper.rb. Basically two blocks, one to tell which code to preload and one to tell which to reload on every test run. Code outside these blocks is run at both times.

3. Modify spec/spec_helper.rb

4. Modify spec/spec.opts

Add --drb to spec/spec.opts so that rake spec uses drb to communicate with spork. Set TM_RSPEC_OPTS --drb in TextMate so that the rspec TextMate bundle uses drb as well.

5. Fire up spork for rspec

spork rspec

6. Kickstart your RSpec Tests

If you find out that some of your code wasn’t reloaded properly, spork -d shows you which classes are loaded and where from. There shouldn’t be any of your app/classes in it. If that’s the case, the reason for this can be either a eager plugin (e.g. older versions of thinking_sphinx) or you used your code in initializers or the environment. Put such code in spork blocks like these in your spec_helper for the testing environment, this will prevent wrong preloading.

Here is an example how the spec_helper could look like:

require 'rubygems'
require 'spork'

ENV["RAILS_ENV"] = "test"

Spork.prefork do
  require File.expand_path(File.dirname(__FILE__) + "/../config/environment")
  require 'spec'
  require 'spec/rails'
  require 'machinist/active_record'
end

Spork.each_run do
  require 'spec/blueprints'
end

Oh and btw, you can use spork for Cucumber as well, just modify cucumbers env.rb (same as you did with your spec_helper) and then launch spork with spork cucumber

And one last thing, as we are talking about faster test cycles, take a look at parallel _specs, to run your specs on multiple cores.

Happy TDDing!

On a recent Rails project we have been working a lot with PDF documents. The application generates PDF invoices and account statements, places markers and comments into existing documents and also merges multiple single page PDFs into one. We do all this with a few PDF tools:

• the PDF::Writer library for Ruby to generate PDFs
• pdftk – to merge multiple PDFs into one or overlay them
• pdftotext part of the xpdf library, to extract texts from PDFs

On OSX all of these can be installed via MacPorts. Debian has packages as well.

While PDF::Writer is a Ruby library pdftk is a command line tool. We simple call it using Kernel.system and check the return code via the $? variable.

unless system (“pdftk #{source_pdf} … output #{target_pdf}”)
raise PdfError(“pdftk returned error code #{$?}”)
end

With all this PDF processing the need for testing the contents of the generated documents arose. We have factored out all the PDF processing into a bunch of extra classes which we simply unit test with RSpec: make sure the parameters are passed to the command line correctly, that the right exception is thrown for each return code etc.

In addition to unit testing we also write customer driven acceptance tests with Cucumber, where we assert on a high level the outcome of certain actions. With HTML pages we can simply use the built-in steps that in turn use Webrat to parse the HTML like this:

Given a purchase over 200 EUR
And an invoice
When I go to the start page
And I follow “Invoice”
Then I should see “200 EUR”

Now in our case the invoice link links to a PDF but we still want to know what’s inside the document. The solution we came up with looks like this:

Given a purchase over 200 EUR
And an invoice
When I go to the start page
And I follow the PDF link “Invoice”
Then I should see “200 EUR”

What this does in the background is follow the link as usual, write the response into a temporary file, convert that to text using pdftotext and write the result back into the response. This way we can make assertions about the contents of the PDF almost as if it were an HTML page (except for tags of course). Here is the implementation:

When ‘I follow the PDF link “$label”‘ do |label|
click_link(label)
temp_pdf = Tempfile.new(‘pdf’)
temp_pdf << response.body
temp_pdf.close
temp_txt = Tempfile.new(‘txt’)
temp_txt.close
`pdftotext -q #{temp_pdf.path} #{temp_txt.path}`
response.body = File.read temp_txt.path
end

On Thursday I gave a talk about Bug fighting with TDD and how it will help you to write better software at the Ruby User Group in Berlin. Sorry, I won’t put the slides online, although people asked about it. I think they are a pretty useless collection of keywords and pictures without the talking. Instead I like to provide a collection of links to dive into the whole testing matters.

First a collection of tools you can choose from for various testing scenarios. They all have their pros and cons, it just comes down to which will do the job best for you.

Tools for Unit Testing

• Rspec
• Test/Unit (part of Rails)
• shoulda

Tools for Mocking

• Flexmock
• Mocha
• RR

Tools for Acceptance Testing

• Webrat
• Cucumber
• HtmlUnit (with JRuby)
• Selenium

Tools to generate test objects

• Machinist
• FactoryGirl
• ObjectDaddy

Misc tools

• Autotest
• Tarantula
• Culerity

To save you the initial searching where to start, I collected some links where you find basic information about testing in general or with certain tools.

Getting started

• A Guide to Testing Rails Applications
• An introduction to RSpec
• Functional Testing awesomeness with Webrat
• How I Test Controllers, 2009 Remix
• Shoulda tutorial
• Mock objects explained and intro to Mocha (de)

For everyone who wants to get more into the subject, I drilled down my feed reader to collect some interesting blog posts on testing matters.

Opinions and In-sign

• The Great Test Framework Dance-off
• Thoughts on Developer Testing
• Spikes, test-after, and learning how to test
• A case against a case against mocking and stubbing
• A critical look at the current state of Ruby testing
• One Line Tests Without the Smells
• Mocks Aren’t Stubs
• Testing as Communication: Real-World Techniques Notes
• When duplication in tests informs design
• Writing Code That Doesn’t Suck (vid)

So much for my two cents to make the software world better.

—

Does your company need help with software testing? We can help: check out our new product Scene Investigation.

Last week was my first Erlang gig. I paired up with Jan Lehnardt (core committer of CouchDB). Our goal was to implement a new statistics module for CouchDB within one week. Jan knew more about the CouchDB code and Erlang, I contributed my knowledge about testing and pair programming.

Integration Testing in JavaScript

Since the testing infrastructure in CouchDB left some room for improvement we decided to introduce some new concepts. The existing tests were written in JavaScript penetrating the database via its HTTP/JSON interface. The tests were really long and coarse grained which lead to a problem: When one of the tests broke it wasn’t very easy to figure out which one. We didn’t want to replace or add too much to the existing, self-made test framework so we just added a bit of sugar: BDD style specs with meaningful names:

var response_count_tests = {
’should show the number of 404 responses’: function(name) {
// Given
restartServer();

// When
CouchDB.request(‘GET’, ‘/some_nonexistant_url’);

// Then
TEquals(1, CouchDB.stats_request(‘httpd’, ‘not_found’), name);
}
};

var all_tests = [response_count_tests, ...];

for(var i in all_tests) {
test_group = all_tests[i];
for(var j in test_group {
test_group[j](j);
}
}

The TEquals function is just our extension of an existing function T which is responsible for reporting errors when the assertion doesn’t match. By passing the name of the test spec we can now display that name along with the error message:

Error in ’should show the number of 404 responses’: ‘CouchDB.stats_request(‘httpd’, ‘not_found’)’ should be ‘1′ but was ‘0′.

Unit Testing in Erlang

In addition to those integration tests we wanted to add some unit testing on a lower level. Turns out the only unit testing framework for Erlang seems to be EUnit, which is a very barebones (at least when you come from something like RSpec) xUnit type of tool. You define a test by appending “_test” to your function name and then you have a bunch of assert statements at your disposal. Well that’s at least something and it worked. Here is an example:

should_return_zero_if_nothing_has_been_counted_yet_test() ->
test_setup(fun() ->
Result = ?MODULE:get({httpd, average_request_count}),
?assertEqual(< <"0.00">>, Result)
end).

We added the test_setup for starting and tearing down processes needed for our tests. Then we pass a fun with the actual test code. We got through the week pretty ok with this but I really missed a few things:

Erlang

I’ve read the book, I’ve watched numerous screencasts but this was my first real life project in Erlang. Many people complain about its syntax and while this isn’t the most beautiful language I’ve ever seen I have to say it’s not really a big problem. I got used to it on the first day.

What is really powerful in Erlang is pattern matching. Instead of simply assigning a value to a variable you can always match against the structure of that data to extract parts of it:

Data = {“Hans”, “Wurst”, {born, 1980, 10, 23}}.
{FirstName, _, {born, Year, _, Day} = Data.

This example lets you pull out numerous details from the tuple in a single statement. You can do the same for case statements or when overloading functions. So this pattern matching is a powerful concept seen everywhere in Erlang code. It makes the code you write pretty dense which is cool on the one hand but sometimes also makes it hard to read so you have to be careful how much you want to cram into a single line of code.

CouchDB is built using make – which felt like 1990. I’ve never used make much myself so I’m not really qualified to talk about it but we did spend a good amount of time in our MakeFile because of some whitespace problem, and we still haven’t managed to integrate our new tests into the build process. I’d love to see something like Rake being used instead.

All in all this have been very pleasant first steps though. Pairing up for this turned out to be a very good idea, I have learned a lot about Erlang and am eager for more now.

Since the day we started upstream testing was our obsession. When we began all we had was the Rails built-in unit, functional and integration tests. When we learned about RSpec the world became a brighter place. When the RSpec team released their StoryRunner we were thrilled. We started writing stories in English and were able to run them. Whoa. It soon turned out the first version had some serious issues with staying DRY. More versions were released, we rewrote our stories, step by step StoryRunner got better and we got better, too.

Then Cucumber came out and it was a huge leap ahead again. With the integration of Webrat we could write much shorter stories, but more importantly we could test that our links and forms were working as well. Everything was working perfectly and the world was a happy place – as long as we didn’t use any JavaScript/AJAX…

Welcome to Culerity. Culerity is a small gem that integrates Cucumber with Celerity, a wrapper around HtmlUnit which in turn is a Java library that parses HTML and runs the embedded JavaScript code. And unlike Watir or Selenium this all happens in headless mode: without hijacking your browser.

The problem with Celerity has been that it only works in a JRuby environment, which means you either had to run your application in JRuby as well (which might not even work with certain libraries and plugins) or somehow work your way around it by running your tests in JRuby and your application in whatever Ruby you wanted to use and somehow glue it together. Culerity now fills this gap. After installing it you run your Cucumber features as usual. Celerity now spawns a Java process in the background, sends all the Celerity calls to this process and evaluates the results back in the original Ruby environment — everything works (almost) as if you were running just in your single Ruby process. For an easier start Culerity comes with the same set of Cucumber step definitions that are provided by a default Cucumber/Webrat installation. This means you can reuse your step definitions written for Webrat.

To get started you can either clone the GitHub repository or simply install the gem (langalex-culerity). Then follow the instructions in the README. If you find something is not working for you or could be improved please feel free to fork the source code and show it some love. The codebase is really small right now and fully spec’d.

—

Does your company need help with software testing? We can help: check out our new product Scene Investigation.

I recently started using Webrat in one of our project’s test suites and fell in love with it immediately. Webrat is a functional testing library for Ruby on Rails which you can use in your rails integration tests (or better: the RSpec Story Runner) to let your tests walk through your application. The big difference between Webrat and a “standard” Rails Integration Test is that with Webrat you click on actual links and submit actual forms, instead of just sending requests to you application.

This closes an important gap to testing in a real browser (like with Selenium) and now gives us much more confidence in our tests – we now not only know that our models and controllers integrate, but we also know that they work with our views. That had been an unpleasant problem for a while. All our tests were passing but we still had way too many problems with invalid links and forms submiting to the wrong action. No more.

Webrat provides a sort of DSL (everything is a DSL nowaday isn’t it? ) for interacting with all the HTML elements you would find in a HTML document like links, checkboxes, radio buttons, text fields, selects and submit buttons. A test case might look like this:

visits ‘/’
clicks_link ‘Sign up’
fills_in ‘username’, :with => ‘joedoe’
selects ‘Male’
checks ‘Terms of Service’
clicks_button

Webrat now fetches the start page of the application, parses it and follows the first link labelled with “Sign Up”, gets the corresponding page, parses the form, fills in the values and submits it. Whenever a response comes back with anything else than a 200 (or 302, in which case that redirection is followed automatically) or an element on the page can’t be found, Webrat complains with an error and you know there’s probably something wrong in your view.

In Order to get this working with Story Runner I have wrapped most calls into When calls, for example:

When ‘I click on $link’ do |link|
clicks_link link
end
When ‘I submit the form’ do
clicks_button
end
When ‘I select $option’ do |option|
selects option
end

And now my Story looks like this:

Scenario: successful signup
When I go to the start page
And I click on Sign up
And I select Male
And I check the Terms of Service
And I submit the form
Then the page should show Thank you for signing up

I’m not sure yet if it makes sense to release those wrappers as a plugin, we’ll see. After all it’s just a few lines of trivial code.

By the way, Webrat is hosted on awesome github, so after using it for 30 minutes (seriously) I forked my own version and added support for the Rails date helpers (selects_date Date.today, :from => 'signup_date' instead of selects 'December', :from => 'signup_date_2i'; selects '2008', :from => 'signup_date_1i'; selects '01', :from => 'signup_date_3i') and Emails (e.g. clicking the activation link in a signup email). (Bryan Helmkamp, if you read this: Why haven’t you pulled this (and in fact any other fork) into the main line yet?)

All in all the experience has been fantastic. Webrat is on all my Rails projects now. The only – conceptual – problem so far is that I can’t use it to test AJAX. But that’s the topic of another post. And there’s a fork on github that experiments with celerity, which can do AJAX and everything.

We recently introduced a new feature in autoki called the social feed. It’s basically a yellow box displaying any events on the platform relevant to the current user, like a friend has posted a new photo, or a new interesting car was uploaded. The data model behind this is pretty straightforward, we have a FeedEvent class and all kinds of subclasses, e.g. a MessageReceivedEvent. Each event belongs to a user and an event source, in this example the user would be the user who received the message and the event source would be the message itself. For each user, we simply display all the events that belong to him or her.

Now the question was this: How do we create these events? The most straightforward way would probably have been to create them in the models, so the Message model would have an after_create callback that created the event. What we didn’t like about this solution was that we would put a whole bunch of logic into the models that didn’t really belong there. Why would a Message care if there was some kind of event feed? Plus these events would be all around in our unit tests and make the bloated and probably sloooow (again). So we wanted to use the observer pattern to remove the creation of the event from the models.

(more…)

Last week was RailsConf 2007 in Berlin. Only a couple of days have passed and it already seems so far away. Time for some blogging before everybody forgets that it even took place and nobody’s going to read this So here’s the interesting part of the sessions I attended:

A Half-day of Behavior-driven Development on Rails (rspec)

This was the first session on tutorial day and for me one of the most interesting. It basically gave a looong introduction to behavior driven design, how it evolved from things like TDD and who realized what while enganged in which project back in the good old times. One of the interesting parts for me was when they talked about the whole story writing/specification process. I had heard most of it before but it was a good refreshment. The central statement was to write “Software that matters”, and to achieve this, you’d have to get the specs right – as we XPers know this should be done by collecting user stories. The suggested format for such a story was this:

(more…)