The more layers you add when testing your application the slower these tests become.

A unit test takes a fraction of a second to run. A simple integration test maybe up to a second. When JavaScript and a real browsers gets involved probably longer. But unnecessary long test runs hurt productivity. Up until a couple of weeks ago I had to make trade-offs either in speed, test-coverage or maintainability of my test suite when it came to real full stack integration/acceptance testing. Here is how I perceive these problems.

A standard Cucumber (a framework to write natural readable acceptance tests) setup uses Webrat which interacts with your app through a simple Rails integration session, so no JavaScript. The best what Webrat could do in this scenario, when it comes to JavaScript, is to recognize some common patterns in your html and send according requests – but still, most of the JavaScript code is not covered.

Webrat can also be used to drive Selenium, which interacts with your web app through a real browser. This will cover Javascript, CSS and possible browser issues. But compared to Webrat Selenium is snail mail slow.

With Culerity or Steam there are alternatives to Webrat which use HtmlUnit: HtmlUnit is a “GUI-Less browser for Java programs”. It models HTML documents and provides an API that allows you to load pages, fill out forms, click links, etc… just like you do in your “regular” browser. They are significantly faster than scripting a normal browser with Selenium at the cost that you don’t see how design works or if any browser issues cause trouble and it is still slower than Webrat without a special backend.

In order to run acceptance tests which require full JavaScript support separately from those who don’t to get shorter test runs I use two separate environments as they are generated out of the box either by Cucumber or Culerity. This leads to duplication of the setup code and step definitions, which means that there is more to maintain and more that could possible break.

Capybara addresses the shortcomings of a two environments solution and still allows you to be flexible in what backend you use for integration testing on a per scenario basis in Cucumber. It does that by using Cucumber’s tags feature.

Capybara “is inspired by and aims to replace Webrat as a DSL for interacting with a web application. It is agnostic about the driver running your tests and currently comes bundled with rack-test, Culerity and Selenium support built in”. In the future we certainly will see other backends. I saw that a backend for env-js (a pure JavaScript browser environment) is a already in the making.

Below is an example that runs one scenario with Selenium and the other with Culerity.

  @selenium
  Scenario: Searching a blog post using autocomplete
    Given a blog post titled "Autocomplete made easy" containing "Some example"
    When I go to the search page
      And I fill in "Search" with "Auto"
      And I follow "Autocomplete made easy"
    Then I should see "Some example"

  @culerity
  Scenario: delete found comments
    Given a comment by author "Thilo"
    Given a comment by author "Tilo"
  When I search for "ilo"
    And I check the element with xpath "//p[contains(@class,'delete')]/input"
    And I press "Submit"
  Then I should see "Are you really sure?"
  When I press "Yes"
  Then I should see "Comments were delete"
  And I should see "0 Comments"

You might have noticed the XPath expression in the second scenario. From time to time you will have to deal with XPath expressions when writing custom steps as XPath is used to locate elements within the DOM. The default steps can be scoped with CSS selectors or XPath, Capybara takes care of the conversion to XPath then.

Using Capybara with Cucumber is straightforward:
First get the gem:

gem install capybara

On OSX you may have to install libffi, you can install it via MacPorts with:

sudo port install libffi

Then go to your rails project and generate the cucumber files as followed:

script/generate cucumber --capybara

And you are ready to go to mix and match your acceptance with tags. For further informations like using capybara without cucumber, asynchronous JavaScript etc. checkout Capybara’s README.

Capybara definitely eases maintaining acceptance test and will give impulses to improve other tools, e.g. env-js.

Are you stuck in a maintenance nightmare or simply want to bring your testing skills to the next level? Get in touch with us.

On last week’s RailsCamp UK I started hacking on a new CouchApp called HejHej. Its purpose is to help me learn Swedish, but what’s more important here: I wrote this app BDD style using Cucumber, the famous BDD tool and Culerity, my humble addition that allows me to test any webapp (including client side JavaScript) with it.

The whole thing is available on Github so you can check out the features and steps there. In the following post I will show the necessary steps to test your own CouchApps with Cucumber.
(more…)

On a recent Rails project we have been working a lot with PDF documents. The application generates PDF invoices and account statements, places markers and comments into existing documents and also merges multiple single page PDFs into one. We do all this with a few PDF tools:

• the PDF::Writer library for Ruby to generate PDFs
• pdftk – to merge multiple PDFs into one or overlay them
• pdftotext part of the xpdf library, to extract texts from PDFs

On OSX all of these can be installed via MacPorts. Debian has packages as well.

While PDF::Writer is a Ruby library pdftk is a command line tool. We simple call it using Kernel.system and check the return code via the $? variable.

unless system (“pdftk #{source_pdf} … output #{target_pdf}”)
raise PdfError(“pdftk returned error code #{$?}”)
end

With all this PDF processing the need for testing the contents of the generated documents arose. We have factored out all the PDF processing into a bunch of extra classes which we simply unit test with RSpec: make sure the parameters are passed to the command line correctly, that the right exception is thrown for each return code etc.

In addition to unit testing we also write customer driven acceptance tests with Cucumber, where we assert on a high level the outcome of certain actions. With HTML pages we can simply use the built-in steps that in turn use Webrat to parse the HTML like this:

Given a purchase over 200 EUR
And an invoice
When I go to the start page
And I follow “Invoice”
Then I should see “200 EUR”

Now in our case the invoice link links to a PDF but we still want to know what’s inside the document. The solution we came up with looks like this:

Given a purchase over 200 EUR
And an invoice
When I go to the start page
And I follow the PDF link “Invoice”
Then I should see “200 EUR”

What this does in the background is follow the link as usual, write the response into a temporary file, convert that to text using pdftotext and write the result back into the response. This way we can make assertions about the contents of the PDF almost as if it were an HTML page (except for tags of course). Here is the implementation:

When ‘I follow the PDF link “$label”‘ do |label|
click_link(label)
temp_pdf = Tempfile.new(‘pdf’)
temp_pdf << response.body
temp_pdf.close
temp_txt = Tempfile.new(‘txt’)
temp_txt.close
`pdftotext -q #{temp_pdf.path} #{temp_txt.path}`
response.body = File.read temp_txt.path
end

Since the day we started upstream testing was our obsession. When we began all we had was the Rails built-in unit, functional and integration tests. When we learned about RSpec the world became a brighter place. When the RSpec team released their StoryRunner we were thrilled. We started writing stories in English and were able to run them. Whoa. It soon turned out the first version had some serious issues with staying DRY. More versions were released, we rewrote our stories, step by step StoryRunner got better and we got better, too.

Then Cucumber came out and it was a huge leap ahead again. With the integration of Webrat we could write much shorter stories, but more importantly we could test that our links and forms were working as well. Everything was working perfectly and the world was a happy place – as long as we didn’t use any JavaScript/AJAX…

Welcome to Culerity. Culerity is a small gem that integrates Cucumber with Celerity, a wrapper around HtmlUnit which in turn is a Java library that parses HTML and runs the embedded JavaScript code. And unlike Watir or Selenium this all happens in headless mode: without hijacking your browser.

The problem with Celerity has been that it only works in a JRuby environment, which means you either had to run your application in JRuby as well (which might not even work with certain libraries and plugins) or somehow work your way around it by running your tests in JRuby and your application in whatever Ruby you wanted to use and somehow glue it together. Culerity now fills this gap. After installing it you run your Cucumber features as usual. Celerity now spawns a Java process in the background, sends all the Celerity calls to this process and evaluates the results back in the original Ruby environment — everything works (almost) as if you were running just in your single Ruby process. For an easier start Culerity comes with the same set of Cucumber step definitions that are provided by a default Cucumber/Webrat installation. This means you can reuse your step definitions written for Webrat.

To get started you can either clone the GitHub repository or simply install the gem (langalex-culerity). Then follow the instructions in the README. If you find something is not working for you or could be improved please feel free to fork the source code and show it some love. The codebase is really small right now and fully spec’d.

—

Does your company need help with software testing? We can help: check out our new product Scene Investigation.

yesterday while skimming through my Twitter timeline I stumbled across aslak hellesoy’s cucumber. cucumber is a rewrite of the rspec story runner, allowing you to write user stories in plain text and make them executable by adding a bunch of ruby magic.

the features of both – story runner and cucumber – are basically the same. cucumber just gets rid of all these little glitches that disturb your work flow when working with story runner and adds that little bit of polish that makes the difference. but see for yourself:
(more…)