Boom amazing in action at JSConf.EU

Last weekend at JSConf.EU I gave another talk on CouchDB and CouchApps. This gave me an excellent excuse to hack on boom amazing instead of preparing my talk. Boom amazing is my personal presentation tool. Instead of slides my presentation is laid out on a single large surface and I can pan around, zoom in and out and rotate my viewpoint to show specific contents.

In the end I managed to add a bunch of new features and still get my talk done. The following post explains how boom amazing works and how you can create your own fancy presentations with it.

What is behind it?

Boom amazing consists of a few relatively simple building blocks:

• One or more SVG files that contain all the texts and graphics
• a CouchApp that includes some HTML and CSS, and JavaScript
• some client side JavaScript to handle manipulating SVG and moving around (using JQuery and the jQuery SVG plugin)
• CouchDB: stores all the data: the presentation itself (as document attachments), the CouchApp and all the camera positions and data for replaying presentations
• A browser that displays the SVG file, I use PlainView which has a fullscreen mode.

(more…)

I just released Couch Potato 0.2.14 and amongst other things it has a new feature i think is pretty neat: you can unit test your (JavaScript) views using RSpec and Ruby.

You can declare a view in Couch Potato like this:

class Comment
  property :post_id
  view :by_post_id, :key => :post_id
end

This will generate a pair of map/reduce function and push them to Couch Potato. The map function looks something like this:

function(doc) {
  if(doc.ruby_class == 'Comment') {
    emit(doc.post_id, 1);
  }
}

And here’s a unit test for that function:

describe Comment, 'by_post_id' do
  it "should map to the post_id" do
    Comment.by_post_id.should map({:ruby_class => 'Comment', :post_id => 3}).to([3, null])
  end
end

As you can see all you have to do is pass a Ruby Hash that looks like your CouchDB document and the expected results. You can also pass in multiple results if you expect your map function to emit multiple key/value pairs.

Testing a reduce function works the same way:

describe Comment, 'by_post_id' do
  it "should reduce to the number of comments" do
    Comment.by_post_id.should reduce([], [1, 1, 1]).to(3)
  end
end

For testing re-reducing you simply call .should rereduce(...).to(...).

How it works

So how come you can test JavaScript functions in pure Ruby? Well, by stealing other people’s tricks. I recently contributed a few patches to mustache.js which is a new templating library ported to JavaScript by @janl. It has a test runner currently implemented in Ruby which generates JavaScript code on the fly, runs it using Spidermonkey and reads back the results. I have added a few steps to this process:

1. A custom RSpec matcher collects the map function, a Ruby Hash representing the input and the expected output
2. The input is converted to JavaScript using the JSON gem
3. JavaScript code is generated that runs the document through the map function and prints the resulting JSON.
4. The Ruby code runs spidermonkey, collects the output and parses it back to Ruby using again the JSON gem
5. The results are compared to the expected values.

You can see how it works by looking at the code for the RSpec matchers.

I think this addition lowers the barrier to test your views quite a bit. Although I’m usually not a big fan of “one language to rule them all” and love writing JavaScript, being able to write all the necessary tests in Ruby when working on a Ruby project makes things way easier.

On last week’s RailsCamp UK I started hacking on a new CouchApp called HejHej. Its purpose is to help me learn Swedish, but what’s more important here: I wrote this app BDD style using Cucumber, the famous BDD tool and Culerity, my humble addition that allows me to test any webapp (including client side JavaScript) with it.

The whole thing is available on Github so you can check out the features and steps there. In the following post I will show the necessary steps to test your own CouchApps with Cucumber.
(more…)

Wow, this was the month of conferences. First we visited Barcelona for the Euruko: a great conference taking place every year in a different city. We attended it the 3rd time. The talks ranged from practical like “Cooking with Chef“, over entertaining like “Fun with Ruby (and without R***s), program your own games with Gosu” to just geeky like the lightning talk about “Vimmish and how much fun gramma parsers can be”. I really liked the 2 days 1 track format and the people I met there. And you can be sure that we will be in Krakow next year too.

5 days later I visited Amsterdam for Ruby on OS X, which I already blogged about.

And a little more than a week later, until yesterday, four of us attended the RailsWayCon here in Berlin, which tries to fill the gab that the RailsConf Europe left. The first day was reserved for whole day tutorial sessions.

The second day offered a lot of advanced topics to choose from in 3 tracks. I chose to hear more about Asynchronous Processing from Mathias Meyer and the nitty gritty details about Events from Lourens Naudé. The keynote about the Present and Future of Programming Languages by Ola Bini was also very interesting.

Upstream also took an active part at the conference: Alex gave his talk about CouchDB Frameworks for Ruby and CouchApp (using his new presentation tool boom_amazing) and I introduced MacRuby, the Ruby that plays nice with Objective-C.

On the third day Yehuda Katz revealed some more details about Rails 3. The talk by Michael Koziarski about Rails Performance was good for a reality check. In the afternoon everybody was tired after 3 days of conference and the talks lost quality. Still a very good conference with potential.

After my talk about Ruby CouchDB frameworks at Scotland on Rails where I dismissed a few of of the libraries available (including my own Couhch Potato) as not fitting the CouchDB way of doing things, I have been hacking away the past few weeks working on a complete overhaul of Couch Potato.

As a first result I have just released version 0.2 of the framework. Its new goals are simplicity, embracing the CouchDB semantics and testability. In order to achieve this I had to introduce some major changes:

I disconnected models from the database – there are no more save/get/find methods in the models. Instead you can hand the models to a database object that will save/load documents for you – this allows for unit tests that are completely disconnected from the database

I have dropped associations and thrown away all the ActiveRecord like view creation/querying, replacing it with a new, more CouchDB like system. That new system is lighter, simpler and easier to extend.

The following paragraphs will show you how to work with the new Couch Potato.

Saving / loading models

As I said I have decoupled the models from the database, a model doesn’t have permanent access to the database anymore. Instead you instantiate a database object yourself and tell it to load or save a model object. This change isn’t so much about CouchDB as it is about testability. Having the database separated means you can now have true unit test of your models without talking to the database. Here is how you save/load models:

class Book
include CouchPotato::Persistence
property :title
end

CouchPotato::Config.database_name = ‘my_db’ # in Rails this is done for you
db = CouchPotato.database

book = Book.new :title => ‘The Passionate Programmer’ # good book
db.save! book # saves the book or raises an exception

db.load book._id # the original book

The database is responsible for running the model’s validations and lifecycle callbacks, saving the document to the database and afterwards setting the _id and _rev on the model. This way a lot of the persistence related logic is removed from the models making them more lightweight and most importantly easier to test (see below).

New Views

Overhauling of the views had two main goals:

• provide a simple and extensible way for saving/querying views that works the way CouchDB works
• since models don’t have access to the database anymore, find a new way to query them

Here is how you create and query a simple view:

class Book
view :by_title, :key => :title
property :title
end

db = CouchPotato.database
db.view Book.by_title # no parametters
db.view Book.by_title(:key => ‘The Passionate Programmer’, :descending => true) # just use any of the parameters CouchDB accepts

The way views work is now essentially reversed (inversion of control, rings a bell?). Instead of the view method calling the database the new view method creates a specification for a CouchDB view. The view is passed to the actual database in order to query CouchDB. Again this makes testing easier (see next section) but it also gives Couch Potato a clean interface in order to make creating views easier through abstractions. There is now a hierarchy of view specification classes that you can use to more easily create and query views. The above example used the CouchPotato::View::ModelViewSpec which is the default. This spec creates a view whose map function emits one (or an array of) attribute of the model and returns full documents.

Other view specs let you create more customized views, for example the RawViewSpec, which lets you define your own map/reduce functions and return the raw CouchDB results hash.

class Book
view :title_length, :type => :raw, :map => “function(doc) emit(doc.title.length, null)}”
property :title
end

db.view Book.title_length # returns something like {:rows => [{:key => 25}]}

For more examples see the Documentation in the CouchPotato::View::*ViewSpec classes.

Testing

As I have mentioned repeatedly decoupling the database from the models makes testing easier. With the new Couch Potato you can unit test your models without hitting the database once. First of all that makes your test lightning fast, and secondly it allows you to more easily test for example your lifecycle callbacks because you can call them directly passing them a stub or mock database.

Here’s an example: (with RSpec)

class Book
property :title
property :slug
end

describe Book, ‘create’ do
it ’should generate a slug’ do
book = Book.new :title => ‘The Passionate Programmer’
book.run_callbacks :before_create
book.slug.should == ‘the-passionate-programmer’
end
end

If you don’t like calling the run_callbacks method you can also use the actual database object but without a real connection to CouchDB. Couch Potato is based on the excellent CouchRest – a more low level CouchDB framework. The CouchPotato::Database constructor expects an instance of a CouchRest database which we can replace with a stub:

describe Book, ‘create’ do
it ’should generate a slug’ do
book = Book.new :title => ‘The Passionate Programmer’
db = CouchPotato::Database.new stub(‘couchrest database’, :save_doc => {})
db.save book
book.slug.should == ‘the-passionate-programmer’
end
end

For more details and examples please see the README, the RDocs and watch this blog. If you want to know all about Couch Potato I encourage you to read through its source code and specs – it’s not that much code actually. Note that all this is still work in progress and time will show how well all of this works. I would be happy to hear your feedback.

After my Ruby sittin’ on the Couch talk about Ruby Frameworks and CouchDB at Scotland on Rails last week a bit of a debate started. In the talk I did compare the available Ruby Frameworks and discussed how well they fit the CouchDB way of doing things. In my conclusion I recommended using either CouchRest or RelaxDB for development and at the same time urged people not to use one of the ActiveRecord like libraries like CouchFoo.

As it seems to me the intention of my talk hasn’t reached all of its audience yet I’ll try to make my point again using this blog post.

The frameworks I looked at could be divided into two classes: the ones using CouchDB semantics (e.g. CouchRest/RelaxDB) and the ones trying to provide an ActiveRecord like interface for the applications (e.g. CouchFoo). The reasons I recommended not to use ActiveRecord semantics are:

CouchDB is not about relations

In ActiveRecord we have to model our domain models so they fit into a relational schema. That means flat tables and relationships between two tables, through a third and fourth table etc. To get results from the database we usually join a few tables and get back the resulting rows, nicely converted into Ruby objects for us. That is (sort of) fine for a relational database because all it has are those tables but it doesn’t work at all with CouchDB.

To use CouchDB to its full potential you need to understand and use its views

CouchDB doesn’t have a concept of tables at all. And the way you pull your data from CouchDB is fundamentally different. Instead of joining data from different tables via an SQL query you procedurally build up an index of data by providing the map and reduce functions which you then query.

In order to fully use CouchDB you have to write custom map/reduce functions and abstracting that into an API that was designed for generating SQL queries doesn’t allow you that.

I could go into more details but that really is my main point. I know from my experience that changing the wiring in your head after too many years of ActiveRecord is hard so I don’t expect anyone to immediately agree with me but I do believe that the only viable way of using CouchDB is through the interface that was designed for it and not by an abstractions that just happened to be there first.

I’m looking forward to a lively discussion, either in the comments or on other blogs.

Last week was my first Erlang gig. I paired up with Jan Lehnardt (core committer of CouchDB). Our goal was to implement a new statistics module for CouchDB within one week. Jan knew more about the CouchDB code and Erlang, I contributed my knowledge about testing and pair programming.

Integration Testing in JavaScript

Since the testing infrastructure in CouchDB left some room for improvement we decided to introduce some new concepts. The existing tests were written in JavaScript penetrating the database via its HTTP/JSON interface. The tests were really long and coarse grained which lead to a problem: When one of the tests broke it wasn’t very easy to figure out which one. We didn’t want to replace or add too much to the existing, self-made test framework so we just added a bit of sugar: BDD style specs with meaningful names:

var response_count_tests = {
’should show the number of 404 responses’: function(name) {
// Given
restartServer();

// When
CouchDB.request(‘GET’, ‘/some_nonexistant_url’);

// Then
TEquals(1, CouchDB.stats_request(‘httpd’, ‘not_found’), name);
}
};

var all_tests = [response_count_tests, ...];

for(var i in all_tests) {
test_group = all_tests[i];
for(var j in test_group {
test_group[j](j);
}
}

The TEquals function is just our extension of an existing function T which is responsible for reporting errors when the assertion doesn’t match. By passing the name of the test spec we can now display that name along with the error message:

Error in ’should show the number of 404 responses’: ‘CouchDB.stats_request(‘httpd’, ‘not_found’)’ should be ‘1′ but was ‘0′.

Unit Testing in Erlang

In addition to those integration tests we wanted to add some unit testing on a lower level. Turns out the only unit testing framework for Erlang seems to be EUnit, which is a very barebones (at least when you come from something like RSpec) xUnit type of tool. You define a test by appending “_test” to your function name and then you have a bunch of assert statements at your disposal. Well that’s at least something and it worked. Here is an example:

should_return_zero_if_nothing_has_been_counted_yet_test() ->
test_setup(fun() ->
Result = ?MODULE:get({httpd, average_request_count}),
?assertEqual(< <"0.00">>, Result)
end).

We added the test_setup for starting and tearing down processes needed for our tests. Then we pass a fun with the actual test code. We got through the week pretty ok with this but I really missed a few things:

Erlang

I’ve read the book, I’ve watched numerous screencasts but this was my first real life project in Erlang. Many people complain about its syntax and while this isn’t the most beautiful language I’ve ever seen I have to say it’s not really a big problem. I got used to it on the first day.

What is really powerful in Erlang is pattern matching. Instead of simply assigning a value to a variable you can always match against the structure of that data to extract parts of it:

Data = {“Hans”, “Wurst”, {born, 1980, 10, 23}}.
{FirstName, _, {born, Year, _, Day} = Data.

This example lets you pull out numerous details from the tuple in a single statement. You can do the same for case statements or when overloading functions. So this pattern matching is a powerful concept seen everywhere in Erlang code. It makes the code you write pretty dense which is cool on the one hand but sometimes also makes it hard to read so you have to be careful how much you want to cram into a single line of code.

CouchDB is built using make – which felt like 1990. I’ve never used make much myself so I’m not really qualified to talk about it but we did spend a good amount of time in our MakeFile because of some whitespace problem, and we still haven’t managed to integrate our new tests into the build process. I’d love to see something like Rake being used instead.

All in all this have been very pleasant first steps though. Pairing up for this turned out to be a very good idea, I have learned a lot about Erlang and am eager for more now.

Scotland on Rails will take place On March 26-28 2009 in Edinburgh. The organizers have just published the list of speakers. Pretty nice lineup including Jim Weirich, Yehuda Katz, Pat Maddox, Chad Fowler and more. And the best thing: I will be speaking about using CouchDB with Ruby/Rails, starring live coding and couch potato – my own little persistence layer for CouchDB.

The DevHouseBerlin weekend is over, but don’t worry if you missed this one. This first one was such a success that we will do more of them.

Thanks to all of those who came, helped, organized and shared their knowledge to make this such fun. I’m not lying when saying that everything turned out to be better than expected.

The upfront organization was minimal. Special thanks to Jan and Alex! We even got some unexpected PR beforehand thanks to an online article about the DevHouse. In response we bumped up the available slots to 20. Which were all filled up soon.

The location Box 119 which houses the offices of upstream, finnlabs and rocket rentals was an ideal location for such an event with its medium large office rooms and a bigger foyer. Box 119 had a solid infrastructure, the WiFi and Internet didn’t let us down once. We even had a decent coffee machine, major factor!

Saturday was packed. I guessed more than 30 people at once filled the DevHouse. Everybody was very open and eager to share. The presentation slots for the evening were filled up already by the early afternoon. I even lost my planned slot because of getting out of bed a bit later than planned.

Right from the beginning major hacking was going on. In the afternoon the talks started. In parallel there was tons of time to chat, eat and hack at will within a very relaxing atmosphere.

PHP raised in my opinion again, after I learned through Falko’s Meta Programming PHP talk that it also has a method_missing equivalent. Nico gave some helpful user experience insights by analyzing start pages on the projector. It’s amazing how many – in retrospective obvious things – you can do wrong on your start page. With Fabian we had a non technical discussion about Economy, after his talk about Fixing Money, which unfortunately left out the solution. After that talk I heard some creepy stories from The Enterprise at dinner.

At 2am at least a dozen people were still in the house. I left at 3:30am still full awake thanks to the optimal Club Mate supply.

Sunday wasn’t that busy. People dropped in later, I had plenty of time to get a presentation slot this time. Some realtime programming battle was going on in the afternoon. You can find Gregor’s battle tank A.I. on Github. Filip showed some quick code kung fu by building a poll app with Django in his talk. This little python Framework is worth a look, if you need to get up small apps fast.
My first talk on Bug Fighting with TDD went well. People seemed to be very interested in the subject. I had some time to play with JRuby and was able to get HtmlUnit to run with Webrat, although my solution is still sort of hackish.

CouchDB was all over the place of course with Jan around. He helped interested folks to get it up an running and helped out with problems related to CouchDB. Alex used the time to enhance his CouchDB persistence layer couch_potato. It will make the transition for rails people much easier.

People who stayed late this sunday – which excludes me – witnessed as the very first the rise of cloudplayer the most beautiful online music player in the world wide interweb, that Erik & Henrik gave the last touches at the DevHouse.

These where just some of the talks, projects and peoples that I get to know that weekend. There was so much more as you can see from the photos and the wiki.

Wow, it was just great. And next time will be even better. We will tell you more soon on this

Update: the gem is now available, see the installation instructions below.

After several weeks of incubating on my computer it’s finally time to get real: I have just open sourced Couch Potato under the MIT license. You can get Couch Potato on github now. For an introduction to CouchDB and ruby please read my previous blog post A CouchDB primer for an ActiveRecord mindset. The following is a very short introduction into using Couch Potato. If you want to know more you can start with the README.

The goal of Couch Potato is to create a migration path for users of ActiveRecord and other object relational mappers to port their applications to CouchDB. It therefore offers a basic set of the functionality provided by most ORMs and adds functionality unique to CouchDB on top.

Installation

Couch Potato is available as a gem from http://gems.github.com, so you can just do

$ sudo gem source –add http://gems.github.com # if you haven’t alread
$ sudo gem install langalex-couch_potato

$ irb
>> require ‘rubygems’
>> gem ‘couch_potato’
>> require ‘couch_potato’
>> CouchPotato::Config.database_name = ‘name of the db’

Alternatively you can download the sources from github. If you are using rails just copy the files into vendor/plugins, create a RAILS_ROOT/config/couchdb.yml file (see the README for the format) and you are ready to go. For other applications you will have to require the lib/couch_potato.rb file and then set the database name by calling CouchPotato::Config.database_name = 'name of the db'.

As Couch Potato is still very young you can expect its feature set to grow quite a bit in the near future. What you can download now is the very core together with a few features giving you a glimpse of what is about to come:

Persistence

Create a new class and make its instances persistable by including the Persistence module. As there is no schema in a CouchDB you have to declare the properties you want to persist:

class User
include CouchPotato::Persistence

property :name
end

Now you can save your objects:

user = User.new :name => ‘joe’
user.save # or save!

Properties:

user.name # => ‘joe’
user.name = {:first => ['joe', 'joey'], :last => ‘doe’, :middle => ‘J’} # you can set any ruby object that responds_to :to_json
user._id # => “02097f33a0046123f1ebc0ebb6937269″
user.created_at # => Fri Oct 24 19:05:54 +0200 2008

You can of course also retrieve your instance:

User.get “02097f33a0046123f1ebc0ebb6937269″ # => < #User 0x3075>

Associations

As of now has_many and belongs_to are supported. By default the associated objects are stored in separate documents linked via foreign keys just like in relational databases.

class User
has_many :addresses, :dependent => :destroy
end

class Address
belongs_to :user
property :street
end

user = User.new
user.addresses.build :street => ‘potato way’
user.addresses.first # => < #Address 0x987>
user.addresses.create! # raises an exception as street is blank
user.addresses.first.user == user # => true

When saving an object all associated objects are automatically saved as well. All these save operations are sent to CouchDB in one operation which means the whole process is atomic across all objects saved, plus only one database roundtrip is required making it much faster.

As CouchDB can not only store flat structures you also store associations inline:

class User
has_many :addresses, :stored => :inline
end

This will store the addresses of the user as an array within your CouchDB document.

(more…)