Agile project management makes the client an integral part of the development process. This article explains what your responsibilities and powers as a client are, and helps you to understand project management the way we see it.

In the traditional (waterfall) model of software development someone writes a specification of the whole software and then the programmers fail to implement it within the given time/budget. To be honest I don’t know a lot about this way of doing software, but I’ve seen it fail.

But I do know a bit about how we write software. In contrast, agile development works in small steps. There is no upfront specification, as we acknowledge that this specification would be incomplete and quickly outdated by changing requirements. Instead we work in short iterations (at upstream typically ~2 weeks) where we only plan ahead for a short amount of time.

By aggressively prioritizing what is most important we are able to release software that fulfills the needs of the business with less clutter in a shorter amount of time.

For me this sentence is really the essence of agile project management as it shows its most important point: prioritizing what is important for your business.

What is software the fulfills the needs of the business? It’s a software that solves your specific problem. If you want to sell something online it allows your customers to click a buy button. If you need to send invoices it will put a bunch of numbers on a sheet.

What is less clutter? Less clutter means if you want to sell something online it neither has a tag cloud nor a facebook like wall for every product where users can post their video reviews. If you need to send invoices it will not let you upload pictures for it.

What is less time? Less time means we deliver your product faster because we spend less time on implementing features because there are fewer features. It does not mean that we can program faster because we are doing agile (seriously we can’t).

What is prioritizing? You do the things first that are most important.

So what is important? And here is where it gets difficult.

What is important is your – the client’s – decision. The problem is that a lot of things can be (thought of as being) important. For a web site, the start page has to have a maximum conversion rate, SEO is important, scalability is important (isn’t it?), usability is important, choice of technology is important, the database schema is important… you get the idea.

But do you know what’s really important? That your business problem is solved. And do you want to know how many kinds of business problems there are? Well, there’s exactly one: make (more) money.

I think it actually is that simple, but from what I’ve seen it can become really hard to focus on that one goal, when you think you have to consider all that other stuff as well, but let me say this again: as a client it’s not your responsibility to think about the database and scalability (you won’t have to scale in the beginning). Your one and only problem is: how do I make money with this. Everything else comes after it.

A good way to focus on solving that problem might be to ask yourself this question: How do I make (more) money?

Case Study

Let’s stick to the example of an online shop and let’s assume you want sell e-books. Of course you look at the competition first, and there is amazon.com. They sell e-books and from what it seems they are making a whole lot of money from it, so that must to be a good starting point. From their site you collect a bunch of requirements you also want in your shop:

• Categories: Books must be in categories so people can find them.
• Tags: With tags people can click on tag clouds and then see all the books to that tag.
• Related books: “People who have bought this might also like…” – I love that feature
• User reviews/ratings: when users review the books we can make a top list and then everyone will buy the top books.
• Licensing: You have talked to the publishers already and they have these complicated licenses where you may only sell a certain book in a certain country but only at full moon when it’s raining – so the shop needs to take all this into account.
• Internationalization: We need at least English, Spanish and Mandarin to reach more markets
• Statistics: You want to know at what time and weather conditions people buy books from where, on which browser and operating system and which IP address they used to do it. From that data you will be able to optimize the hell out of the start page.

So, that looks like a reasonable list of requirements doesn’t it? With this shop you could steal a nice market share from amazon right? Well, let’s take that list to the test: does it help me to make money?

• categories: no
• tags: no
• related books: no
• reviews/ratings: no
• licensing: no
• internationalization: no
• statistics: no

Surprised? I hope at least a little bit. So what would make you some money?

In order to make money
as the owner of the shop
I want people to buy a book

With all the fancy features above you forgot (sorry, I forgot) the only important feature of the site: a button that says “Buy this e-book”. By the way the format of that feature request comes from the BDD (behavior driven development) people. This is the generic template:

In order to <business value>
as <role>
I want <feature>

This is a brilliant template that you can use to take all your feature ideas to the test. Remember that the only valid business value here is to make money. And that the only role that is important here is you as the stakeholder of the project.

Let’s look at your features:

Categories

In order to make money
as the owner of the shop
I want categories

How is a category going to make money? A category is a means to, well, put your books into categories. You might at some point decide that you can sell more books by putting them into categories because that big list of 1000 books on you start page crashes people’s browser and they can’t buy your books anymore, but for now you haven’t even sold a single book yet, so categories are definitely not going to make you any money right now.

Tags

A few years ago someone declared that if you want to be “Web 2.0” you need tags. That’s why a lot of websites have tags nowadays. Like categories you can’t make any money out of tags.

Related books

While this is a popular feature at amazon, your yet to be launched shop wouldn’t even have enough data at the beginning to make any sensible claims about what books are related. Again, at the beginning this won’t matter at all. You can always add it later, when the money making assertion becomes true.

Reviews/Ratings

While reviews can be a good way to promote books and build trust for your customers, when you launch your site there won’t be any reviews because nobody has bought a single book yet. So unless you charge people for being able to write reviews this is not going to pay your rent.

Licensing

Sure, we can build you a system that grabs the weather from weather.com, can calculate the moon phases over the sahara and figure out people’s countries by their IP address, but it’s going to cost you. How about this: You start and market your shop in only one country. This can be your home country or the country with the simplest licensing model. After you have sold enough books there we can start working on the rest (and you can pay us from the money you have made already).

Internationalization (I18n)

Again, we can build you any system you want, but be warned, some people read and write from right to left or top to bottom. WIth I18n you are opening a whole new can of worms. How about we do I18n later, after you have proven that your business model works in one country (or language region, which might cover a lot of countries (thinking English/Spanish/Mandarin)).

Statistics

Sure it’s important to measure the success (or failure) of your business. But before you have a business there isn’t really much to measure is there? And how about this: for now every time someone buys a book you open up your spreadsheet app on your computer and put a date and a price in it. I know it may seem archaic but it works. If you want to go crazy you can even use a real book made of paper, but since you’re selling e-books you probably don’t want that.

All these examples boil down to this: test your features agains the BDD template: can I make more money now when we implement this. A lot of them are really not that important now and you can use something simpler instead.

Conclusion

Deciding what is important is the one big way that you can control the project. Programming takes time and we don’t usually have a lot of ways to make that faster within a project. But choosing wisely the features that you really really need to start selling your products can make the difference between a 4 week and a 6 months project. We do the best we can to help you with those decisions but ultimately they are yours.

If we spend 6 months working on your project and it’s still not done then that’s not a programming problem, it’s a management problem, because we have been working on too many features, of which many probably weren’t as important as you thought (no offense). On the other hand if we make it in a few weeks, we’ll congratulate you and bring some drinks to your early launch party.

(On a side note we recently released cobot.me after only 4 person weeks of development it doesn’t have a lot of features, but that’s a different post.)

When working with our clients we use pivotal tracker to manage requirements and priorities. Which works very well, as it keeps overhead low and offers a very focused communication channel for feature related questions.

The last step for a successful delivery of a feature requires our customer to accept that feature or hand it back to us if something isn’t satisfying. To enable our customers to do that we provide them with a staging environment, which is a non public version of our customers’ product with the latest code integrated so that he can try out new features before they go live. We deploy to the staging system directly from the master branch usually, from time to time we introduce feature branches if we think something takes longer or might need refinement after more customer feedback. By the time of an actual live deployment we normally have ironed out any remaining issues which shown up during the client’s review on the staging system.

This simple system works pretty well and effective if you are in control of the staging environment, have only one person as the customer to talk to and your developers’ communication works like in a hive mind (which it does, mostly ). This system reached its limit when we worked with PaperC lately, one of our valued clients.

We coached their programmer in BDD and pair programming some time ago and still working with them very closely to get improvements and new features out of the door fast. The limits with our simple workflow became visible when I paired with Patrick together for PaperC and they also had a pair on site with us at co.up. At first delivered tickets began to stack up waiting for acceptance, also because the staging system became unstable as issues weren’t fixed fast enough.

Things got more complicated as we put features on staging which weren’t supposed to make it into the next live deployment as they where part of a bigger, not yet thought to an end, interface change. These features just should be shown in action to see how things can go on from what we did so far. This incident cost us some extra time to put these feature related commits out of the master branch into a feature branch. So we switched to create a feature branch for everything that was more than “add a new input field to that”. This left us with a lot of branches we had to keep track of when deploying to the staging system, but at least we could undo merge commits easily when a feature wasn’t suppose to be deployed live. Then we started a discussion on how we could do this in a better way. Before I describe the workflow which I think is best suited, as we already employed it successfully in the past, I’d like to mention two suggestions I’m not comfortable with an why.

One idea was the introduction of pair branches, where every pair commit their work to. And these branches would be merged into master for acceptance. Apart from not really resolving the issue, it just mitigates it onto a pair branch, and it contradicts the principle of collective code ownership.

Another proposal was to create a special production branch and selectively cherry-pick commits which are supposed to go live. This points into the right direction, as it gives you more control of what goes live, but it might leave you with a lot of commits to pick from, if you have a developer like me who commits rather often. And it doesn’t give you a single point to roll back from if a feature that has multiple commits get pulled off the release.

So now for my preferred and practically proven solution. First off, it requires an extra server which I refer to as experimental. It is a lightweight (not so close to production setup) staging system with a database schema that should be easily migrate-able up and down like on a CI server. It should be fed with some test data automatically. It doesn’t have to be as much data as on staging, just enough to try out most features.

Now to the git workflow part. Smaller changes and bug fixes go directly to master as always. Bigger features should be done in separate branches and merged to master once things are done (Don’t forget to delete the remotes if the feature gets accepted). The master branch should be deployed to the experimental server at least once a day, likely more often so that features can be quickly accepted. Features that aren’t necessarily ready for production yet can be deployed to experimental for acceptance testing simply by changing the branch from which the experimental environment should be deployed from the master branch to the feature branch and rebuild the database if required. Before an imminent release the master should be pulled to staging so that any remaining issues can be fixed. Such fixes should happen on the staging branch and can later be merged back to master. If everything is fine, merge staging to production and deploy to the live system if you are ready. This way you always have a clean production branch, it’s easier to pull off features from a release and get faster feedback for completed features without worrying to mess up the next production release.

What is your approach to managing commits and releases? Do you find this post helpful? Feel free to leave some feedback in the comments.

As of last week upstream is now present in 2 more countries: Finland and Sweden. We see a strong increase in the web development market in the scandinavian area so we want to be there before the rest of the world notices. Ok ok, so actually that’s not the reason. Both Frank and I moved for personal reasons.

Frank is now situated in Turku, Finland and I’m in Lund, Sweden. We will both be staying there for about half a year and continue to work for upstream remotely. As our company pair programs all the time so are we – only remotely now, using iChat’s screen sharing/audio chat and good old Textmate. It actually works smoother than it did when we tested it in the office’s wireless network. Scandinavian communication infrastructure is pretty good.

While we are here in the north we will be attending and possibly speaking at a few conferences in the area like Nordic Ruby and the Scandinavian Web Developer Conference (SWDC).

If you are interested or know anyone who would want to hire us for a web project either in Sweden or Finland please drop us a note. While remotely working for the Berlin office is ok it would be nice to work on our own. We create high quality, fully tested webapps in a few week’s time, we help with code quality and scalability problems, we can do training/coaching on various programming/agile topics, remotely or on site.

Oh and if you don’t have a project for us but live in the area say hi so we can invite you for a beer or two.

Upstream is taking the december off. Except from support for running projects we haven’t taken on any new projects for this month. The reason is that we wanted some time to think about the future of our company, but also to have time for other things.

We have already done this for a weekend in January when we went to Maine. It was a fun weekend but we spent a lot of time walking around at the beach and in the sauna, which left too little time for developing any significant ideas. So this time we are doing it for a whole month.

For the thinking part we will be doing a few meeting throughout the month to talk about different aspects, like the kinds of customers we want to attract in the future, the kind of work we want to be doing, how much and where we will want to work and also how much money we want to be making with all this. We are not going to do any fancy brainstorming-like things. Instead everyone spends their time doing whatever he or she thinks will produce the best ideas – be it sleeping in every day, long breakfasts or running around in the park. We will then gather occasionally to collect and exchange ideas.

For the “doing other things” part: first of all we want to give our new office and coworking space some love. The walls are all white and empty yet and there are still a few ikea boxes stacking up. Another part is to attend some of the many events Berlin has to offer, but that we mostly have to skip because of work.

I already spent yesterday at the coworking day, where Germany’s coworking community gathered to discuss the issues of setting up and running coworking spaces. That was my first day off and I already got a few new ideas from it that could help upstream in the future. Next up is TedX Kreuzberg, where the smart people of Kreuzberg will come and tell us about their brilliant ideas. I’m very much looking forward to that.

I will report back on our progress here from time to time. So far I can only say: “Please try this at home”. Not having to worry about projects and day to day problems all the time is very refreshing and I am confident 2010 will be a great year for us.

We just moved into our new office at Adalbertstrasse 7, 10999 Berlin, the old one just got too small. Too small for Upstream and too small to offer a good co-working experience. During the last years, also with the help of our co-working space, we were able to grow our network of skilled freelancing professionals which allow us to do more and bigger projects and offer our customers the best people for the job.

We had a good time with our old office. For example we co-worked together with Jan Lehnardt (@janl) of CouchDB fame, taught Ruby and TDD to then still new developers like Lena Herrmann (@kilaulena), found skilled developers like Patrick Hüsler (@phuesler), found great designers like Kristina Schneider (@kriesse), won good PHP Developer like Robin Mehner (@rmehner) over to Ruby, ran DevHouses and Cockpit Nights to learn and share, coded with international recognized personalities like Pat Allan (@pat), had lots of well known guests in our office and some good parties too. But the old upstream office just got to its limits. I’m happy that all our co-workers moved with us.

The new office will allow us to keep a productive and inspiring work environment, which I think is essential to deliver great products, and evolve all the great things we did so far. For example Alex will expand our efforts in providing an affordable space for independent workers that combines a relaxed and social atmosphere with the infrastructure and productivity of a well equipped work place with co.up (@co_up). This way I think we will improve our network even further and get peoples and ideas together for the best outcome possible.

Another great thing about the new office is that its interior and its central location next to Kottbusser Tor underground station allow us to keep hosting the Ruby User Group Berlin meet-up as part of our involvement in the local developer scene.

So if you are in need of a desk (see co.up for details), want to get in touch with the tech scene or are looking for skilled people for your project, visit us in our new office.

I look forward to a great time with more great people and hope that our recently started community photo wall will grow fast with photos of well known and yet not so well known faces.

Boom amazing in action at JSConf.EU

Last weekend at JSConf.EU I gave another talk on CouchDB and CouchApps. This gave me an excellent excuse to hack on boom amazing instead of preparing my talk. Boom amazing is my personal presentation tool. Instead of slides my presentation is laid out on a single large surface and I can pan around, zoom in and out and rotate my viewpoint to show specific contents.

In the end I managed to add a bunch of new features and still get my talk done. The following post explains how boom amazing works and how you can create your own fancy presentations with it.

What is behind it?

Boom amazing consists of a few relatively simple building blocks:

• One or more SVG files that contain all the texts and graphics
• a CouchApp that includes some HTML and CSS, and JavaScript
• some client side JavaScript to handle manipulating SVG and moving around (using JQuery and the jQuery SVG plugin)
• CouchDB: stores all the data: the presentation itself (as document attachments), the CouchApp and all the camera positions and data for replaying presentations
• A browser that displays the SVG file, I use PlainView which has a fullscreen mode.

Read the rest of this entry »

I just released Couch Potato 0.2.14 and amongst other things it has a new feature i think is pretty neat: you can unit test your (JavaScript) views using RSpec and Ruby.

You can declare a view in Couch Potato like this:

class Comment
  property :post_id
  view :by_post_id, :key => :post_id
end

This will generate a pair of map/reduce function and push them to Couch Potato. The map function looks something like this:

function(doc) {
  if(doc.ruby_class == 'Comment') {
    emit(doc.post_id, 1);
  }
}

And here’s a unit test for that function:

describe Comment, 'by_post_id' do
  it "should map to the post_id" do
    Comment.by_post_id.should map({:ruby_class => 'Comment', :post_id => 3}).to([3, null])
  end
end

As you can see all you have to do is pass a Ruby Hash that looks like your CouchDB document and the expected results. You can also pass in multiple results if you expect your map function to emit multiple key/value pairs.

Testing a reduce function works the same way:

describe Comment, 'by_post_id' do
  it "should reduce to the number of comments" do
    Comment.by_post_id.should reduce([], [1, 1, 1]).to(3)
  end
end

For testing re-reducing you simply call .should rereduce(...).to(...).

How it works

So how come you can test JavaScript functions in pure Ruby? Well, by stealing other people’s tricks. I recently contributed a few patches to mustache.js which is a new templating library ported to JavaScript by @janl. It has a test runner currently implemented in Ruby which generates JavaScript code on the fly, runs it using Spidermonkey and reads back the results. I have added a few steps to this process:

1. A custom RSpec matcher collects the map function, a Ruby Hash representing the input and the expected output
2. The input is converted to JavaScript using the JSON gem
3. JavaScript code is generated that runs the document through the map function and prints the resulting JSON.
4. The Ruby code runs spidermonkey, collects the output and parses it back to Ruby using again the JSON gem
5. The results are compared to the expected values.

You can see how it works by looking at the code for the RSpec matchers.

I think this addition lowers the barrier to test your views quite a bit. Although I’m usually not a big fan of “one language to rule them all” and love writing JavaScript, being able to write all the necessary tests in Ruby when working on a Ruby project makes things way easier.

On last week’s RailsCamp UK I started hacking on a new CouchApp called HejHej. Its purpose is to help me learn Swedish, but what’s more important here: I wrote this app BDD style using Cucumber, the famous BDD tool and Culerity, my humble addition that allows me to test any webapp (including client side JavaScript) with it.

The whole thing is available on Github so you can check out the features and steps there. In the following post I will show the necessary steps to test your own CouchApps with Cucumber.
Read the rest of this entry »

Last weekend the second DevHouseBerlin took place. A public event and the perfect excuse for you to not be available for your non-geek-, girl- and boyfriends during the weekend.

DevHouseBerlin stands in the tradition of the SuperHappyDevHouse. For one Weekend we opened up the office space from finnlabs, rocket rentals and us again to host everybody who is curious about technology in general and programming in particular, offering the opportunity to work on, show off or discuss whatever you like together. We provided music, drinks – especially Club Mate -, food – super delicious deluxe sandwiches – and other basic infrastructure, although our supposedly capable new WiFi router failed miserably on serving 30+ devices.

The 25 available seats were gone weeks before the actual event so we didn’t blog beforehand so that you wouldn’t be disappointed. But as we will move to a bigger office soon we will scale the next DevHouseBerlin too. So here are some highlights and results you missed.

We had an international guest and hero with us, Pat. During the last weeks he got the full blown Berlin tech culture while pairing with us, giving his sphinx talk at rug-b (Ruby User Group Berlin) and joining the dev house, where he obviously had fun although he was constantly besieged by fans . He even managed to get some new features for thinking sphinx done (1, 2).

We also planned a realtime battle challenge, a programming game, in which robots controlled by programs are fighting each other. But despite collaborative effort we couldn’t get it running for OS X so alex, till, lukas and pat took instead the challenge to re-implement it with ruby and javascript. A slightly too big endeavor for these two days with so many other things going on. See their progress and fork it on github.

A major source of distraction was the Segway which @jodok parked in the office. Although you look totally dorky it is a lot of fun, even just riding down the corridor.

Dispite the calamity on the corridor Till and Christian proved that you can write web apps fast and well with PHP, they rewrote planet php, a php blog aggregator, in a couple of hours on the first evening.

Dmitry and Hans-Gunther brought a 19″ server rack to play around with the Xen Hypervisor, well a 19″ rack is certainly the right stage for Xen. Hans-Gunther also saved our evening by providing back-up Internet through his G1 while our router was recovering.

Something else which had to do with big scales was Mahout, a bunch of machine learning libraries built on Hadoop, an open-source implementation of frameworks for reliable, scalable, distributed computing and data storage. Some really mind bending stuff if you ask me. Too bad I missed the presentation by MaineC. If you are still studying you might want to attend her Mahout course at the TU-Berlin.

You and I also missed the discussion about the best editor – emacs or vim – and – even worse – CouchDB in 30 minutes. An Introduction to CouchDB, the schema-free document-oriented Database, given by Jan, the european mister CouchDB himself. Of course he was around all the time during the DevHouse and gave help to get on track with CouchDB and related stuff.

I could rave on and on how cool the second DevHouseBerlin was, but just take a look at the wiki page where you can find more projects, people and photos. So that even if you were not with us this time you might take something out of it.

See you next time at DevHouseBerlin.

Yesterday I added a new feature to Couch Potato, my persistence layer for CouchDB that I call Dynamically Inherited Methods™ and found the implementation I came up with interesting enough to share. Here’s the problem:

class User

  def self.string_attr_reader(name)
    define_method name do
      instance_variable_get("@#{name}").to_s
    end
  end

  string_attr_reader :nickname
  attr_writer :nickname
end

user = User.new
user.nickname = 123
user.nickname # => '123'

This code adds a macro to the class user called string_attr_reader. When you call this macro on the class and pass it a name it generates a method that returns the instance of the same name converted to a string. This all works perfectly until you want to override the generated method to add some behavior:

class User
  def nickname
    super.gsub('6', '7')
  end
end

The enhanced method now replaces all occurrences of 6 with a 7 – except it doesn’t. When you run the above code you will get an exception where Ruby complains there is no super to call. That’s because we defined the nickname method in the User class so we didn’t inherit it, hence no super.

The one way out of this is to use alias_method, or alias_method_chain when you are using Rails:

class User
  def nickname_with_six_replaced
    nickname_without_six_replaced.gsub('6', '7')
  end
  alias_method_chain :nickname, :six_replaced
end

First of all this isn’t very beautiful anymore, and secondly using alias_method_chain when you could use standard object oriented metaphors (like inheritance) doesn’t make a lot of sense – except that you can call yourself a metaprogramming programmer, yay.

Anyway, there is another way – actually involving much funkier meta programming – to solve the problem:

class User
  def self.string_attr_reader(name)
    accessor_module.module_eval do
      define_method name do
        instance_variable_get("@#{name}").to_s
      end
    end
  end

  private

  def self.accessor_module
    unless const_defined?('AccessorMethods')
      accessor_module = const_set('AccessorMethods', Module.new)
      include accessor_module
    end
    const_get('AccessorMethods')
  end
end

The updated code now dynamically creates a Module called User::AccessorMethods and includes it into the User class. All methods generated by string_attr_reader are now added to that new module instead of the class. The result is that when overriding those methods you can now call super because they’re are inherited from the new module.

class User
  attr_writer :nickname
  string_attr_reader :nickname

  def nickname
    super.gsub('6', '7')
  end
end

user = User.new
user.nickname = 678
user.nickname # => '778'

While this code involves some fairly crazy metaprogramming which would be too much for a simple example like the above, I think libraries like CouchPotato can still benefit, as the application code can become cleaner by not having to do any metaprogramming but resort to standard object oriented ways of programming.