The #1 Software Development Post for Sep.'09

Best Dundies Ever - Pam (The Office)

Actually, I believe that on September 9th I read the best post for the whole year. Since I can't wait till December 31st, I decided to celebrate earlier, by handing out the "Best September Post" award. I am referring to Stephan Schmidt's The high cost of overhead when working in parallel piece.

Stephan's post argues that working on several projects in parallel is much less efficient than working one project at a time. He shows that by calculating the time spent on status meetings throughout the whole period. Typically, the scheduling of such meetings is based on calendar events (weekly, bi-weekly, monthly, etc.) and not on measured progress (because measuring productivity is hard). Thus, when working in parallel you will have more meetings for the same amount of project progress. As a direct result, you end up with a higher meetings/real-work ratio.

Stephan's bottom line, that working in parallel is less efficient than working sequentially, may seem quite trivial. Most developers already know that context-switching between projects incurs significant costs in time and mental energy. They know they are more productive when they can concentrate on one goal. They know that working on several projects is not-unlike juggling: you get to touch each one, but for very little time.

So, how come a post with a trivial conclusion is my #1 post this month? It's all about the argument. Stephan did a brilliant job taking a well known phenomena and providing a solid explanation for it. This is remarkable because most of the practices in the field of software engineering have only a vague explanation which often leads to endless, pointless, debates.

Even unit testing, which is clearly one of the most effective practices, has no real explanation. We do unit testing because it works, but there is no explanation as to how come testing your program against very few inputs dramatically improves its correctness with respect to an infinite space of "real world" inputs.

Stephan's post managed to do it. It supplies a rock solid, non disputable, easy-to-digest, explanation to a software development phenomena. This is not a trivial thing. Go read it.

My #1 Testing Pattern

Sex... to save the friendship - Jerry Seinfeld (The Mango)

This pattern is not the one that I most frequently use. Far from it. It is just the one the I like the most because it delivers a sense of reassurance and peace.

So, sometimes you lose your confidence. You no longer believe that your test suite is doing its work. Lost faith threatens to bring down the whole development cycle: you cannot refactor/change/add because a Green bar will no longer indicate "no bugs".

This problem often happens after a long refactoring session. When you're finally done you run the tests expecting a Red - you're sure you broke something. To your surprise you get Green. You now have doubts. Is the Green really Green?

Whatever the reason for this distrust is, this is a very dangerous situation.

Luckily, there is a simple solution: Start breaking your code.

Pick a random point in your code and change it. Return null from a method. Throw an exception. Comment out an assignment to a field. Whatever. Just break the code. Now run the tests again. If your tests are OK you will get a Red bar. This will raise your confidence level: the test suite just showed you that things are under control. You should now undo this bug that you just introduced and repeat this process with a different program-breaking change. The more you repeat it, the more confidence you will gain.

On the other hand, if you get Green after breaking the code, then your distrust is justified. You do have a problem. You should write more tests, starting - wait for it - right now.

4 Reasons Why Plugin Architectures are Awesome

It seems that we are surrounded by plugin-based programs. Examples? Firefox, Eclipse, JEdit, ForBugz, the list is long. This post enumerate the reasons why plugins are so great. Only the first point is obvious. The other are subtle but just as important. Personally, I like reason #2 the most.

1. Happier Users

With plugins, users are not limited to the functionality built into the base product. They can enhance the product/customize it to their needs by installing third-party plugins. This is the obvious reason. In fact, many people think this is the only reason.

2. Happier Developers(!)

Plugin-based programs are usually very easy to debug. Let me tell you how a typical debugging cycle works in such programs: You start (as always) by writing a test that captures the bug. Then you gradually remove plugins from the program until you arrive at the minimal program that still produces the bug. If you do it via a binary-search (over the set of plugins) the process will converges quite quickly.

Usually, you end up with a very short list of plugins. A few educated guesses will help you spot the bad plugin right away. Even if you're out of guesses, you're still in a very good position: the amount of code that needs to be scanned has been dramatically reduced.

Compare that to a program that is not plugin based. Such a program cannot function without any of its parts. Thus, you have no systematic way to reduce the amount of code involved with a bug.

3. Plugins are Agile

Breaking up a user request (story or even theme/epic) into tasks is usually a breeze in a plugin-based program. One should simply break the story functionality-wise and implement each piece as a separate plugin. This is much easier than breaking a story according to implementation-oriented tasks. Bottom line: less overhead due to iteration planning.

4. Easy Collaboration

A real story: A few months ago my team was asked to assist another team with their product. That product had plugin support so we simply implemented our additions as plugins. We were not afraid to break things because we had no access to the source code of the core (big plus). Also, the core was written in C++ but the plugin system is in Java (second big plus).


That's my top four reasons. As you can see only the first point is user-oriented. The other three points are all developer-oriented. Surprising, isn't it?

Why Aircraft Carriers are not Agile

In Podcast #31 of Reversim, one of the podcasters (I think it is Ori) mentions this anecdote:

I once had the chance to talk with a consultant. When I asked him what he thinks about agile he said: 'Have you ever seen an aircraft carrier that was developed using an agile methodology?'

(Disclaimer: (1) The anecdote is stated in a somewhat vague manner. I took the liberty of rephrasing it into a concrete statement. Hopefully I got it right. (2) I am not saying that either Ori or Ran are in favor of the statement. They just mentioned it)

So, the topic of this post is this simple question:

Why aircraft carriers are not built using an agile methodology?

And my answer is this:

Because the engineers can know up front that it will float

That's it. That's the answer. That's the point I am making. The remainder of this post is just the rationale.

An aircraft carrier is one of the most complicated machines ever built by man. Still, there is one perspective wherein the complexity of an aircraft carrier is a mere fracture of the complexity of almost any computer program. I am speaking about the ability to reason about the machine from its blue prints.

If you take the blue prints of an aircraft carrier you can determine whether the thing will float or sink. It is a simple matter of volume vs. weight. Of course, calculating the volume or the weight from the blue prints is not always a trivial task, but it is still doable. In fact, one can use a computer to do this. Bottom line: the blue prints give a precise answer to the most fundamental question related to any vessel: will it float?

In computer programs things are different. Even if you take the actual source code you will not be able to get a precise answer to the fundamental question of programs: will it compute the correct output? It is not a difficult question to answer. It is an impossible question to answer. All the computers in the land will not help you answer this question. That is what Church/Turing had proven in the 1930s (I know I mention these guys in almost every post - but that's how I see things: all difficulties in software construction are due to the properties proven by these two).

Note that we are speaking about the source code. The blue prints (UML diagrams, design documents, whatever) provide even less information about the program than the source code. Therefore, they cannot provide any better answer.

Under these circumstances it is not surprising that agile methodologies which promote a trial-and-error type of process are on the rise. When you don't know how a machine will behave unless you build it and run it, the only viable option is to built it, run it, see what it does and make the necessary adjustments.

Going back to the original question, let's play this little experiment: Suppose we live in a universe where there is no practical way to determine whether a thing floats or sinks. There are some vague heuristic for building vessels that float, but every now and then ships sink as soon as they leave the dock. So, in such a universe

will you volunteer to go aboard an aircraft carrier that has been carefully planned but never left the dock?

Software Design Comes to Hollywood

Don't let yourself get attached to anything you are not willing to walk out on in 30 seconds flat if you feel the heat around the corner - Heat (1995)

So Gregor Hophe is seeing design patterns in Starbucks. As for me, I'm seeing software practices (I am not using the term "Principles" because of this) in movies. The quote in the subtitle is probably my favorite. It is taken from Michael Mann's Heat where Neil McCauley (Robert De Niro) says it twice. Neil uses this line to explain how professional criminals like him should treat the relationships that ties them to the real, honest, life.

I think this goes to programmers as well. We should be willing to walk out on any piece of code the minute we understand it becomes a liability. Every line/method/class is a viable candidate to being eradicated as part of a wide scope refactoring. In other words

Don't let yourself write any code fragment you are not willing to walk out on in 30 seconds flat if you feel the technical debt payment coming around the corner

Note that this goes not only to source code. If you are doing too much up front design you are likely to get attached to diagrams/blue prints/specs. Neal Ford calls it irrational artifact attachment.

Individuals who do not follow Neil McCauley's discipline get attached to their past work. They think that some piece of code is so well written or so central to the inner workings of the program that they only allow small patches to be applied to it. They never get to see that what is really needed is a substantial refactoring. This often leads to code that is a big mess of hacks piled up on top of something they are not emotionally ready to get rid of.

There was only one principle and that was principle-22

I think the title pretty much summarizes my approach towards software design. If I put bluntly, then I'd say that

the only universal principle in software is the one saying that there are no universal principles in software.

Hmmm.

Maybe this was a bit exaggerated. So let me rephrase:

there are very few universal principles in software. The vast majority of principles are not universal.

A careful examination of of "truths" about software would reveal that many of them are not more than a rule of thumb whose applicability is limited. Let's take the Law of Demeter: a principle that is based on a clear distinction between good and bad. In theory, this well defined order seems like an excellent vehicle for taming the usually chaotic structure of software. In practice, this principle often leads to questionable consequences. It is rear to see it widely used other than in toy examples.

[Following Yardena's comment, the OCP example was rewritten as follows]

Another example is the Open-Closed Principle: A class should be open for extension (i.e: subclassing) but closed for modification (i.e.: manipulation of the source code). Here's how Robert C. Martin summarizes this principle:

... you should design modules that never change. When requirements change, you extend the behavior of such modules by adding new code, not by changing old code that already works

Got it? good modules never change. The principle suggests that one can foresee the future to such an extent that one can develop a class up to a point where its source code will never need to change. Ever. I bet that at least 50% of all programmers do not buy this premise. Refactoring, which has gained immense popularity as a highly effective development technique, is based on a completely opposite premise.

How about the Interface Segregation principle? This allegedly universal principle has zero relevance in dynamically typed languages which have no notion of interfaces.

These are just three examples off the top of my head. Similar loopholes can be found in almost every software principle.

The funny thing is that we already know that it is practically impossible to reason about the important aspects of software. Turing and Church had already proven it. Yet, we (hopelessly) keep on trying to formulate "truths" about something that we can hardly reason about.

I think it is much more about psychology than anything else. We want to conquer new frontiers. It is in our genes. Our attempts to formulate software principles are all about taming the wild beast called "software" and making it pleasant and manageable. Sorry, it will not work. There are all sort of notions that are beyond our scope of reasoning. Software is one of them. We'd better accept this fact.

The Story of Waterfront

I think the last straw for me, WRT to Clojure, was a conversation I had with Alex Buckley during OOPSLA'08. While speaking about JVM support for dynamic languages Alex mentioned Clojure and Rich Hickey's talk that was part of Lisp50@OOPSLA. This conversation gave me the motivation for delving into Clojure.

In a nutshell Clojure is a Lisp-dialect that runs on the JVM. Being Lispish it offers a flexible and concise notation. Being a JVM language it allows complete and straightforward interoperability with Java. In other words: In order to write serious Clojure programs you only need to learn language constructs. You don't need to get acquainted with new libraries.

Clojure comes with a REPL which lets you quickly evaluate Clojure expressions. In order to utilize the benefits of a REPL your programming work-flow should be different. Instead of the write-compile-run (or TDD's test-write-compile) rhythm, you need to switch to a more constructive mode: you write an isolated small expression, you run it, and then you add it into the program. Here's how Steve Yegge describes it:

You're writing a function, you're building it on the fly. And when it works [for that mock data], you're like, "Oh yeah, it works!" You don't run it through a compiler. You copy it and paste it into your unit test suite. That's one unit test, right? And you copy it into your code, ok, this is your function.

So you're actually proving to yourself that the thing works by construction. Proooof by construction.

After adopting this work-flow I noticed that I constantly copy code from my text editor into the REPL. It didn't take long for this sequence to start annoying me. I wanted a tighter integration. I wanted an Editor which is also a REPL. I envisioned an editor where I can right click on a piece of text, choose "Eval" and immediately see the results in an "Output" window.

This vision was (and still is) the seed of Waterfront.

I used Clojure itself as the implementation language (what better way is there to learn a language than to use it something serious?). In fact, once I had the baic functionality working, I used Waterfront for developing Waterfront. These two choices helped me in detecting all sorts of features that are needed for the Clojure developers. Here are a few examples:

First, I needed parenthesis matching. It's a must if you have a Lisp-like syntax. I wanted to be able to see the doc of Clojure functions. When manipulating Java objects I needed to see the list of methods/fields/constructors of classes. I also noticed that when I write proxies (using Clojure's proxy function), I constantly switch back and fourth between Waterfront and the Javadoc of the proxied class, copying the names of the methods I want to override.

I tried to address these needs as best as I can. Currently, Waterfront supports the following features:

• CTRL+E: Eval current selection, or the whole file if the selection is empty


• Edit -> Eval as you type: When turned on (default) periodically evaluates your code. Boosts productivity as many errors are detected on the spot.


• Syntax and Evaluation errors are displayed on: (1) The Problems window; (2) The line-number panel, as red markers.


• Source -> Generate -> Proxy: Generates a proxy for the given list of super-types, with stub implementations for all abstract methods.


• F1: Doc or Reflection
  Shows the doc (as per Clojure's (doc x) function) of the identifier under the caret.
  Shows the synopsis of a Java class if there is a class symbol under the caret (e.g.: java.awt.Color).


• CTRL+Space: Token-based auto completion.


• Full parenthesis matching.


• An extensible plugin architecture.


• Eval-ed code can inspect/mutate Waterfront by accessing the *app* variable. For instance, if you eval this expression, ((*app* :change) :font-name "Arial"), you will choose "Arial" as the UI font.


• Eval-ed code can inspect the currently edited Clojure program. For instance, if you eval this expression, ((*app* :visit) #(when (= (str (first %1)) "cons") (println %1))), the output window will show all calls, made by your code, to the cons function.


• Other goodies such as undo/redo, toggle comment, recently opened files, indent/unindent, Tab is *always* two spaces, ...

In terms of implementation, Waterfront is based on the context pattern. It allows event handlers to communicate in a functional (side-effect free) manner. On top of this there is a plugin-loader mechanism which loads the plugins that are specified in Waterfront's configuration file. This means that functionality can be easily added or removed (extremely useful when debugging!).

The overall response was warm. I plan to move Waterfront into clojure-contrib in order to provide Clojure newbies with a lightweight IDE to ease the transition into the language.

I am almost done telling the story of Waterfront. The only thing missing is the answer to this question: why Waterfront?

Here's the answer: Back in the 90's there was this TV show called: Homicide: Life on the Streets, describing a homicide unit in Baltimore's police department. One of the story lines - that runs throughout all seasons - is about a Baltimore bar called "The Waterfront" which is owned and operated by three Baltimore cops: Det. Meldrick Lewis, Det. John Munch, and Det. Tim Bayliss. Needless to say, I like this show very much.

My Whereabouts this Winter

A lot of things have happened in the last couple of months. First, I finished school. I practically finished my Ph.D. though I still need to work on a few things in the dissertation. Consequently, I started working for a big, multi-national, corporate. I got acquainted with MDD tools, Software product lines and multi-core issues. In between I discovered Clojure and initiated Waterfront, an open source, Clojure-based, lightweight IDE, for Clojure.

I announced Waterfront on the Clojure group this Tuesday, and it was received with a very warm response. A by-product of the Waterfront effort was the formulation of the Application Context Pattern. A pattern that solves the contradiction between GUI programming and functional programming.

All this stuff resulted in a lot of subjects that I'd like to blog about. The problem, like always, is time. My plans for future posts is to start by writing a bit about Waterfront, and then to fully describe the application context pattern. Hope to find the available time slots for that.

Is Your Program Greener or Longer?

Q: Is A cucumber greener or longer?
A: Greener. It is also green sideways.

Most programs can be viewed from several different perspectives. If we take a web-application build around the MVC architecture, we can view the program as an assembly of three subsystems: the model, the view, and the controller. At the same time, we can also see the program as an assembly of pieces of code each one dedicated to a web page.

This dichotomy is not uncommon. In each program one can see several orthogonal decompositions. Although the the decomposition into packages/classes is very dominant, other conceptual decompositions are just as viable.

For the rest of this post I'll concentrate on two decompositions: the architectural decomposition sees the code as a collection of subsystem, such as: data access, user interface, security, persistence. The functional decomposition sees the program as a collection of features (user-visible functionality) such as: login, register, delete, submit, search, ...

We can make the reasonable assumption that in most programs the number of features is far greater than the number of subsystems. Also, to keep things simple I will assume that each feature "touches" each subsystem.

(Although it very difficult to provide an algorithm that identifies all features or all subsystems in a program most programmers can usually recognize, by means of intuition, both the architectural and the functional decompositions of their program).

The big question is this: What's the easiest way to develop a program. Is it feature by feature, or subsystem by subsystem?

This question may seem to be making as much sense as asking if a cucumber is greener or longer. If we have 40 features, 4 subsystems, and every feature contributes 100 lines of code to each subsystem then it does not matter whether our work is split subsystem-wise or feature-wise. Either way total effort will be 40 times 4 times 100 = 16,000.

I think that the effort model employed by the last paragraph is flawed. I believe that the effort required for completing a programming task is more than the effort for completing two smaller tasks that amount to the bigger task. In computer science lingo, this means that:

the effort function is super linear

If f is a super-linear function then f(a+b) > f(a) + f(b). An obvious example is sorting. Sorting an array of 1,000 elements is more work than sorting 10 arrays of 100 elements. I believe that the development effort function is super linear. In fact, it seems to be closer to a quadratic function than to a linear function.

This means that It will take less effort to complete a program if your programming tasks follow features (many small tasks) than subsystems (few large tasks). The project will seem harder if developed according to architectural decomposition.

The reasoning suggested here is not a mathematical proof. It is merely a phenomena I witnessed. Nonetheless, the effort is super-linear thesis provides an explanation for issues such as: The success of TDD; the small-tasks practice of agile methodologies; the effectiveness of REPL tools; and more. So, although we cannot prove this thesis it may very well be one of the axioms of programming.

The Democratic Approach to Programming

Greg Stein published this post (see also here) which links to a 1996 question by one, Larry Page. In the question Mr. Page complains that Java's HTTP client API does not allow him to set the User-Agent field of requests. Although this may be a hoax (conspiracy fans may even say that it was planted by someone from Google who does not like Java), it certainly raises a few thoughts.

It may be seen as a yet another argument in favor of dynamic typing/monkey patching. I'll put this issue aside. I think that the thin interface perspective is just as interesting.

With thin interfaces an object provides only a small set of services. This leads to layering of abstractions: A new abstraction provides a new set of services on top of existing ones. We end up with a hierarchy of abstractions, each one tailored for a different type of usage.

With fat interfaces the API tends to evolve into a single uber-abstraction. As this debate shows this one-size-fits-all approach may be either good or bad, depending on your point view. Larry Page witnessed the negative side. According to his question, the underlying abstraction failed to provide the service he needed. It is just like you are given this big machine to help you do something, but it does not do exactly what you need. You can either disassemble the machine and build it all over again which is a lot of work (after all, it is a big machine) or you can build a new machine from scratch. Either way, it's a lot of work. If you already rely on this machine for other purposes then even replacing it will be expensive. As Jake Blues puts it, you find that "you're really up Shit Creek".

It seems that Shit-Creeks like this are quite rare with thin interfaces. Instead of one abstraction with many services you have many abstraction each providing a small set of services. Pluralism of abstractions promotes modularity because you can assemble them according to your needs. Tweaking is easier since you're not as bounded by contracts because each contract is relatively small.

As a direct result, thin interfaces put you, the client, in charge. This is a more democratic approach to programming. Let's spread the power around.