4 Reasons Why Plugin Architectures are Awesome

5 comments
It seems that we are surrounded by plugin-based programs. Examples? Firefox, Eclipse, JEdit, ForBugz, the list is long. This post enumerate the reasons why plugins are so great. Only the first point is obvious. The other are subtle but just as important. Personally, I like reason #2 the most.

1. Happier Users

With plugins, users are not limited to the functionality built into the base product. They can enhance the product/customize it to their needs by installing third-party plugins. This is the obvious reason. In fact, many people think this is the only reason.

2. Happier Developers(!)

Plugin-based programs are usually very easy to debug. Let me tell you how a typical debugging cycle works in such programs: You start (as always) by writing a test that captures the bug. Then you gradually remove plugins from the program until you arrive at the minimal program that still produces the bug. If you do it via a binary-search (over the set of plugins) the process will converges quite quickly.

Usually, you end up with a very short list of plugins. A few educated guesses will help you spot the bad plugin right away. Even if you're out of guesses, you're still in a very good position: the amount of code that needs to be scanned has been dramatically reduced.

Compare that to a program that is not plugin based. Such a program cannot function without any of its parts. Thus, you have no systematic way to reduce the amount of code involved with a bug.

3. Plugins are Agile

Breaking up a user request (story or even theme/epic) into tasks is usually a breeze in a plugin-based program. One should simply break the story functionality-wise and implement each piece as a separate plugin. This is much easier than breaking a story according to implementation-oriented tasks. Bottom line: less overhead due to iteration planning.

4. Easy Collaboration

A real story: A few months ago my team was asked to assist another team with their product. That product had plugin support so we simply implemented our additions as plugins. We were not afraid to break things because we had no access to the source code of the core (big plus). Also, the core was written in C++ but the plugin system is in Java (second big plus).


That's my top four reasons. As you can see only the first point is user-oriented. The other three points are all developer-oriented. Surprising, isn't it?

Why Aircraft Carriers are not Agile

14 comments
In Podcast #31 of Reversim, one of the podcasters (I think it is Ori) mentions this anecdote:
I once had the chance to talk with a consultant. When I asked him what he thinks about agile he said: 'Have you ever seen an aircraft carrier that was developed using an agile methodology?'

(Disclaimer: (1) The anecdote is stated in a somewhat vague manner. I took the liberty of rephrasing it into a concrete statement. Hopefully I got it right. (2) I am not saying that either Ori or Ran are in favor of the statement. They just mentioned it)

So, the topic of this post is this simple question:
Why aircraft carriers are not built using an agile methodology?

And my answer is this:
Because the engineers can know up front that it will float


That's it. That's the answer. That's the point I am making. The remainder of this post is just the rationale.

An aircraft carrier is one of the most complicated machines ever built by man. Still, there is one perspective wherein the complexity of an aircraft carrier is a mere fracture of the complexity of almost any computer program. I am speaking about the ability to reason about the machine from its blue prints.

If you take the blue prints of an aircraft carrier you can determine whether the thing will float or sink. It is a simple matter of volume vs. weight. Of course, calculating the volume or the weight from the blue prints is not always a trivial task, but it is still doable. In fact, one can use a computer to do this. Bottom line: the blue prints give a precise answer to the most fundamental question related to any vessel: will it float?

In computer programs things are different. Even if you take the actual source code you will not be able to get a precise answer to the fundamental question of programs: will it compute the correct output? It is not a difficult question to answer. It is an impossible question to answer. All the computers in the land will not help you answer this question. That is what Church/Turing had proven in the 1930s (I know I mention these guys in almost every post - but that's how I see things: all difficulties in software construction are due to the properties proven by these two).

Note that we are speaking about the source code. The blue prints (UML diagrams, design documents, whatever) provide even less information about the program than the source code. Therefore, they cannot provide any better answer.

Under these circumstances it is not surprising that agile methodologies which promote a trial-and-error type of process are on the rise. When you don't know how a machine will behave unless you build it and run it, the only viable option is to built it, run it, see what it does and make the necessary adjustments.

Going back to the original question, let's play this little experiment: Suppose we live in a universe where there is no practical way to determine whether a thing floats or sinks. There are some vague heuristic for building vessels that float, but every now and then ships sink as soon as they leave the dock. So, in such a universe

will you volunteer to go aboard an aircraft carrier that has been carefully planned but never left the dock?


Software Design Comes to Hollywood

2 comments
Don't let yourself get attached to anything you are not willing to walk out on in 30 seconds flat if you feel the heat around the corner - Heat (1995)


So Gregor Hophe is seeing design patterns in Starbucks. As for me, I'm seeing software practices (I am not using the term "Principles" because of this) in movies. The quote in the subtitle is probably my favorite. It is taken from Michael Mann's Heat where Neil McCauley (Robert De Niro) says it twice. Neil uses this line to explain how professional criminals like him should treat the relationships that ties them to the real, honest, life.

I think this goes to programmers as well. We should be willing to walk out on any piece of code the minute we understand it becomes a liability. Every line/method/class is a viable candidate to being eradicated as part of a wide scope refactoring. In other words

Don't let yourself write any code fragment you are not willing to walk out on in 30 seconds flat if you feel the technical debt payment coming around the corner

Note that this goes not only to source code. If you are doing too much up front design you are likely to get attached to diagrams/blue prints/specs. Neal Ford calls it irrational artifact attachment.

Individuals who do not follow Neil McCauley's discipline get attached to their past work. They think that some piece of code is so well written or so central to the inner workings of the program that they only allow small patches to be applied to it. They never get to see that what is really needed is a substantial refactoring. This often leads to code that is a big mess of hacks piled up on top of something they are not emotionally ready to get rid of.

There was only one principle and that was principle-22

3 comments
I think the title pretty much summarizes my approach towards software design. If I put bluntly, then I'd say that

the only universal principle in software is the one saying that there are no universal principles in software.

Hmmm.

Maybe this was a bit exaggerated. So let me rephrase:
there are very few universal principles in software. The vast majority of principles are not universal.


A careful examination of of "truths" about software would reveal that many of them are not more than a rule of thumb whose applicability is limited. Let's take the Law of Demeter: a principle that is based on a clear distinction between good and bad. In theory, this well defined order seems like an excellent vehicle for taming the usually chaotic structure of software. In practice, this principle often leads to questionable consequences. It is rear to see it widely used other than in toy examples.

[Following Yardena's comment, the OCP example was rewritten as follows]

Another example is the Open-Closed Principle: A class should be open for extension (i.e: subclassing) but closed for modification (i.e.: manipulation of the source code). Here's how Robert C. Martin summarizes this principle:
... you should design modules that never change. When requirements change, you extend the behavior of such modules by adding new code, not by changing old code that already works
Got it? good modules never change. The principle suggests that one can foresee the future to such an extent that one can develop a class up to a point where its source code will never need to change. Ever. I bet that at least 50% of all programmers do not buy this premise. Refactoring, which has gained immense popularity as a highly effective development technique, is based on a completely opposite premise.

How about the Interface Segregation principle? This allegedly universal principle has zero relevance in dynamically typed languages which have no notion of interfaces.

These are just three examples off the top of my head. Similar loopholes can be found in almost every software principle.

The funny thing is that we already know that it is practically impossible to reason about the important aspects of software. Turing and Church had already proven it. Yet, we (hopelessly) keep on trying to formulate "truths" about something that we can hardly reason about.

I think it is much more about psychology than anything else. We want to conquer new frontiers. It is in our genes. Our attempts to formulate software principles are all about taming the wild beast called "software" and making it pleasant and manageable. Sorry, it will not work. There are all sort of notions that are beyond our scope of reasoning. Software is one of them. We'd better accept this fact.

The Story of Waterfront

6 comments
I think the last straw for me, WRT to Clojure, was a conversation I had with Alex Buckley during OOPSLA'08. While speaking about JVM support for dynamic languages Alex mentioned Clojure and Rich Hickey's talk that was part of Lisp50@OOPSLA. This conversation gave me the motivation for delving into Clojure.

In a nutshell Clojure is a Lisp-dialect that runs on the JVM. Being Lispish it offers a flexible and concise notation. Being a JVM language it allows complete and straightforward interoperability with Java. In other words: In order to write serious Clojure programs you only need to learn language constructs. You don't need to get acquainted with new libraries.

Clojure comes with a REPL which lets you quickly evaluate Clojure expressions. In order to utilize the benefits of a REPL your programming work-flow should be different. Instead of the write-compile-run (or TDD's test-write-compile) rhythm, you need to switch to a more constructive mode: you write an isolated small expression, you run it, and then you add it into the program. Here's how Steve Yegge describes it:

You're writing a function, you're building it on the fly. And when it works [for that mock data], you're like, "Oh yeah, it works!" You don't run it through a compiler. You copy it and paste it into your unit test suite. That's one unit test, right? And you copy it into your code, ok, this is your function.

So you're actually proving to yourself that the thing works by construction. Proooof by construction.


After adopting this work-flow I noticed that I constantly copy code from my text editor into the REPL. It didn't take long for this sequence to start annoying me. I wanted a tighter integration. I wanted an Editor which is also a REPL. I envisioned an editor where I can right click on a piece of text, choose "Eval" and immediately see the results in an "Output" window.

This vision was (and still is) the seed of Waterfront.

I used Clojure itself as the implementation language (what better way is there to learn a language than to use it something serious?). In fact, once I had the baic functionality working, I used Waterfront for developing Waterfront. These two choices helped me in detecting all sorts of features that are needed for the Clojure developers. Here are a few examples:

First, I needed parenthesis matching. It's a must if you have a Lisp-like syntax. I wanted to be able to see the doc of Clojure functions. When manipulating Java objects I needed to see the list of methods/fields/constructors of classes. I also noticed that when I write proxies (using Clojure's proxy function), I constantly switch back and fourth between Waterfront and the Javadoc of the proxied class, copying the names of the methods I want to override.

I tried to address these needs as best as I can. Currently, Waterfront supports the following features:
  • CTRL+E: Eval current selection, or the whole file if the selection is empty

  • Edit -> Eval as you type: When turned on (default) periodically evaluates your code. Boosts productivity as many errors are detected on the spot.

  • Syntax and Evaluation errors are displayed on: (1) The Problems window; (2) The line-number panel, as red markers.

  • Source -> Generate -> Proxy: Generates a proxy for the given list of super-types, with stub implementations for all abstract methods.

  • F1: Doc or Reflection
    Shows the doc (as per Clojure's (doc x) function) of the identifier under the caret.
    Shows the synopsis of a Java class if there is a class symbol under the caret (e.g.: java.awt.Color).

  • CTRL+Space: Token-based auto completion.

  • Full parenthesis matching.

  • An extensible plugin architecture.

  • Eval-ed code can inspect/mutate Waterfront by accessing the *app* variable. For instance, if you eval this expression, ((*app* :change) :font-name "Arial"), you will choose "Arial" as the UI font.

  • Eval-ed code can inspect the currently edited Clojure program. For instance, if you eval this expression, ((*app* :visit) #(when (= (str (first %1)) "cons") (println %1))), the output window will show all calls, made by your code, to the cons function.

  • Other goodies such as undo/redo, toggle comment, recently opened files, indent/unindent, Tab is *always* two spaces, ...



In terms of implementation, Waterfront is based on the context pattern. It allows event handlers to communicate in a functional (side-effect free) manner. On top of this there is a plugin-loader mechanism which loads the plugins that are specified in Waterfront's configuration file. This means that functionality can be easily added or removed (extremely useful when debugging!).

The overall response was warm. I plan to move Waterfront into clojure-contrib in order to provide Clojure newbies with a lightweight IDE to ease the transition into the language.

I am almost done telling the story of Waterfront. The only thing missing is the answer to this question: why Waterfront?

Here's the answer: Back in the 90's there was this TV show called: Homicide: Life on the Streets, describing a homicide unit in Baltimore's police department. One of the story lines - that runs throughout all seasons - is about a Baltimore bar called "The Waterfront" which is owned and operated by three Baltimore cops: Det. Meldrick Lewis, Det. John Munch, and Det. Tim Bayliss. Needless to say, I like this show very much.

My Whereabouts this Winter

0 comments
A lot of things have happened in the last couple of months. First, I finished school. I practically finished my Ph.D. though I still need to work on a few things in the dissertation. Consequently, I started working for a big, multi-national, corporate. I got acquainted with MDD tools, Software product lines and multi-core issues. In between I discovered Clojure and initiated Waterfront, an open source, Clojure-based, lightweight IDE, for Clojure.

I announced Waterfront on the Clojure group this Tuesday, and it was received with a very warm response. A by-product of the Waterfront effort was the formulation of the Application Context Pattern. A pattern that solves the contradiction between GUI programming and functional programming.

All this stuff resulted in a lot of subjects that I'd like to blog about. The problem, like always, is time. My plans for future posts is to start by writing a bit about Waterfront, and then to fully describe the application context pattern. Hope to find the available time slots for that.

Is Your Program Greener or Longer?

4 comments
Q: Is A cucumber greener or longer?
A: Greener. It is also green sideways.

Most programs can be viewed from several different perspectives. If we take a web-application build around the MVC architecture, we can view the program as an assembly of three subsystems: the model, the view, and the controller. At the same time, we can also see the program as an assembly of pieces of code each one dedicated to a web page.

This dichotomy is not uncommon. In each program one can see several orthogonal decompositions. Although the the decomposition into packages/classes is very dominant, other conceptual decompositions are just as viable.

For the rest of this post I'll concentrate on two decompositions: the architectural decomposition sees the code as a collection of subsystem, such as: data access, user interface, security, persistence. The functional decomposition sees the program as a collection of features (user-visible functionality) such as: login, register, delete, submit, search, ...

We can make the reasonable assumption that in most programs the number of features is far greater than the number of subsystems. Also, to keep things simple I will assume that each feature "touches" each subsystem.

(Although it very difficult to provide an algorithm that identifies all features or all subsystems in a program most programmers can usually recognize, by means of intuition, both the architectural and the functional decompositions of their program).

The big question is this: What's the easiest way to develop a program. Is it feature by feature, or subsystem by subsystem?

This question may seem to be making as much sense as asking if a cucumber is greener or longer. If we have 40 features, 4 subsystems, and every feature contributes 100 lines of code to each subsystem then it does not matter whether our work is split subsystem-wise or feature-wise. Either way total effort will be 40 times 4 times 100 = 16,000.

I think that the effort model employed by the last paragraph is flawed. I believe that the effort required for completing a programming task is more than the effort for completing two smaller tasks that amount to the bigger task. In computer science lingo, this means that:

the effort function is super linear

If f is a super-linear function then f(a+b) > f(a) + f(b). An obvious example is sorting. Sorting an array of 1,000 elements is more work than sorting 10 arrays of 100 elements. I believe that the development effort function is super linear. In fact, it seems to be closer to a quadratic function than to a linear function.

This means that It will take less effort to complete a program if your programming tasks follow features (many small tasks) than subsystems (few large tasks). The project will seem harder if developed according to architectural decomposition.

The reasoning suggested here is not a mathematical proof. It is merely a phenomena I witnessed. Nonetheless, the effort is super-linear thesis provides an explanation for issues such as: The success of TDD; the small-tasks practice of agile methodologies; the effectiveness of REPL tools; and more. So, although we cannot prove this thesis it may very well be one of the axioms of programming.

The Democratic Approach to Programming

2 comments
Greg Stein published this post (see also here) which links to a 1996 question by one, Larry Page. In the question Mr. Page complains that Java's HTTP client API does not allow him to set the User-Agent field of requests. Although this may be a hoax (conspiracy fans may even say that it was planted by someone from Google who does not like Java), it certainly raises a few thoughts.

It may be seen as a yet another argument in favor of dynamic typing/monkey patching. I'll put this issue aside. I think that the thin interface perspective is just as interesting.

With thin interfaces an object provides only a small set of services. This leads to layering of abstractions: A new abstraction provides a new set of services on top of existing ones. We end up with a hierarchy of abstractions, each one tailored for a different type of usage.

With fat interfaces the API tends to evolve into a single uber-abstraction. As this debate shows this one-size-fits-all approach may be either good or bad, depending on your point view. Larry Page witnessed the negative side. According to his question, the underlying abstraction failed to provide the service he needed. It is just like you are given this big machine to help you do something, but it does not do exactly what you need. You can either disassemble the machine and build it all over again which is a lot of work (after all, it is a big machine) or you can build a new machine from scratch. Either way, it's a lot of work. If you already rely on this machine for other purposes then even replacing it will be expensive. As Jake Blues puts it, you find that "you're really up Shit Creek".

It seems that Shit-Creeks like this are quite rare with thin interfaces. Instead of one abstraction with many services you have many abstraction each providing a small set of services. Pluralism of abstractions promotes modularity because you can assemble them according to your needs. Tweaking is easier since you're not as bounded by contracts because each contract is relatively small.

As a direct result, thin interfaces put you, the client, in charge. This is a more democratic approach to programming. Let's spread the power around.

Source Code Quality Estimation is Hard

1 comments
As you might have guessed I don't believe in metrics that estimate software quality by measuring source code. In the most general case they are bounded by this paradox. Even if not taken too generally, there are still considerable difficulties related to such metrics.

In order to avoid confusion and misinterpretation let me say that (a) the items on this list reflect current knowledge. Future research may find solutions to the problems therein; (b) I am not claiming here that every metric is bad nor that all reasons hold for all metrics.

1. Source code quality metrics will never be able to detect infinite loops.

In other words, you can get a good score for a program that does stupid things at run time.

2. Lack of normalization

Most metrics are not normalized: they do not have a scale by which scores can be compared. With a non-normalized metric the distance between 15 and 20 may completely different than the distance between 25 and 30. In other words, if 100 is the perfect score, then it may be that getting from 99 to 100 is a million times more difficult than from 98 to 99.

3. Indirectness: metrics do not measure the "real thing"

The real thing is software quality, which is a vague notion . For the sake of the discussion let's define quality as "number of bugs". Source code metrics do not measure number of bugs directly. They measure artifacts which are only remotely related to number of bugs.

Think about a 100-meter dash. Instead of determining the winner by directly measuring the running times of the contestants, you measure the number of practices they had in the last four weeks.

4. Blind spots

Source code metrics are mostly blind to DSLs, to the universal design pattern and a bunch of other useful techniques.

5. Detecting the negative

Many metrics work by by collecting negative points. E.g., the Cyclomatic complexity metric gives you negative points for branches in your code. The problem is that Spotting "bad" things is not equivalent for spotting "good" things.

The first implication is that such a metric can only help you when you are doing something wrong. It does not tell you that your are doing something right. In a way, it leads to mediocrity.

Moreover, let's see what happens when you use a metric that detects the negative and also has blind spots (which, sadly, most of the metrics are so) . Such a metric will encourage you to make the blind spots as bigger as possible. As the blind spot grows, the amount of negative things that can be detected lessens. The metric simply beats its own purpose.

6. Focus on things that can be measured

By definition, metrics ignore things that cannot be measured. Focusing only on the measurable makes sense as much estimating the quality of a novel by calculating the number of sentences per chapter.

Longer is Sometimes Better

4 comments
The post that I swore never to write


(I never wanted to write this post. In fact, I wanted not to write it. I think it focuses on stylistic aspects which are overrated. I do think that a clear style is important, but I don't think we should be religious about it. In other words, If the method's style gets an "A" I don't see any point in trying to raise its score to "A+". Writing another unit test will give a much higher ROI).

A comment by Martin in response to one of my latest posts said that sometimes longer is better. Since people tend to prefer compactness, I think that it may be worthwhile to mention some points in favor of length. Please bare in mind that I am not saying that longer code is better. Not at all. Shorter should be the norm, but it should not be followed religiously.

Width vs. height

Programmers often attempt to reduce their method's height (number of lines) by consolidating two (or more) lines into one. If this attempt increases the width of the method (number of columns) then they actually make it more difficult to understand the code. This is because we don't read content left to right, line by line. Our eyes movement follows the F-shaped pattern:

  • Users first read in a horizontal movement, usually across the upper part of the content area

  • Next, users move down the page a bit and then read across in a second horizontal movement that typically covers a shorter area than the previous movement

  • Finally, users scan the content's left side in a vertical movement

Here's how it looks like:



(Copyright © 2006 by Jakob Nielsen. ISSN 1548-5552)

This means that we have greater chances of reading things on the left hand side of the screen. We can better understand greater parts if most of the code is located closer to the left.

One may argue that the cited research studied users looking at web-sites rather than programmers looking at source code. Thus, maybe the F-pattern is not relevant WRT programming. This is a valid argument, but it needs to be verified by research. To the best of my knowledge (and I'd be glad to stand corrected), there is no support for this argument in the literature.

Stacktraces

You get a bug report from a user. The stacktrace shows a null pointer exception on this line:

m1().m2().m3().m4()

You don't know which method call returned null. Was it m1(), m2(), or m3()?
If each call were on a line of its own you'd have much more information. Sometimes this can make a huge difference.

A single-line if

Writing the "else" part in the same line as the if() makes the code shorter but more difficult to debug:

for(int i = 0; i < getTop(); ++i)
if(get(i).equals(b)) return;

Suppose you're interested in the return statement, but since you have a single line if() You cannot place a breakpoint on the return statement. Placing a breakpoint on the if() is not practical if getTop() returns a large value (even 50 is large value in this context).

Another example:

void f(int n, int m) {
int x = g(n);
h(m, x);
if(!isValid(x)) return;

// do something useful
}

You are trying to find out why f() is not doing what it was supposed to do. So you step through f() and suddenly, surprise!, you find yourself in another method.

After some thinking you realize that you are back in f()'s caller. That's because the last thing you remember is that you were standing on "if(!isValid(x) return;".

Now, here are your thoughts: "Aha! That's exactly what's wrong. I should not exit from f() under these circumstances. Is the problem in g() that computes x's value or in isValid() that checks it? Gee, I don't know. If only I knew what x's value is. Oh boy! I have to start the debugging session all over again"

This problem will be a no-problem if you split the if() across two lines. During debugging you will have a visual indication that the if() succeeded.

Note that this discussion is pertaining not only to if()'s but also to the ternary operator, "?:".

Temporary variables explain the code

There is this refactoring step of Replace temp with query. Each application of this refactoring saves one line - the line where the variable was defined.

Martin Fowler is a very smart guy. He knows that reducing line count is not the only game in town. That's why immediately next to "replace temp with query" he placed Introduce explaining variable. He knows that a variable that is used only once is still useful: it explains the code, and it worth to have it, despite the extra line that it occupies.

BTW, this duality is quite common Fowler's Refactoring book. See, for example, Hide delegate vs. Remove middle man. It seems that Fowler is well aware that every decision has its trade offs.

Capturing return values

So many I found my self debugging a method and not knowing its return value. This happens when the code is written by someone who wants to save a line. Instead of writing

int f() {
// do something
int result = g();
return result;
}

He writes

int f() {
// do something
return g();
}

The problem with the latter form is that it hides f()'s return value. If this is the common style, then when you step out of f() you find that f()'s caller is returning f()'s return value directly to *its* caller, which in turn.....

This is so annoying. You have to step out of many methods. When you get to inspect this value, you realize that it is wrong but now you lost the original stack frame. Think what happens if your code was called from library whose source code is out of your reach.

(Although this is a special case of the previous point, it is related to events so annoying that it deserves a section of its own)

Summary

It should be noted that other than the first point (the F-shaped pattern) the points made here do not capture some cosmic truth about programming. They depend on the language, on the tools and probably on some cultural factors. You'll have to reconsider them when you switch to a new language/tool/team.

Finally, by all means, don't try to make your code longer. Try to make it shorter. But as you make it shorter, think about these points. They may help you avoid code that is too short.