Where does a good S/W design come from

One of the 12 principles behind the agile manifesto says

The best architectures, requirements, and designs emerge from self-organizing teams.

I have seen cases where I think some teams interpret this as good architecture emerges by itself as long as you follow good practices, e.g. XP, TDD, Scrum, etc. Caveat: I haven’t asked them…
This does not happen in reality. I have experienced colleagues who calls this a “manifested architecture” (every system has an architecture, deliberate or not), in a quite derogatory manner. The common problem with a manifested architecture is that it may fulfil all functional requirements (user stories, features, use cases, …) but does not really support any key quality attributes.
Good design, at all levels, are based on a vision of what the system should be, in terms of functionality/properties and structure. In a self-organised team this is hopefully a shared vision. You can appoint a single person to capture and disseminate this vision (commonly known as an architect), or you can use other principles to define and share the vision. The latter is my positive interpretation of the agile principle above. But you don't hope it happens without concious effort and don't do anything about it...
So how do you learn good design? The best (only) way to do this is to look at other good designs, with as much detail as available. This provides the “type of context dependent knowledge that research on learning shows to be necessary to allow people to develop from rule-based beginners to virtuoso experts”. (Bent Flyvbjerg)

Is there any use of the V-model?

I’m at the 13th International Conference on Agile Software Development in Malmö Sweden. As usual when I listen to presentations these tend to trigger new liner of thoughts instead of pondering the details of what was said. I guess this is a flaw in how I internalise knowledge from others.
Not surprising, nobody here mentioned the V-model. Too bad,since I think it is one of the most important metaphors regarding software (and system) development there is. But it is also on of the metaphors that have been used in ways which are completely inappropriate.
My “condensed understanding” of the strengths of the V-model is this:
It defines a set of activities on the left side of the V with associated artefacts aimed at exploring the problem and detailing the solution down to the actual product (the code!). The important part is that these design activities have corresponding activities of verification & validation on the right hand side of the V, in a one-to-one relationship between the design and V&V activity. The number of levels is individual, but usually about 3-5, including code is reasonable.
Where it goes wrong is when the V is laid out in time as a template for progress of a project. This may make sense if you are designing for manufacturing-heavy products, but is useless for software, the time between understanding the problem (top-left part of the V) and where you verify that your understanding is correct is just way too long to be competitive. This is definitely not news for software developers, and is a key point in agile software development (but probably phrased differently).
But it seems to me that in order to optimise the efficiency and speed there is a tendency to downplay the activities above the point of the V ( the working code). Big-upfront-design is seen as such a bad thing that it ends up in no design at all, there is a direct jump between formulating the problem (e.g. user stories) and start coding.
I believe that in an agile project all activities in a V must be executed in each sprint in order to claim to be “truly agile”. Each sprint means a better understanding the problem, a better design to solve it, and a better implementation in the code. Of course all of these are verified and validated. That does not say that these activates are equally large, or the size of the V is the same in alls sprints, but if you leave out the conscious architecture in the sprints you don’t learn to do better in the next sprints.
You should design the activities in the V to provide the maximum added value with respect to the spent effort, which obviously depends on thousands of context-specific factors), but saying they are irrelevant is only reasonable for the simplest of systems, where you have a definition of the problem and code.
Personally I rather define the result of the activities, i.e.the artefacts, and let the teams decide themselves on how to populate them. I know blog readers may cry about excessive documentation, but what I am talking about is the necessity to externalise  information necessary to share between developers, and teams, and to preserve this information over time, something which code never can do.
Long rant, and I’m not even close the exhausting my thoughts on the subject….

Concrete vs. abstract and detail vs. overview...

There often seems to be confusion about two concepts relevant to representing design information, including models.
First there is the scale of abstract versus concrete. Second there is the concept of overview versus details.

If one looks at how a design progresses over time, regardless if one follows a waterfall or agile process, one moves both from abstract to concrete and from overview to details, but they are not the same thing. And I think that agile and waterfall puts different emphasis on on to make these transisions, and where to start and end (topic for another blog post!)


Going from overview to details is simple, the more you know about the problem and the solution the more information is added. think of going from a sketch where you only elaborated on the key classes to understand the system to a class diagam with all classes that are implemented. Using a map as analogy; you increase the scale to see more details, the coastline gets more details, or you see all roads and not only the big ones.

Going from abstract to concrete is also about adding details, and I think that is where the confusion begins. But the details that are added are not of the same type as in the more abstract representation. Here it could be adding all attributes of classes or ports to components. You don't get this information by adding more classes or structure components. In the map analogy it would mean adding indvidual houses on a map which previously only had roads, lakes and forests. It is a new type of information which you cannot deduct (transform) from the more abstract representation.

Keeping it in one’s head

A friend of mine recommended this text on why programmers work at night. Entertaining but still true.

But the blog post referenced  another interesting text written by Paul Graham about the necessity of holding a program in one’s head to be an outstanding programmer. I couldn’t agree more with that he says. I also agree with the corollaries that it is detrimental to treat developers as interchangeable parts in an organisation.

But what got me thinking was the fact that an architect must keep an entire system in the head. Where the programmer has the code as something tangible to base this mental model upon the architect has at best an abstract model and at worst only her own internal representation.
So for an architecture to be successful it both seems to be necessary that there is somebody who has the ability to internalise and reason about an entire system, and that the mental representation is possible to grasp in it’s entirety with (at least) one single mind. There are probably a lot of systems out there that fulfils neither….

More on-line lectures on software architecture

I was discussing with one of my colleagues at the department if there were any introductory or summary presentations on-line about software architecture. I have already listed some on-line lectures about software archtiecture in the blog, but a quick search revelad some interesting ones:

• An Introduction to Software Architecture and the Risk-Centric Model by George Fairbanks. George is a an excellent lecturer and has written one of the best books available on software architecture. If you have the oportunity to discuss with him, take it. Too bad the sound is so bad in this video clip.
• Conscious Software Development: Architecture with Jeff Mckenna. Why not look through all parts in the series while you are at it?
• Software Architecture Document by Philippe Back. I don't know who this guy is, but he manages to convey a lot of essentials in 12 minutes
• SEI has a number of webinars related to software architecture. Unfortunately I can't get them to play on my computer, maybe you can?
• Out and About: Software Architecture with Simon Brown. A short clip with a view on what a developer needs to do to be an architect.
• Considering a Software Architect Career by Marc Ferrentino. A very short clip on what skills you need if becoming an architect.

Set-based architecture

I listened to a presentation from Durward K. Sobek II about set-based concurrent engineering, which is a development paradigm(?) from lean development (more precisely from Toyota Product Development System).

The original principle, as I understand it, is that a designer should work with a set of design proposals towards manufacturing and in the dialogue between what is desirable (form engineering) to what is possible (from manufacturing) iteratively narrow it down to a single design. He concluded the presentation with how one would start with this in the small and the advice was to present two designs instead of a single one the next time

It made me thinking about how an architecture is “presented” to the stakeholders, especially the developers. would it be possible to present two architecture proposals to developers and let them identify the constraints from their perspective in narrowing it down to a single, agreed, architecture.

The angry programmer?

My wife thinks I should start another blog named "Ulriks programming nightmares". Or maybe just add another topic of this blog where I mention examples of programmed systems that aren't good. Or to be more precise, where the programmer was too lazy to do a good job, which really gets me going. Unfortunately I get upset when I see it among the students which is a bit harsh, they are here to learn after all.
The examples I have seen so far is everything from the logic of the elevators at the university to how the booking system of SJ places people at a train. Seriously, if there are 3 reserved seats in a wagon, why must two of them be next to each other?
Enough ranting, but is this a good idea to expand the topic of the blog with? Coming to think of it, there must already be sites like that out there. If not there should be, it is always more educational to learn from bad examples than good.

Documentation

When it comes to documentation I am firm believer in the quote of Antoine de Saint-Exupery:

“A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.”

As I see it there are a few main purposes of producing documentation:

To convey understanding - This shows my quite liberal view of documentation since I consider drawings on a white board or a napkin to be documentation. But I also consider this to be one of the most important purposes of documenting things. From a designer’s viewpoint these “documents” are vital in going from an inner vision to an operational image that can be shared by others. Common examples of documentation conveying understanding in large organisations are presentations given at various meetings. These documents can be saved for posterity, but the ability to convey understanding is much smaller for those who weren’t there. On the other hand are written documents one of the most efficient means through history to build on knowledge of others which you never had the opportunity to meet.

To formalise agreements – If there is a business agreement between two organisations it is almost inevitable not to have a more formal document detailing the technical content of the agreement, the specification. many organisations, including my own, also use documents to formalise the agreement between internal teams. Your mileage may vary how efficient this is in various organisation.

To preserve information – The human memory is not infallible. Documentation is a great support to minimise the decay that is inevitable when only relying on the human mind. Software is also about maintenance and any body who has been given the responsibility to update legacy code that is undocumented know how difficult it is (this scenario includes the purpose of understanding as well)

I don’t buy “the code is the documentation” at all. The code by itself does not fulfil any of the purposes above.

I have recently worked in a project which solely relied on Trac to capture all information relevant to the project (except finances). I am positively surprised how much you can achieve with so little formal documentation in combination with having everything in the same place available for everybody in the project. I will try to do a more formal evaluation according to the purposes above when the project is near it’s end.

When to decide?

I have recently experienced several occasions where people want to decide things as early as possible in a project. This seems to be based on previous experiences that this was difficult in the last project and we want to eliminate things to worry about late in the project since there will be new things to worry about then anyway. So why not solve the major problems you are aware about anyway?

I think this kind of thinking is detrimental for the success of the project, especially if the decision influences the architecture (Grady Booch said something like "the architectural decisions are those costly to change").
If one looks at the reason the problems came up in the first place it is usually things like the implications (especially technical) of the decisions was not fully understood, the requirements were not suitable, or my personal nightmare that my team will not suffer any consequences whatever the outcome so why not decide this now...

If one has superficial understanding of the consequences or if the requirements are unstable it is a better strategy to postpone the decision as late as possible, and also delegate it to those who are most affected/concerned. But this means you and your organisation is comfortable with living with uncertainty. Which may be very difficult depending on the culture or the organisation. For example if all of your project revolves around a stage-gate process (very common in the automotive industry) it really does not fly well to say "let's postpone this decision for another 6 months and make progress with what we can".

It may sound I ampreaching the agility gospel, but that is not the case. I have full understanding that once you have made an architectural decision you don't want to change it. I just want to argue for the heuristic that you should not decide something because you can, postpone it to when you have make it.
And as a consequence I really dislike when you have to decide things just to feed the process, especially if the process is very slow to change.

WICSA 2011

I was at WICSA last week. I am a lousy trendspotter, but here is what I have seen as “trends” so far:

There seems to be several efforts, especially on the tool side, that focuses on capturing and navigating architectural information. Is this a sign that architecture information is getting bigger and bigger and is distributed among different sources/artefacts? I thought one of the tangible deliverables from an architect was a comprehensive documentation/model/wiki/whatever with architecture information. As a contrast there wa a tutorial the last day by George Fairbanks on writing a 1-page architecture document. I would say this was a highlight of the conference, to bad most people had gone home by then.

In general it seems architects are more aware of the need to adopt to agile development. I don’t think there is any contradiction, contrary I believe that it is necessary for agile developers to be more aware of “architectural thinking” and what benefits there is of having an explicitly defined architecture. But I do agree that often architecture is the same as Big Upfront Design. At the panel debate I understood better the historical background; many of the originators of the agile manifesto were active developers already in the late eighties and nineties when a lot of focus where on software design. people who started as developers in the last decade don’t have that background and think that the last years focus on process is all that is necessary for developing good software.

There was some discussions  on architecture-based testing (this is a good strategy to define a new research area, combine two or more buzzwords), but it was confusing. some people seemed to mean an architecture where it was easy to verify the quality attributes it was designed to achieve. others seemed to mean an architecture for a systems that was easy to test for testers. I like the latter better, and hope there will emerge more patterns for this than the general patterns of encapsulation etc.

Compared to the last WICSA in 2009 I think the acceptance rate was much higher, above 40%. I think this could be one explanation to why some papers had a rather weak scientific methodology. One other thing I did not like at all were some studies based on industrial practice where the results were too polished. if you present a case study you should include all the small (and big) problems that occur in real settings, otherwise the cases are not of more interest than textbook examples.

Reference architecture, what is it?

I am at WICSA 2011 and the words “reference architecture” come up now and then in discussions. I am not sure I like the definition in Wikipedia, I think relies too much that you already have knowledge or experience of a reference architecture.
One simile (parable?) would be to a building code, a set or rules that underlie the actual architecture and construction of a house. The architect is free to build any type of house, as long as the code is adhered to.
But I like to use cooking as an example. A recipe is like an architecture, when you actually cooks the recipe and serve it is the implementation of software. You need to do certain things that are implementation specific, like tasting to see if the amount of salt suits your taste etc, and maybe you need to double all the ingredients to fit your  dinner party.
Julia Child writes in her book Mastering the Art of French Cooking that there are 6 principal ways to make a sauce, one being an emulsion of melted butter with egg yolks, which is common the common method for béarnaise, hollandaise and choron sauces. The type of emulsion is similar to a pattern while the three different recipes are designs.
For a restaurant; a product line architecture would be the menu they have based on the ingredients they stock, for example chicken could be used in more than one dish.
So what is a reference architecture for a restaurant? It would be the “rules” that guides and controls what is possible or desirable. It could be for example driven by the business domain, e.g. it should focus on French or Hunan cuisine. Woking as a “development method” would not be relevant in the former, but certainly in the latter.
Or it could be driven what is technically possible, e.g if the kitchen has no deep fryer it is impossible to for example make french fries.
Or there could be other restrictions, such a desire to only use locally produced ingredients.
Maybe I went to far with this simile (parable)?

Architectural views in practice

My research is not really focused on identifying the “best” architectural views for in-vehicle software. But as a side effect to the studies I am presently doing I got to think about three views that could actually make a difference in how well large projects succeeds (or fails). I have thought about the need of addiitonal views, but cannot think of an immediate need of more.

Integration is always an in issue in large projects that has some form of iterative development. With the increasing inter-dependencies between various parts of the system it is almost impossible to know what should be integrated when, for example what can actually be tested. There actually is already an architectural view defined that addresses this concern, the anatomy. I would describe the anatomy as visualisation of the complete system seen from an integration perspective, for example if a feature depends on a MOST interface it is no use to test it if the interface is not implemented. The visual picture means everybody should have the same understanding of the “.
The anatomy should dictate the order of development, delivery schedules and integration order. And the progress should be measured in how much of the anatomy is implemented, not in customer features. There is a relationship between customer features and the anatomy, but it is not one-to-one. you can read more about “Integration Centric Development and Anatomies” in a PowerPoint presentation or the article “Manifesting Shared Affordances in System Development – the System Anatomy” by L. Taxén and J. Lilliesköld.

The second view is the systems view, i.e. how the complete system is decomposed in smaller parts. I prefer to see this view as equivalent to the development view in Kruchten’s 4+1 views, i.e. it is aligned with the development teams. One can discuss if the system decomposition follows the organisation, as predicted by Conway’s law. The other extreme is that the development teams follow the most “logical” decomposition, i.e. ECUs, which is the most common unit to be outsourced to suppliers in the automotive domain.

The functional content of a vehicle is often defined by a list of customer features, which can be quite long, and this in itself can be considered an architectural view that concerns for example the business project, the marketing department and the end customer.
What is important for the development project is the relationship between this feature list and the two other views, i.e. what systems/development teams are responsible to implement the features and how the features relate to the anatomy. I don't know if these two relationships/mapping between views should be considered as separate views, which would make the total number of views 3+2.