Science progresses on the willingness to take widely held beliefs and test them. It is one thing to say that a medicine will “cure all that ails ye” but it is another thing altogether to prove that a particular medicine will have a particular effect on health. Proof is expensive, but science does not march forward without it.
For quite a while, our team has been working diligently to increase the use of architectural models (and architectural thinking) among our IT units. There are thousands of employees engaged in developing software in MSIT, and it is not always easy to reach a wide audience and make a compelling case. Our arguments have been based on appeals to logic (clearly, it works), metaphor (it works in other areas of engineering) and anecdotal evidence (it worked on project Foo and see the benefits they got).
But at the end of the day, we have not been using science. We never ran a true scientific study to demonstrate that the use of a particular architectural practice improved the outcome of software engineering in a specific and measurable way, compared to a project that did not use architecture.
I’d like to see us, as an industry, get to the point where we can perform a scientific experiment on the efficacy of using software architecture practices to improve software development outcomes.
We’d need a protocol for the experiment, and controlled variables, to reproduce the effects of using architecture. We’d need a specific definition of what it means to “use architecture” and the specific practices that we believe will have an impact on outcomes. We’d need a clear way to measure the outcome of the experiment so that, when repeated, anywhere in the world, the experiment could produce comparable results.
And here’s the rub: I’d like to see it be something that YOU can help with. The community of architects and developers that care about things like SOA, Architecture, and Planning… we are the people who can perform the experiment in 1,000 settings around the world. We are the ones that can prove, or disprove, the hypothesis of software architecture.
If anyone has experience with this kind of experiment, I’d love to hear from you, just to see if this idea is too difficult for the community to do. Personally, I’m not sure where to start. If you have ideas, please reach out to me.
PingBack from http://www.anith.com/?p=37455
Nick,
what you are proposing is not much different than what the NZ States Service Commission has tried to do among NZ Government with it’s version of the FEAF.
However in my opinion where it failed was that insufficient thought was given to "where the rubber meets the road". Theoretical models are fine for Architects who operate at the level of thought quite a lot but they’re no use when talking to a programmer, a tester, or a DBA.
For example you need to be able to take the Technology Reference Model and create a view that can explain to a person in an instant what the interfaces, protocols, standards, and technologies are that they have at their disposal.
The same for should be able to be done for a services and data reference model.
The business and performance models should be able to be viewed in a form that makes sense to business users, without continually having to train them in new notations or jargons.
Hello Robert,
I hope all is well with you in NZ.
I’m not sure that your experience, while instructive, is an example of a scientific experiment.
Did you create a controlled experiment where two projects were launched, one with the use of architectural practices and the other without the use of architectural practices, but with other variables controlled?
Without a direct comparison of at least two situations where the variables are well understood and controlled, we have anecdotal evidence and not a scientific outcome.
Did you create such an experiment?
Not sure how you would launch two projects with the only variable being whether architectural practices were employed (or not). One significant issue being that people are involved and they would be different between these two projects.
Second, it would be quite an effort simply to identify all the variables to control unless the proejct were exceedingly simple.
My "guess" is that a more reasonable approach would be multivariate analysis, rather than strict scientific method where only one aspect at a time is allowed to vary.
I’m commenting on the design of your experiment, rather than putting the NZ experience as a similar experiment. The point is a shared architecture hasn’t got off the ground because a lack of practically applicable artifacts.
What a surprise to see someone’s trying to make a science of what I thought is just a bureaucratic tool.
Good, but, what medicine and health have to do with it?
I’m a doctor and my experiences with IT technology are disastrous.
Also, recently I found out that sofisticated IT program used in banks doesn’t recognize a person, but only a number (of the account). It is possible to misuse that option in a very many ways.
When we come to a point where human existence would be a mouse click away, we could then turn of the light and go to sleep for ever.
Now, Nick Malik ask your self who you really are and what are you really up to, because IT is a PROJECTION OF IT’S CREATOR, his mirror, so if you are a thief you’ll create a mischievous program and EA and SOA which would favor your sick goals…With all the respects to your person, I’m adressing to IT community and yes, your effort to make IT a science is worth attention and might help but also might not.. Who can tell?
Hello Natalija,
Your message is certainly one of the most provacative responses I’ve had. I used an expression common in the USA and associated with false doctors. (We call them quacks, or snake-oil salesmen, or more recently, suppliment manufacturers).
These are people who make claims about the health effects of a medicine with no evidence that they work.
Sounds like IT? It is. All the time, we have claims that adding a practice, or removing a practice, will produce positive results. Neither side produces evidence.
Your profession, medicine, is the opposite of mine. There is a great deal of deference to science in your profession. I’d like to see a bit of that scientific approach rub off onto my profession as well.
I am sorry to hear that your experience with IT has not been good. If you had been a person who wanted to buy a house in 1400 in Europe, you would have been buying a house from a craftsman, with little or no architecture involved. Some of those houses remain today, but most do not. What we cannot see, through the distant lens, is how different must have been the experience of each person in their own house.
Today, software engineering is similar to home construction in 1400 A.D.
I believe that we need to make the leap forward that other engineering branches have made, and I believe that part of that leap is the embrace, as a profession, of science.
What am I really up to? I am trying to fix a profession that creates software that you cannot stand to use, that you find to be an affront to your own humanity. You, and the millions of professionals like you, in all walks of life, are our customers.
In business, customers are the king. If we don’t listen to customers, our business fails. IT has, all too often, blundered around trying to understand what it means to listen to customers.
Science and Engineering can help. We are a craft-hall trade. I believe we can do better. I believe that if we improve who we are, as a profession, we can build software that humanizes, that supports artistic endeavor, that encourages our own connections with one another. As long as we are blundering around, we cannot.
That is what I am up to.
I appreciate your support.
Hi Nick,
Away from your discussion with Natalija, I wanted to throw another point in the discussion. From my point of view one step towards the readiness to prove the "Architecture hypothesis" are architectural patterns when you define them as general and reusable solutions. Of course they are depending on context but with reusability you get quite near to our desired goal.
Dont know if you are aware of this but that link seems to be very useful in that context: http://eampc-wiki.systemcartography.info
Thanks again for this good discussion!
Hello L,
Thank you for the link to your site. The site looks, on the surface, to be very valuable and I will be digging in as deeply as I can.
— N
Nick,
What you are proposing is indeed very difficult and that is the reason why decisions regarding software development methodology have been largely based on advocacy (the same issues exist in software engineering as in EA). I have recently completed a PhD in empirical software engineering. Running experiments is very difficult as experiments require a high degree of control; otherwise we end up comparing apples to oranges. For example, let’s say a company wants to compare procedural programming to object-oriented programming. How do you compare which approach is better? One way is to develop the same software system twice. That right there is prohibitively expensive for most companies. Say you decide to go ahead with it; do you use the sample developers? If you do then you have to account for learning (these developers already know the pitfalls of developing the system).
When humans are involved this introduces variance you can only partially limit (an example of how we can limit his is to specify that all participants in the experiment must have a comp sci degree). Experiments in psychology give us insight as to how to deal with experiments that involve humans. Further, we need to have many data points to have results in which you can have a large degree of confidence in. Thus you should re-implement the same software many times.
I have just skimmed over some of issues that make such experiments very expensive and difficult – the reason why this is not done very often, even though it should be.
The experiment I conducted was about evaluating the costs and benefits of modeling during software maintenance. It took three years of my life and involved 20 professional software developers working, each individually working for a period of one to two weeks. Check it out here: http://www.jamesdzidek.com/publications.
For an introductory book on the topic of empirical software engineering check out: Experimentation in Software Engineering. An Introduction, by Wohlin, Runeson and Höst.
Regards,
James Dzidek.