The dark cloud of the economic downturn has produced a silver lining within Microsoft IT: an increased emphasis on Agile development techniques. This does not mean that MS IT is new to using Agile. Far from it. Agile development practices have been used in various IT groups here for nearly a decade.
What is new is the desire to address the basic development and governance processes themselves to remove the “Bias towards Waterfall” that exists in many of them. That bias can create a situation where someone can choose to be Agile or choose to comply with corporate policy. That’s a devil’s choice.
For example, we have a governance point that is used throughout all levels of the IT organization called “Baseline” where all design and specification (and estimation) is done, and signed off. This is supposed to occur before software is delivered. Does that favor waterfall? You betcha. This governance point is mentioned in one of the metrics all the way up to the CIO scorecard! BDUF is built in to the DNA itself.
But if you look at the reasons that people give for requesting BDUF (Big Design Up Front), it goes like this: you can reduce risks by requiring that everyone understands and agrees with the requirements, features, scope, timelines, responsibilities, deployment model, etc…
Right. Pigs fly.
I do see the intent, and it is honorable. Reducing risk is a good thing and we want to increase the number of systems that are delivered on time and on budget with fewer defects. But I think that Agile methods have proven that there is another way to accomplish the goal of “improved delivery.” Unfortunately, they do not fit into the same neat little box.
Where waterfall requires 600 pages of documentation to be written up front, Agile requires that most of those pages are never written. It is not correct that Agile methods produce design documents “as you go.” That is nonsense. Agile methods produce 10 pages of diagrams, and nearly no accompanying text.
So is architecture a BDUF process?
Some of it is. If you follow the RM-ODP model or 4+1 views, you are going to be tempted to over-specify and over-architect. After all, the 4+1 views present a mechanism for creating a “complete” model of a software system. That’s great, if you want to create a complete model of a software system. But why would you want one? Why would it be a goal to have a complete description of a software system outside the software itself? Would a “partial” description serve the needs more effectively?
BDUF architecture is rarely a good thing.
Agile architecture, however, is often a good thing. Agile architecture is the act of producing enough architecture to meet the needs of the project, and nothing more, and producing it in a timely fashion, with a minimum of effort. Agile architecture RARELY produces a detailed class model in advance.
Agile architecture often produces a high-level component model and context model to establish the existence of components, their names, and perhaps even the division of responsibilities in the dev team itself. (think: Scrum of Scrums). Agile Architecture nearly never produces diagrams of technical things, like the structure of a message. Dev tools do that, from the code itself.
Agile architects will produce models using a dev environment tool, like Visual Studio or Sparx Enterprise Architect, not a diagramming tool like Visio that cannot easily be connected with the code.
Agile architects will use diagrams to communicate between people and to express artifacts into code where developers have real freedom to make the magic happen.
Agile architects will not only leverage patterns, but also complete reference models. They will consume a well-built reference model, inject the components that will be developed for the application, and get the team off on the right start. They will not “craft a new architecture” for every need, but will build-from-pre-existing-designs 80-90% of the time using frameworks that produce maintainable, secure, reliable, and easily deployed systems.
Failure to architect a system is a failure to deliver. On the other hand, hand-crafting every system is also a failure to deliver. The difference between the two, and the middle-way that defines success, is agile architecture.
And when you have to migrate a 20 years old information system, with OS ranging from Unixware 7.1 (1999) to Windows NT, on Compaq (sic) hardware no more sold for years, with at least 16 versions of the information systems, composed of more than 700 applications, with some parts which cannot and will never compile on 64 bit machines, distributed in 19 countries like China, Russia or Brazil, etc. for a company of 55,000 people and revenus about $20B in the retail industry, what do you do ?
This a real real life context of architecture need. Do you still think that 10 pages of diagram is enough ? I agree with the fact that 600 pages will, perhaps, not help more.
My point is that we have a good overview of what is necessary when designing a single software with few interactions (self-contained software which may drive Internet services for example). But how agile (and this is a real question because I would like to find new way to tackle this kind of BIG redesign) can scale to Information Systems redesign, where agile teams are not the only one involved, but where there are hundreds of stakeholders…
I agree with you Avinash, the problem is (at least from my point of view), the burocracy behind the software development companies. In order to get accountable ones usually think (by mistake) that more formal is more productive.
Hello Pericles,
(I’m wondering if that is a psuedonym and you just happen to be a fan of greek history. If so, you chose well.)
For large "rearchitecture" situations, why in the world would you start from scratch? You would find a reference architecture that works for the specific design influencers in the system (let’s say: distributed reliable networks with unreliable links, rich user interface, distributed control of user authorization, data security being more important than throughput, etc).
You would then create diagrams that illustrate how the system you are implementing FITS within that reference architecture. You would develop, using that model, and code factories, for a single location as a proof of concept and get direct user feedback, including the users who maintain and host the application, move the data, and wire up the interchanges.
We make the mistake of assuming that one user is more important than another, and that one user gets to decide what the features are. The economic factors that are driving the replacement of your widely distributed legacy application may not be functional. They may be based on reducing risk or reducing the cost of ownership. In that case, the customers that matter the most are the people directly impacted by changes in risk or cost of ownership. That would normally include the engineers and helpdesk technicians who look after the app.
You can use a pattern for a system that allows information to move freely, set up two systems, get data to move using the patterns, and ask the folks who maintain it if they could cope with these changes. Focus on their needs to reduce risk.
These become the features that you prioritize first. Your "end customers" may not even be aware of your project, beyond the fact that a single representative is sitting in meetings once a month. They wouldn’t know until the framework has been proven to reduce costs, but that’s when a sprint would occur that had a user interface on it… one that the end users may recognize.
User interface does NOT have to be the first thing developed.
Data manipulations are the most difficult thing to work out, but that is NOT ARCHITECTURE. That is requirements and design. Architecture is the structure. You can develop the structure without knowing the details of the algorithms (although the more knowledge, the better). Just as you can develop a basketball arena that can also serve other sports, concerts, and commercial purposes.
That said, it doesn’t help to design an arena if the customer needs an office building. Pick your pattern carefully. Understand the design influencers that drive the selection of a pattern.
This IS agile architecture. Don’t develop everything from scratch, and don’t delve into the details of component design. DO understand the responsibilities of components, the movement of information, and the prioritization of the various system quality attributes needed to select and modify the basic pattern.
There is no reason to believe that replacing that legacy system would be done using the same architecture that it was developed with. It may have been a distributed client-server app when it was developed, but it could be a RIA now, hosted in five data centers with database synchronization.
I’ve done MANY system replacements. Like you, I have to live with the decisions of my predecessors (some of those older decisions are my own). Many were right at the time, with the technology available. Now they are not. That’s life. But we can now deliver an architecture that is based on system patterns and reference implementations, or even COTS packages, much faster and much more reliably than in the past.
And we should.
— Nick
Thanks Nick for your answer (FYI, Pericles is a carefully chosen pseudonym ;-).
I do agree with you. This is what I feel, but even when you think you are right, your brain get polluted by consulting people, IT vendors, methodologies and tools vendors. But that is out of subject.
To add something to what you’ve said, my experience shows that the most critical part, and that’s obviously not new, is people.
First, you have to convince people who implemented the old system that: what’s a problem today, was a solution yesterday, and that’s part of the game, they are not guilty.
Then, you have to convince top managers that agile is not a methodology, but a framework, a tool box or a philosophy. Could also be applied to cook for a big party ! Our rational brain like methodologies and top managers like thinking that a tool is necessary a solution!
You have also to convince project managers that agile was developped by talented people for talented people. You have to focus, in your team, in learning, learning, learning with a great mix between senior developpers and others. To me, agile can work only if the dev teams are smart enough to understand the goals, the drivers, the requirements of the project and be able to understand architecture and design issues involved. Could be a tricky job. And where I work, finding some key talented people is a secret treasure search!
Finally, you have to convince yourself that, as an architect, you get the right balance between instinct, creativity and rationality, and that you do not have to follow a predefined path, but create your own, being flexible and resolving things one at a time, keeping in mind the big picture.
My 2¢
I agree, Pericles. Just as your "namesake" had to be a leader, orator, and politician in order to lead the golden age of Athens to build some of the greatest architecture in history, we sometimes have to lead in our IT departments in order to build great systems.
One more reason that architecture is necessary for Agile projects: if there will be more than a few sprints, and the project will last longer than a few months to deliver, then a good architecture protects the pace of delivery. In other words, without a good architecture, "developer debt" begins to pile up quickly and later sprints spend half of their time refactoring from earlier ones. Good architecture avoids <some> of that later-sprint refactoring.
Thx. -N-
So this opportunity to go Agile was a generational issue, and the layoffs facilitated its resolution?
Hello David,
I would not make that characterization. Layoffs had nothing to do with this shift.
We’ve been on this road for a long time.
— Nick
Nice reading. Thanks! Really refreshing.
In my opinion or what I have understood about agile is mostly interpreted wrongly. Agile is a methodology and XP, SCRUM are its implementation.
Agiles itself says it works better for team size of upto 12-14 people. Other thing Agile assumes the people who going to develope are not fresh engineers, if they are they are paired with experienced ones.
Documentation of 600 pages is not required as Buisness or clients are not sure of what they wants…. they chages happens very fast, may be everyday requirement change, can we re-write the whole big document or update it everytime–guess how costly that would be.
Agile says be adaptive in nature rather than pridictive as real world is not pridictive. Agile works well when the team who uses it understands it and follows its values-principle, you can agile the practices depending on team environments.
If you ask any true Agile team if they are happy with documentation you will hear yes… cause even if you write 600Pages no one has that much time to read it and that makes it painfull for new team members…. Diagrams of 10 pages are enough if code is written properly, good design and comments. I still feel Agile can be good for big teams or mroe than 12-14 people, properly implemented it should work on 100 people team as well.
Agile development doesnt require Agile Architecturewhat but if you follow its principles – values you will have Agile Architecture automatically.
–Akshya
Hello Akshya,
It sounds like you recited Agile practices well. I disagree that "Agile" is a methodology. It is more of a philosophy and set of principles. XP, Scrum, Crystal, etc, are methodologies.
That said, if you read those methodologies, you will percieve a distinct disdain for architecture. This is probably because architecture was used as an objection, by Waterfall enthusiasts, to Agile’s adoption.
I’m making the statement that Architecture is useful, even necessary, for Agile software development, and I’m making a case that the architectural document, delivered during an Agile project, will look quite different from an architectural document developed during a Waterfall project.
But not for free.
Architecture is not included in most Agile methodologies. I’d like to see that change.
HI Nick,
Thanks for your comments, Yes I agree it is more like a philosophy but still if I look back at other blogs many people discuss it as methodology and reason I after thinking a lot seems to be is if one thinks it as philosophy one can easily go in different direction than agile wants one to take. In real world whole development cycle seems agile as its literal meaning. But its still falls under those things where you have very thin line of understanding.
I agree sometime if projects are big and going to run for long time its nice to have a architecture but from personal experience people usually we have a architecture during agile projects although not completely visible from documentation produced during development. If you look at Martin Fowler or Kent Beck’s blogs can easily find out how and when you gets Agile Architecture in developments. Best suited to agile is an evolving architecture from different iterations.
But yes if I have to work on a project such a big research project will not go for agile approach as for research you think more towards quality and better solution rather a good business solution. So you can plan 6 month ahead and do all documentation like rational approach of development.
When I started my first agile project was discussing loads of question, comparisons with other approach and how it works. It is really nice thing for consulting projects or project where requirement changes very fast and nothings is fixed but requires a good agile team (its most important to me).
Also Nick. I am still big fan of Architecture Designs but Agile and its implementations in XP and Scrum.
I was comparing both couple of weeks back and found XP is little better in the way it Implements Engineering Practices while in SCRUM one can easily fall in trap while implementing Agile principles.
Even with agile in last couple of project we ended up with really nice design and architecture and that’s why I feel its can work but I have also seen couple of other teams who failed to achieve that and my personal guess was the attitude in team which was lacking in other. Hopefully we will be able to agile the Rational approach in XP.
On choosing architecture patterns, can you recommend any materials on how to categorize different styles of architecture?
I know the P&P team started doing it, but wasn’t sure if you are aware of any other significant efforts?
Hello "Colin Jack"
If anything, there are too many possible categorizations of architecture styles, leaving us, paradoxically, with too few well-thought-out taxonomies on which to build.
This is typical of human endeavor in general: we can call a particular building by any of the following monikers, and all could potentially be correct: house, post-modern, shared-common-area style, multi-storey structure, desert-climate structure, energy-efficient structure.
Certainly you can view software from many viewpoints as well. You can look at a particular concept of a stack, and then decide if the software "fits" within a particular part of the stack. (This is how the P&P model approaches this problem).
You can also look at things like patterns of information flow, service orientation, security orientation, coding language, hosting environment, etc, as mechanisms to differentiate different "styles" of software architecture. I’m not sure that any one taxonomy is ‘better’ than another. They are simply mechanisms to navigate from "all possible designs" to "the design I want to use."
If anything the internet has taught us, the notion of "search" is often not served by a taxonomy. In the early days of internet search, we used taxonomies (like Alta Vista) to find web sites. Now, we use full content search services (like Kumo.com and Google), and web.2.0 has given us tag clouds, RSS, and twitter.
Perhaps the best way to "find" the most appropriate architectural style, to suit a problem at hand, is through a tag cloud or a full-text search? You tell me.
I’m not sure that a taxonomy will solve this problem.