Why Business Process Management is more than BPEL

By |2007-07-31T10:41:00+00:00July 31st, 2007|Enterprise Architecture|

I recently heard a presentation from a competitor to Microsoft Biztalk talk about their business process modeling tool.  It competes with Biztalk in many ways: visual programming, rules engine, etc.  However, the salesman said something that threw me off, because the statement was so wildly naieve, that only a salesman could have said it, and only a business person would have believed it.  They said:

Our tool is all your company needs to manage your business processes.

If it weren’t so tragically sad, it would be drop dead funny.

The fact that I can write a business process in a visual language doesn’t mean that a tool allows me to manage my corporate business processes.  There’s a bit more that has to be in there.  Things like:

  • ability to catalog the enterprise’s business processes and make them available, as documentation, in a portal for users without the tool to see and leverage.   The portal cannot present a simple list of processes.  They have to be organized for search and consumption by end users.
  • ability to add attributes to business processes, like maturity, simplicity, and variation from corporate standards.  Ability to search for processes by attributed (simple data search). 
  • ability to add workflow to a process for approving and including new business processes into the corporate portfolio, which provides a linkage point for governance to exist.
  • ability to share processes across a distributed team responsible for owning them, training folks on them, and reinforcing them.
  • ability to analyze the long list of processes in an enterprise and find processes that are ‘similar.’  This supports many things from the simple (search for a process) to the sublime (find two processes that are candidates for process redundancy reduction).
  • ability to estimate or measure the cost of a process.  Of course, the requires the ability to calibrate cost for an organization, and to have that cost vary by geography.
  • ability to collect statistics and measurements about a process from end users along the way and to feed that information to data collection and reporting systems as part of the process management portal.  (Think TSP/PSP).

BPEL is a language, and a nice one at that. It provides a paradigm that allows us to bridge the gap between visual programming and declarative code.  But BPEL and BPMN are simply languages.  They are not tools to MANAGE an enterprise process.  That requires more capabilities than a language can provide.

Free Code – Getting IT out of the Applications business

By |2007-07-30T14:49:00+00:00July 30th, 2007|Enterprise Architecture|

There is one big thing we must do if we are to make IT align with business strategy, we need to get IT out of the role of interpreting the whims and desires of the business.  The good folks in IT are really bad at mind-reading.  As long as we are in the “mind-reading” business, we will never be given credit for what we do well: automation.

The answer: let the business folks write free code.  Not just any business folks.  We let Business Process Developers write free code.

What is free code?  Free code is unmaintainable code that wires together service calls in a way that is inexpensive to produce.  Free code is mashup code.  Bugs can be fixed, but we don’t really maintain it.  If we want to change free code, we write it again.  It was so inexpensive to build that it costs less to rewrite than to modify in any non-trivial way.

Free code, in order to be truly free, needs to be generated from tools that are NOT coding tools.  In other words, software development environments are too rich for free code.  Why?  Because it is too tempting to build expensive code.   We need to differentiate, then, between the rich, highly designed, object oriented code that software developers produce, and the free code that business process developers will produce.

Note: I said that free code is unmaintainable.  Code is unmaintainable because it’s complexity exceeds the ability of a developer to maintain it. Let’s dig a little deeper.  Why do we need to maintain code?  Because code is expensive to write.  Therefore, it is currently cheaper to fix it than rewrite it.  On the other hand, what if code were cheap, or free?  What if it were cheaper to write it than maintain it? 

 Then we would never maintain it.  We’d write it from scratch every time. 

Sure, we can choose to write maintainable code.  We can use practices like patterns, object oriented development, and careful design principles.  On the other hand, we can give our business project managers an environment where they can describe their needs and code is simply expressed from those needs.  If the code that comes out doesn’t meet their needs, the business process developer knows it the moment they run their code. 

What is the value of doing this?

1) Lower the cost of IT through reduced skill requirements.  The skill set of the Business Process Developer is different from that of a software developer.  Traditionally, we’ve sought folks with both skill sets to employ as software analysts.  This usually meant training someone.  What is wrong wit that?  Answer: We’ve created expensive specialists to overcome tool deficiencies.  Why not fix the tools?  Then we won’t need the specialists that cost so darn much.

2) The speed of development goes up.  If the business process developer can change the process wiring readily, then the software developer can focus on making the needed updates to the services themselves.  This removes the coupling between process and code that slows down EVERY project in IT.   

3) Projects become more agile.  Since a business process developer can develop a mashup of services quickly, they can demonstrate that mashup very readily, directly to business stakeholders.  A change can be shown to the business folks quickly as well.  If the business needs change, or their understanding grows, and they need the services to do something more than they do, then this kind of agile process encourages rapid feedback to the developers who own the services themselves. 

4) Solution quality goes up.  Since we can focus our deep design team on developing the services that the business process developers consume, we can improve the quality of those services independently.  This allows for better measurement of quality and an increased focus on the key quality measures inside each service.  Reusability is a natural outcome of high quality services.

What does this mean for our tools:

We need to seperate business process modeling from software development and produce rich tools aimed at the needs of the BPM practitioner.  Those tools need to start and end with an understanding of business capabilities, tied through to business processes, and down to events and business documents against a common information model. 

We need our tools to reduce the leaky abstractions that we currently call ‘services’ by helping developers build services that are very simple to consume by the business process developers.  We need to capture these requirements and act on them through automated mechanisms built in to both the BPM environment and the IDE.

What does this mean for our processes:

The good folks in IT need to formally and officially take control of managing the common enterprise information model and the business event ontology.  If a business wants to change the data and event models, they need to work through a published process that allows and encourages consensus. 

The good folks in IT need to formally allow business process developers to easily develop, test, and deploy their processes.  Deployment is a problem because IT folks normally just ‘fudge’ their way through deployment processes.  If we are going to let business process folks to write code that we deploy, then it needs to be very simple to deploy that code.

Free code makes sense…  We need to align IT to business, and this is one very useful mechanism to do it.  It is time to stop getting in each other’s hair.

Using Massive Multiplayer Online Concepts to Build a Shared Architecture

By |2007-07-28T13:05:00+00:00July 28th, 2007|Enterprise Architecture|

How can we leverage the power, and draw, of massive multiplayer online game experiences to build a shared vision of architecture for an enterprise? 

I’m not in to multiplayer online games.  I have a lot on my mind between being a dad and being supportive of my student-wife, so I don’t really have time to devote to something so self-focused.  On the other hand, my kids love massive multiplayer online (MMO) games and I can see the draw.  You can join and leave any time you want.  You can communicate with friends, work with friends, or work alone.  You are rewarded for the goals that the game wants you to achieve.  Add interesting animation, lively music, and very few bugs, and you get a pretty compelling environment.

What I haven’t seen yet (and perhaps it is the nature of the child-like games my kids play) is a MMO game where every person plays a role to build something instead of defeating something.  It is easy to tear something down.  Divide and Conquer.  Building something up is much harder.  It is not in human nature to oppose entropy.

But my goal, and my job, is to build.  I want to create a framework for all the architects in Microsoft IT to cooperate, contribute, and conspire to make the company more efficient, more agile, more responsive, and more reliable. 

Is it possible to draw on the lessons, and perhaps the environment, of the MMO game, to build that architectural framework?  Can it be so compelling that people will want to spend thousands of hours making it real?  How would we score the ‘points?’  What would be the scenarios?

If anyone knows of someone else trying to combine these ideas, could you drop me a link?  If you know of an MMO game that conspires and rewards the construction of a large and difficult thing that is not within the experiences of any of the players, could you drop me a link to that as well?

It’s an interesting idea.  Not sure it could work, but if it could, it would be very powerful.

How is Business Process Management related to Service Oriented Architecture?

By |2007-07-26T07:23:00+00:00July 26th, 2007|Enterprise Architecture|

I’m at a large training conference this week, doing a presentation on my ideas around Middle-out SOA and abstractions in the center.  I got a rare opportunity: a smart consultant pulled me aside and asked me this question:

How is Business Process Management related to Service Oriented Architecture?

The great part about being asked that question is that I had to work to answer it, which tells me that I hadn’t spend nearly enough time thinking about the relationship between the two.  The consultant who asked wasn’t trying to show me up, nor was he asking for a basic course in SOA.  He had his own ideas, and he wanted a sounding board.

So we stood at a white board and wrote, and drew, and compared models and described ideas.  What we discovered, first and foremost, was this: we both had the same answer.  We had picked up our words from different places, so they were different words, but once we wrote the models up on the whiteboard, the biggest difference was that I drew the SOA stack horizontally while he drew it vertically. 

Turns out, when you dig down, these two activities are twin brothers.  Their parents are the same, and they have many similarities, but the serve different needs in the enterprise.

  • Business Process Management is the organization of business capabilities (people, process, technology, and data) so that common mechanisms can be discovered, simplified, and improved.  The result: a simpler, more efficient and more rational portfolio of processes.  This reduces cost and increases business agility.
  • Service Oriented Architecture is the organization of technical capabilities (activities, events, documents, and data) so that common services can be discovered, simplified, and improved.  The result: a simpler, more efficient and more rational portfolio of services.  This reduces cost and increases business agility.

The similarities are striking because these twin brothers have the same father, data, and the same mother, the business event.  To understand how they relate is to understand their parents.

First, the father: the common data model  

As I have blogged before, we need very little at the “center” in order to make Enterprise Architecture work.  Probably the most important thing we need is a common data model, so that we all know what data we are working with.  Doesn’t have to be a big data model.  On the contrary, the common data model should be as slender as possible

The common data model should contain ONLY enough information to define the things that must be present to define a business document, to identify the unique key for that document, and to define the relationships with other business documents.  It can be developed over time, with some parts vague (just relationships between data areas) while other parts are tangible (thus allowing it to be developed iteratively, as needed by projects).

To extend this ‘family’ analogy a bit further: The brother of the data model, and the uncle of our twins, is the business document.

A business document is the collection of data items that are needed to characterize an object used by the business to support the business capabilities.  These are the things the business acts upon: products, services, programs, invoices, orders, shipments, payments, and many more.  Every ERP system vendor understands the concept of the document.  It is at the heart of what they manage.

And then the mother: the business event ontology 

And just as the father of the twins is data, the mother is the business event.

A business event is an event that defines the boundaries between business processes.  At the instant that a business event occurs, a business document changes state. 

If you truly want to create a rational heirarchy of business processes, and you want to look for both commonalities and variations, you have to know where a process starts and where it ends.  Otherwise, you could find yourself comparing a single apple to a grove of apple trees. 

To compare two processes, they must both start with the same starting event and traverse to the same ending event.  We do this in our language when we say “Let’s look at the Order to Cash process” meaning “Let’s look at the process that begins with the creation of the order (event) and ends with recognition of revenue (event).”  You need the ontology of events.  This is often left out of literature on Business Process Management, because you can diagram a process that spans “from anywhere to anywhere.”  That’s great, if you want to create a diagram.  But if you want to capture knowledge, and you want to rationalize or simplify processes from around the company, you need these events as your boundaries. 

Marriage of the two

Before you manage the processes, and before you design your services, you need these parents.  They have to have time to grow up and become mature, or their children will have no direction, so get consensus that the data entities are correct and their relationships are correct.  Get consensus that the heirarchy of events is correct and the names of the business documents are consistent.  There is no point in doing BPM if you don’t know what your data is, and you don’t know what the events are.  The same holds for SOA. 

This is not a top-down design, but it is a top-down constraint.  If you build your SOA before you understand the data, you are creating multiple versions of the truth.  If you attempt BPM before you identify your events, you will burn huge efforts to compare apples to apple trees, producing no value.

Now, let’s all sit together and share a meal

Like all brothers, SOA and BPM don’t always get along.  They will have different viewpoints of both data and business events.  These two activities are usually performed by different people, in different parts of the organization.  Frequently, the are not connected in any specific way.  Their managers may be different, and their measurable metrics may be different.  Sometimes, they don’t even see how closely related they are.  Yet, there is a link, like the link between twin brothers.  An obstacle for one is an obstacle for both, and victory for one directly benefits the other.

Who works to get these two brothers to agree?  The Enterprise Architect does. 

Through both SOA and BPM, we get exceptional value for the enterprise.  Understanding how they are related helps to build the common shared understanding that these two activities need to truly succeed.

Should corporate bloggers go "internal only?"

By |2007-07-24T10:30:00+00:00July 24th, 2007|Enterprise Architecture|

Todd Biske asks a good question about corporate blogging: how do you build sufficient trust to allow for corporate blogging? 

It’s a good read.  Working for a technology company that is very large allows me a level of freedom not seen by my typical peer.  It is one reason I love working for Microsoft.  Innovation has a place here.

Not every Enterprise Architect can write a blog.  As an EA, I know a lot about company policy, strategy, and direction.  I tell none of that to folks outside the company.  On the other hand, most of the other folks who know the same information do not have a blog: Directors, General Managers, etc.  In most companies, having someone know what I know, and still trusting them not to spill the beans in a blog, requires a level of trust that would be difficult to reconcile.  Microsoft has trusted me and I take that trust seriously.

On the other hand, it is useful to be very open about Enterprise Architecture in general.  I want to improve the craft of Enterprise Architecture through sharing and discussing good ideas.  I want to nudge our industry in particular directions through tools, techniques, and good ideas. I can do a lot of that through working with my friends and collegues inside the amazing Microsoft machine.  On the other hand, I find it valuable to water down the Microsoft kool-aid by taking advice, sharing ideas, and being generally collaborative with folks who work on other platforms and share other concerns.

It’s a fine line.  Many companies are not comfortable allowing their practitioners to walk it. I can do nothing about that except to provide a counterpoint: EAs can be trusted to share without screwing up.  I hope it helps.

All models are wrong, some models are useful

By |2007-07-23T09:51:00+00:00July 23rd, 2007|Enterprise Architecture|

A collegue reminded me of one of my favorite “architecture” sayings yesterday, which I had on my door for a couple of years:

All models are wrong.  Some models are useful

The point is that we don’t create a model to be an exact replica of reality.  We create a model because it is quick and easy and guides us towards reality. 

That goes as much for models that are diagrams, models that are proofs of concept, and models that are communities.  That’s right: communities.

I am part of three communities at work that I value.  A group of SOA experts from around the company, a group of SOA experts in Microsoft IT, and the general Architects community in Microsoft IT.  All three give me things that I could not get on my own: a chance to learn, an opportunity to share, an place to discuss things that I find important, and last but not least, a place to build influence towards solving problems that slow us down.

But a community is a model.  It does not contain every person that SHOULD be there, and certainly not every person who COULD be there, and probably a person or two who should NOT be there.  It is a facsimilie of the real constituency that it represents: a non-scientific subset self selected by passion, support from management, and incentive.

So by extension,

All communities are wrong.  Some communities are useful.

It is up to the members of a community to make it useful. 

I commit to this: I’ll do my best.

What you need to make middle out SOA architecture work

By |2007-07-21T14:03:00+00:00July 21st, 2007|Enterprise Architecture|

In middle-out SOA, we want to do as little as possible from the “center.”  The real value is at the edge, where the services are being created and where value actually lies. 

Our goal, in middle out architecture, is to set up standards, mechanisms, and protocols that everyone will follow.  We will give names to services that need to be created, and we will describe a small set of standard message exchange patterns, complete with the names to be used at the endpoint.  Where possible, we will specify transport mechanisms as well.

Important: in Middle Out SOA, the central group writes NO code.

Let me give you an example.  In the metropolitan area around Seattle, there are quite a few suburban cities.  East of Seattle, across Lake Washington, is the city of Bellevue.  Next to it are the cities of Redmond and Kirkland.  These cities each began from their own centerpoint and grew outward until they collided in the same area.  The Microsoft campus is on a ‘spur’ of Redmond that is surrounded almost entirely with the city of Bellevue. 

Bellevue intersection with Redmond Washington


If you look at the map, you will see that there are PLENTY of roads that are either borders between the cities or cross from one city to the other.  As a driver who drives in this area, I can tell you that the roads have the same name on both sides of the city lines.  Normally, you do not know which side of the line you are on, or where the transition happened. 

Why is it so seamless?  It is not overtly controlled.  The cities are quite distinct from one another.  Rememer, they grew from their own centers out.  So why is it, when they intersected, the arterials lined up and the names are standard and the roads don’t change width or designation? 

It is because the planning departments in these cities all work together on standards.  There is a standard numbering scheme for the entire of King county (5,974 square km) that allows all streets, even ones in rural areas, to have names and numbers that don’t change from city to city.  The roads have a standard width, and roads of a particular arterial classification have standards for sidewalks and traffic lights.  I can find a business by its address, regardless of what side of the street that business is located on.

My car is a data packet that travels over these roads, and I have the routing mechanism in my brain, deciding the route and alternatives as needed to find my destination address.  It works because those roads and addresses are consistent and integrated.

The coordination between the two cities is practical and useful.  Each city has an ordinance requiring its planning department to cooperate with other cities.  That is important.  The ordinance is a fixed decision, made by their governing bodies, that gives their planning boards support and direction.  Without the ordinance requiring cooperation, passed by each city, the roads would not work.

This is the fundamental idea behind middle-out architecture.  Know the MINIMUM set of standards that everyone has to agree on, and get the management to buy in to requiring that minimum amount of cooperation.  Then cooperate.  Each area is responsible for their own governance. 

It is an inherently distributed mechanism, and therefore scales well.  Note that the minimum set useful in an enterprise is larger than the minimum set useful across the internet as a whole (at this time).  Mashups and SaaS services will not have a consistent mechanism for sharing data… at least not until customers demand it.  On the other hand, in an organization, consistency at the data level makes more sense and therefore a common event and schema model can be expected and used.

Middle out is not new.  It is a way of describing what works in other areas.  It is only new to SOA because SOA is new to IT.  We are still learning what other folks already have learned.  We are still catching up.

SOA without executive support

By |2007-07-20T10:12:00+00:00July 20th, 2007|Enterprise Architecture|

Joe McKendrick asked a really good question in a recent blog that I’d like to tackle: how to build SOA when executives don’t care. 

A little background: Joe McKendrick and I have been having an interesting discussion about SOA development styles: top-down, bottom-up, and middle-out. In his most recent post, Joe titled (rather humorously) my analogies for the different styles as “Service-Oriented Anarchy” (bottom up) and “Soviet-Oriented Architecture” (top down).  Gotta love that.

I had drawn the analogy that bottom up was very “local focused” and capitalistic/democratic, and that is why it appeals so much to us, but that enterprises are not democracies.  They are, at best, like a benevolent monarchy, with the CEO (and executive management) setting strategy.

In that post, Joe concluded with a very pertinent point:

“But, again, you have to be fortunate enough to have a benevolent monarch in charge. For too many enterprises, the monarchs are simply clueless and worrying about keeping their golden parachutes packed — not about SOA.

How do you build a SOA in an environment where management has no idea of its value?  What do you do when you are at ‘ground zero’ in a company that doesn’t value elegant enterprise design?

First off, let’s get something straight: there are two uses of SOA.

Use #1: Application Design.  I can use SOA to elegantly seperate my workflow from my activity and entity services when partitioning my application.  This is good.  This is an example of starting small, as Joe advocates.  If you can demonstrate, after you develop a SOA app, that it is easier to integrate with, you may get some good attention from management for doing more.  From the app perspective, SOA is extra goodness.  It doesn’t deliver better functionality, just the ability to integrate better later.  It’s actually a bit of a tough sell sometimes.

Use #2: Enterprise Agility.  If I build enterprise services, then I can compose altogether new applications from existing services.  This provides real speed and value to IT, and delivers on the promise of SOA.  Here, SOA is not just “added goodness.”  When providing enterprise agility, SOA is the clear winner.  It is in support of this goal that SAP has added services to their huge ERP system, and Seibel provides services to their dominant CRM system. 

Everyone wants the Enterprise Agility.  So, how to get it?

  • Bottom up advocates suggest that, in organizations where management doesn’t understand or doesn’t care, we can simply tell the dev leads and architects to use SOA in all applications (Use #1) and store the services in a catalog.  After a while, the catalog grows large enough that those same architects and dev leads can start using those services to achieve enterprise agility (Use #2).  It’s a natural growth, like urban sprawl.
  • Top down advocates (are there any left?) suggest that we need to describe and develop all services first, and then tell everyone to use them.  The catalog matters, but only as a directive center: you WILL use THESE services.  Kinda harsh.  Doesn’t work.
  • Middle out advocates suggest that a small group of people should get together and create a standard approach.  That approach will set standards for identifiers, formats, and protocols.  They will pick patterns and naming conventions that allow flexibility later without sacrificing the ability to deliver now.  They will name the services they need (without designing them).  They will share this information with architects and dev leads, and offer reviews and test harnesses to insure that services, once developed, will be enterprise ready.  They then step back and follow the bottom-up path, allowing applications to develop the services according to those standards.  Gabriel Morgan does a good job of taking a whack at the middle out approach in his blog. 

So, how to develop SOA in an environment where you have no executive support?  First, start small.  Second, use middle out.  Top down builds documentation, but is usually cancelled before a single service is built.  Bottom up builds the wrong services, giving you a sprawling landscape of useless points of integration.  Bottom-up is like trying to make sense of a Pentacostal congregation when they are “speaking in tongues.”  (There’s a lot of noise, but no one is sure what anyone else is saying). 

Middle out is your best hope. 

You can do middle out in a small way.  You don’t need 30 people for a year to make decisions for you, or some huge expensive project to decide what services to build.  A small virtual team can build the standards, and they can be written down in a single document of less than 30 pages, along with some sample code that developers can share.  You can even use a domain-specific language or a software factory if you want to mess with the new stuff in Visual Studio.  I’m working on standards of this kind right now, and hope to publish something by the end of the year for Microsoft IT to use.  (Yes, I will share).

The point is: middle out requires one thing: that you communicate with other architects before you build a bunch of services. 

We can do that much, even without getting the monarchs to notice.

Is Durability in messaging 'bad'? Libor thinks so.

By |2007-07-19T22:00:00+00:00July 19th, 2007|Enterprise Architecture|

Thanks to Harry Pierson for pointing out an interesting post from Libor Soucek.  He links to Harry and myself and states, basically, that durable messaging is not needed.  The logic is a bit hard to follow (possible due to the lack of a paragraph break anywhere in the post).  From what I can tell, he is saying this:

Libor’s logic: Durable messaging is a leaky abstraction.  In order to handle the leaks, you need to code compensation logic.  That creates complexity.  The complexity overcomes the benefits of the durable messaging.  Durable messaging is costly in terms of time.  Therefore, costs exceed benefits.  Therefore, don’t use durable messaging.  (I hope I got that right).


One fact that I think Libor missed: If we Don’t use durable messaging, then we have to put in compensation logic, because messages WILL fail.  (He did admit this, but doesn’t include this fact in his argument).

So, let’s look at the alternatives:

  • Durable messaging requires (according to Libor): complex compensation logic
  • Non-durable messaging requires: complex compensation logic.

Seems like this variable exists in both scenarios (again, according to Libor).  It is hardly a differentiator.

My take is actually that the AMOUNT of compensation logic needed drops dramatically when you have durable messaging.  Compensation logic is still required, but there is less of it, and it is less complex.

Therefore, the comparison SHOULD be:

  • Durable messaging requires (according to Nick): simplified compensation logic
  • Non-durable messaging requires: complex compensation logic.

Additional note: if you have a scenario where a message is being sent regularly, and it is OK to lose one, then you don’t need to use durable messaging.  Of course, if you have a scenario where neither the publisher nor the subscriber will ever fail, then durable messaging is overkill. 

On the other hand, if you have two systems, one is 99.99% reliable, and the other is 99.99% reliable, and you don’t use durable messaging, your reliability of the SYSTEM drops to 99.98%.  You’ve gone from four ‘9’s to three.  Four nines is HARD to get.  Losing that coveted level because of an unreliable message call pretty much kills the business case for message based integration.

I stand by my assertion.