Is it Service Oriented if the message cannot be intermediated?

By |2007-05-14T03:33:00+00:00May 14th, 2007|Enterprise Architecture|

I’m looking at an architectural problem.  I’d love to get feedback.

I’m trying to do two things at once: work on a SOA-based system design, and develop SOA-solutions for other applications to use by working with other teams that are also developing SOA-based solutions.

So, one architect, who I respect and enjoy having rigorous debates with, has suggested that SQL Service Broker is, on its face, a SOA technology.  SSB detects events, sends messages, and reliably insures their delivery to another SQL Server-based system.  It does this by posting information into ‘queue’ tables, and then, using SQL Server, transferring this information into ‘recipient’ tables on another SQL Server.  It’s an inherently SQL-to-SQL technology.

My opinion is that SSB is part of a SOA, but by itself, it doesn’t produce messages that are (a) technology agnostic, and (b) able to be intermediated.  You can use SSB to generate a message, but you’d need to call out to WCF or ASMX or something to actually transmit the message.

It’s not a question of whether a message ‘needs’ to be intermediated. 

I believe that we have message based systems for a purpose, and the purpose is to seperate the caller from the receiver.  If we bind them together in an infrastructure where the message cannot be intercepted, then we have removed one of the key benefits of SOA:

Intermediability: SOA should give us the ability to intercept a message going from point A to point B, and react to that message without informing either end of that pipe. 

In other words, we should be able to introduce an intermediary at any time, without affecting the caller or the sender.  To be clear: We don’t have to “inform” either system if their code and configuration remain unchanged.

Technology independence: SOA should also require that I can replace system A with a system of an entirely different technology, and that does not affect system B, as long as the message is consistent.

In both cases, in my humble opinion, passing a message via SQL Service Broker only, from one SQL Server to another, without calling out to WCF or another messaging based mechanism, fails. 

What do you think?  Are these elements (Intermediability and Technology Independence) truly integral to SOA?  Do you believe that we can deliver on the promise of SOA if we back down on either one?


Mining for Services with Solution Domain Architecture

By |2007-05-04T22:09:00+00:00May 4th, 2007|Enterprise Architecture|

In Microsoft IT, we are reapproaching the problem of SOA governance from two angles: bottom up and top down.

Bottom-up services are services that the IT teams build as part of their normal work.  They are project focused, and tend to be fairly well aligned to line-of-business applications.  If there is any reuse of these services, it is a happy accident.  We make them visible across different IT teams, but they are primarily used by the teams that create them. 

The exceptions are some minor generic bottom-up services that are developed by an ‘edge project’ using enterprise data.  For example: we have an enterprise data store that contains information about employees and the corporate heirarchy.  Using this data, it is fairly simply to construct an organizational chart.  This data is available by subscription to a central MDM data service that we’ve had in place for years.  That service is primarily ETL based, and does not offer up web services. 

That said, if an app were to develop an internal web service that looks at this enterprise data and provides simple retrieval and analysis (to answer questions like ‘where does Nick work?’ and ‘who is Nick’s manager?’) then the service could easily be reused across a wide array of applications.  Why? Because the service is against enterprise data that every application needs to align with.  While the service was developed “at the edge,” it uses only “core” data, and therefore is useful for a wide array of applications.  These services can be readily reused.

So bottom-up services are not the primary win for SOA agility.  To compose an application out of enterprise parts, we need to bind together on enteprise data.  For that reason, we need services that communicate using enterprise (and not local) data, using enterprise identifiers, in a manner that is consistent with enterprise needs.  Those services are rare. 

In fact, those services are nearly always the ‘top down’ services that we know, in advance, that we are going to need. 

How do we know what services we need to create?  Through Solution Domain Architecture.

 Solution Domain Architecture in a Service Oriented Analysis method (part of SOPADOG) that we are pioneering in Microsoft IT to solve multiple problems:

  1. How do we guide various projects to create services that are actually needed by other business applications?
  2. How do we develop services that will allow us to compose agile applications?
  3. How do we minimize dependencies between systems while still creating an integrated infrastructure?

Solution Domain Architecture is an analysis method that starts by grouping the various functions of the enterprise around the data elements that are generically understood to be enterprise data (like invoice, product, asset, identity, etc).  These groups are called Solution Domains (hence the name) and represent an abstract concept.  The solution domain is the abstract collection of system features that are cohesively bound to shared data items.  These sets form the “tools” part of a business capability.

(A business capability is a combination of people, process, technology and data.  The Solution Domain is a mapping against the software part of the technology.)

There is a many-to-many relationship between business capability and solution domain.  It is perfectly normal for a single business capability to require the features of multiple tools (like learning management needing tools in the domains of employee management, portal collaboration, and knowledge management). 

 There is also a many-to-many relationship between solution domains and the technical capabilities of our platform tools (like messaging, authorization, database management, and auditing).  It is, for example, normal to find many tools that need database management.  However they need them in different ways for different databases. 

This creates a three-level view that places business capabilities on top, solution domains in the middle, and technical domains across the bottom.  The diagram below illustrates a small subset of the areas, and their relationships.

(Click the diagram for a larger view)

Solution Domain Model Example

It is important to note that this is NOT a stack diagram.  Each solution domain contains a set of logical systems services.  These services  represent enterprise solution capabilities.  When you make this mapping, each solution domain contains one or more systems, one or more services, and one or more databases.  A solution domain may have many user interfaces.  It may involve systems in many geographies.  It may deliver capabilities to users using many different devices. 

This perspective is not commonly seen, but it is essential to understanding how to create services.

Look for a moment at the wires.  The wires between layers are dependency (for example, Partner Relations depends upon Relationship Management solutions, which depends upon Database Management technologies).  In the model above, I used a light grey color to distinguish them.  These links are useful to understand the demand that the business places on the layer below it.  This is a planning concept.  We want to know what solution domains are used by what business processes because we invest in business processes.  This allows us to easily scope out the areas where an update to a business process will affect the tools.  IT planning is all about the affect on the tools.

One nice thing about mapping this way: if the business strategy says “focus on a particular business process” then you should see your IT investment occur in the solution domains that are mapped to those processes.  If you see your IT spending going to other domains, ones that are not mapped to the strategic processes, then you do not have good IT-Business alignment.

That’s right. I can illustrate the ‘level’ of IT-Business alignment with this diagram, by adding ‘heat map’ colors. 

Let that sink in. 

Now, look at the lines between the solution domains in the same layer.  These are data dependencies.  I used a thicker line in the diagram above.  For example, Obligation Management depends on Product Lifecycle Management.  That is because we are obligated to deliver what we sell, and we only sell what we make, so to know what we sell, we depend on the list of products… which is managed in the Product Lifecycle Management solution domain.

These lines between solution domains represent the flow of information between domains.  They represent the movement of business documents in response to business events.

In short, the lines between solution domains are the enterprise service dependencies.  At one or both ends: a service endpoint.

It really is that simple.  By understanding these dependencies, we can chart out the services that are needed to move the data.  Those services are placed into the catalog before they are ever written, and the EA team learns about them and plans for them.  Then, when a project is proposed, by the business, in Partner Relations, we can see that there will be an impact on Relationship management. 

We then look at the planned services in Relationship Management.  Let’s say that we want there to be 10 services, but currently there are only four.  So the project that the business proposed gets to add one more… because it is the right thing to do… and because by creating that service, perhaps we can now link to the collaboration domain.  In other words, we can show customers their customer service request status on the web site (web portals are part of the collaboration domain).

At the core of this is the fact that these services are driven by the business, but planned by IT, and implemented on demand, when projects arise.  That is Solution Domain Architecture. 

[addenda: I want to add one bit that was pointed out by a friend: Solution domains are an attempt to help understand the demand for services from an enterprise perspective.  There is nothing in Solution Domain Architecture that prevents or opposes the development of services for Line-of-Business specific needs.  Those services are not fully captured by this method, but they are not opposed by this method either.  In other words, top down and bottom up are quite happy together.]

Bad assumptions yield false conclusions

By |2007-05-02T03:30:00+00:00May 2nd, 2007|Enterprise Architecture|

On ZDNet, Robin Harris prognosticates about the eventual failure of Enterprise SOA on his blog.  His article is full of inaccuracies and flawed logic.  I will point out some of the most obvious.

WTF #1 

In his post, he states:

“…there is no way for enterprise IT to monetize the advantages of SOA. Without that, it is game over.”

There are two assumptions here: both of them very poor.

Assumption 1: If your SOA doesn’t save you money, it’s ‘game over’

SOA is not, and never was, about saving money.  An inexperienced novice may suggest that idea, but no one who has ever implemented a service oriented business app would, and certainly no one who is responsible for a SOA infrastructure.  SOA is about business agility.  If you deliver agility, SOA pays for itself overnight.  If you don’t, you’ve wasted the time and money of your employer.   

Assumption 2: SOA cost savings cannot be justified since IT is paid for by taxes.

Not all IT groups are run the way that Mr. Harris suggests that they are.  Many IT groups do ‘charge’ for their services (although rarely do they charge the full cost).  In fact, there are a great many articles and posts that recommend aligning IT to the business by using an effective chargeback model.  In this space, the cost of IT is clear to the business, as is the need to cut it back. 

This, of course, is a spuruous argument, since cost savings has nothing to do with SOA

WTF #2  

Another quote in a long line of flawed statements from Mr. Harris’ blog:

“SOA is about providing services for applications. So how do you measure success?”

Once again, the author appears to think that we would go to all this trouble just to provide services to applications.  That is the silliest thing I have ever heard.  What application ever paid for anything!

We do SOA to kill overlapping and redundant applications.  We do SOA to delete multiple redundant data stores.  We do SOA to improve data integration.  We do SOA to highlight process overlaps.  We do SOA to deliver consistent experience to our customers and partners. 

WTF #3  

Another quote from Mr. Harris, while I’m on a roll:

“If an application is relying on a dozen independent services whose timing and delivery are dependent on a dozen different infrastructures, that application can only be as fast as the slowest service “

That is true if all your services are synchronous.  Most true Enterprise Services ARE NOT Synchronous.  Therefore, this statement is simply not true.

I review and oversee service development.  When I find a service that is synchronous, I treat it as a ‘warning flag.’  I ask for a detailed design review to uncover any flawed assumptions the developers used.  Most of the time, I can successfully move them to an async service by simply changing the assumptions of the service. 

PEOPLE ARE ASYNC.  We are “store and forward” and “I’ll get back to you on that” and “I trust you will handle this.”  We run large and complex infrastructures, made up entirely of other people, using async communications.  (We call those organizations “businesses” but we could just as well call them “armies” or “governments” or “charities”).

WTF #4  

Just to cap off Mr. Harris’ bizarre tirade, he offers up this beauty:

“SOA is fundamentally about making enterprise IT cool like the web, which is a lost cause. “

Give me a break.  I don’t get paid for cool.   I don’t want to get paid for cool.  If I wanted cool, I would hire some 20-something consultant in a black turtleneck to come in an make a pitch for Web 2.0.  I sure as heck wouldn’t try to rally an 6,000 person IT organization around the goals of reducing time to value, reducing data redundancy, improving business intelligence, or reducing operating overhead for an overlapping and overly complex application portfolio.

And I sure as heck don’t think SOA is cool.

His heart appeared to be in the right place… now if we can locate the brain…    

To be fair, what I think Mr. Harris was trying to say was this: cost cutting comes only after the business can fairly see the costs of IT and can offer their input into specific measures for cutting IT costs.  If that were the point of his post, he would find me to be a supporter and fan.  I agree.  In Microsoft, our IT budget is a blended mix of operating cost via a Tax model and business funded expenditures via a direct sponsorship (pay as you go) model. As a result, at least SOME of the costs of IT are visible.  We would do well to make more of them visible.  Regardless, we get many of the benefits of ‘business ownership of expenses’ as a result.  The business leaders scrutinize the IT Project Portfolio with a sharp eye and an accountant’s pencil.  Everything has to be justified. 

Unfortunately, this model solves some problems and creates others.  If every business leader spends money only on their own, short term needs, then the enterprise needs are rarely met.  While the system has its benefits, it also has its flaws.  Chargeback is far from a silver bullet. It can solve some problems, and create others that are just as tenacious, just as insidious, and just as expensive, as the IT-as-a-tax model.   

WTF #5 

If he were just talking about the benefits of the IT-chargeback economic model, Mr. Harris would be making a useful, if somewhat uncontroversial, statement.  On the other hand, Mr. Harris decides to compare the benefits of an IT economic model with efforts in systems architecture and proceeds to announce that various architectures “aren’t the answer,” as though to state that the one silver bullet to solving all IT problems is a chargeback model.  

I really don’t know what to say to that except to point out the obvious: he is comparing tomatoes to trebuchets.  Architecture does not solve the same problems as a chargeback model would.

In his last paragraph, this odd juxtaposition is most clear:

“Prices aren’t a perfect method of allocating resources, but they are the most effective tool we’ve got. SOA isn’t the answer, just as ILM, OOP, client server and all the other nostrums peddled through the years weren’t either.  “

I guess the only rebuttal I can offer for such a bizarre comparison is to point out that chargeback models for SOA are fairly simple to implement… and can be quite accurate given a reasonable security model. 

Mr Harris: you can have your cost model, and I can do it on top of SOA better than any of the other architectures you mention.  Perhaps the next time you decide to launch in to a critique of how vegetables make poor siege weapons, you will think about what it is you are comparing. 

Why "Service Oriented Analysis and Design" Starts Late and Ends Early

By |2007-05-01T09:14:00+00:00May 1st, 2007|Enterprise Architecture|

One thing that comes through from my SOA Governance talk last week: SOA Governance is a set of processes that starts in the early planning stages and crosses through to operations. 

So, with Governance in the back of my mind, I was reading up on some of the work done by various authors in the area of Service Oriented Analysis and Design to understand some of the formalisms that have been proposed.  I wanted to see how others were addressing the planning governance space.  Unfortunately, they all start from the wrong spot.

SOAD starts from the same place as OOAD starts.  There is an assumption that the business wants an application to come into existence and the architect is then brought in to solve the problem.  It is a very project-focused solution.  It is like a business owner buying a plot of land in a city and calling an architect to help him to build his five-storey office building. 

That is the wrong spot to start SOA work.  For SOA to be useful to the enterprise, the services must span applications.  They must be visible outside the scope of the app, but used by them.  In the ‘city’ example: Services are not the foundations of a single building.  They are the electric grid that all the buildings use. 

And not just an ordinary electric grid: imagine an electric grid where every building generates power according to the amount of light and wind that comes to the plot.  Some buildings will supply power and others will comsume it.  Making that kind of grid function cannot be done after the land owner decides to build the building.

SOA has to be built long before the application owner decides to build the application.  There has to be infrastructure in place: patterns, messaging, MDM.  More importantly, there has to be a plan in place for that ‘site’ where the application will sit.  What applications are already in place in the enterprise?  What services do they offer?  What services SHOULD they offer?  What overlaps are you introducing?  How will your data be found?  How will it be secured?  This is the realm of SOA planning.

Starting with the building plan is too late.  SOA needs to start with the city plan. 

So when we talk about Service Oriented envisioning, let’s not start with Analysis and Design.  Let’s start with an entire regimen around Service Oriented Planning, Analysis and Design.  SOPAD.  We have to know where the services belong before we build them.  We have to know what they need to do before we write them.  We have to know how to keep them running before we deploy them.

And then we get to the tail of this dog.  We have to run our services in Operations and we have to Govern our services to insure compliance.  Service Oriented Planning, Analysis, Design, Operations, and Governance.

The term is not SOAD. 

It is SOPADOG: Service Oriented Planning, Analysis, Design, Operations, and Governance.