/Tag: SOA

Sharing the Solution Domain Taxonomy

By |2015-04-23T10:17:18+00:00April 23rd, 2015|Enterprise Architecture|

Sometimes, Enterprise Architecture efforts fail.  This is no surprise to folks in the EA business.  This failure occurred slowly, back in 2007 and 2008.  But it did occur.  It took me a while to realize it. 

I had developed a method useful for Application Portfolio Management as well as for Service Oriented Architecture called “Solution Domains”.  The method is good.  It’s a framework and taxonomy for high level descriptions of software so that generalized services can be created AND so that the portfolio of applications can be rationalized.

The method is good.  But I failed to position it’s use in the appropriate enterprise program in the appropriate way.  I failed.  Not the method.  Where we used the method, it worked brilliantly. 

I’ve learned from my mistakes, but being unwilling to let a good thing go to waste, I’m sharing the Solution Domain taxonomy with the world.  It’s not patentable (I tried).  It is useful, however, because it is a part of a business method that supports Application Portfolio Management in a completely technology agnostic manner as well as Middle-Out SOA.

I’ve put the entire taxonomy on my Enterprise Business Motivation Model site at: http://motivationmodel.com/wp/application-portfolio-management-and-solution-domains/ 

I may return here, at some point, and provide further details on how it can be effectively used.  For now, back to work!

Placing Architecture Properly into Scrum Processes

By |2016-09-28T22:44:52+00:00June 11th, 2013|Enterprise Architecture|

As I’m about to complete my share of a longer engagement on using Lean principles to improve the processes at an online services firm, it occurred to me that the efforts we undertook to properly embed Architecture practices into their Scrum process were novel.  I haven’t seen much written about how to do this in practice, and I imagine others may benefit from understanding the key connection points as well.  Hence this post.

First off, let me be clear: Agile software development practices are not at all averse to software architecture.  But let’s be clear about what I mean by software architecture.  In an agile team, most decisions are left to the team itself.  The team has a fairly short period of time to add a very narrow feature (described as a user story) to a working base of code and demonstrate that the story works.  The notion of taking a couple of months and detailing out a document full of diagrams that explains the architecture of the system: pretty silly.  (more…)

Linthicum’s Challenge: Where does SOA stop and EA start?

By |2010-09-19T23:28:03+00:00September 19th, 2010|Enterprise Architecture|

Tom Graves, David Linthicum, and I recently got into an interesting discussion on Twitter as the result of a, EBiz blog post by David, where he makes the statement that Good SOA is the same as Good EA.  (See ‘Do SOA and enterprise architecture now mean the same thing?’ Yes, they do’).  Both Tom and I maintain that SOA is a good practice for an EA to endorse, but that it does not equate to good EA.  In this post, I’ll answer the question “why?”

I’ll include the last few tweets for context.  Tom started the thread by noting that he disagreed with David’s premise. I joined in.

Nick Malik

 nickmalik

thx @tetradian for pointing out @DavidLinthicum – SOA *is* good IT arch but EA is far more than IT arch, so SOA != EA.

Tom noted

Tom Graves

 tetradian

@DavidLinthicum SOA is a (useful) *style* of architecture, not architecture itself – EA must include what happens when the power goes down

David replied, quoting my tweet and responding (emphasis added):

DavidLinthicum

 DavidLinthicum

RT @nickmalik & @tetradian SOA *is* good IT arch but EA is far more than IT arch, so SOA != EA.<-Love to hear where SOA stops and EA starts.

So this post is about answering David’s question: where does SOA stop and EA start?

Let’s start with the mission of Enterprise Architecture.  In a prior post, I discussed the mission, vision, and business capabilities of Enterprise Architecture.  To quote from that prior post:

Enterprise Architecture exists to do two things:

  1. to Envision an organization that includes the strategic and operational changes that leaders require and demand, and
  2. to Guide changes to the structure of the organization, and the actions of the business units themselves, so that the vision becomes reality.

Look carefully.  Do you see anything about SOA in that mission… or even information technology? 

Enterprise Architecture envisions an organization in its entirety (structures, responsibilities, roles, processes, systems, information, and technologies).  We don’t get to choose to ignore some parts of the puzzle and focus on other parts.  When you solve a problem, you have to consider all of the factors that will come into play.  Enterprise Architecture is accountable for guiding changes to organizations where those changes are necessary to solve a problem. 

What problems are we solving?  Usually, we focus on problems created when an organization is not ready… ready to respond to a new business strategy, ready to respond to a competitive threat, ready to respond to a market opportunity, ready to respond to change.  There are many ways to focus on the problems of the business.  Below are a few that fall within the scope of Enterprise Architecture:

  1. To make sure that the business has decision making mechanisms in place that create trust, apportion accountabilities, measure and report on results, and encourage accountability. 
  2. To make sure that the organization’s abilities (and flaws) are well understood and documented, so that when a change is proposed, valid and useful information is available to the business to make decisions, approach trade-offs, and focus resources on the specific issues that could prevent the business from making that change. 
  3. To make sure that the business frames their strategies and goals in a manner that is aligned with their intent, are prioritized to minimize internal conflicts, and clarifies the specific, measurable, actionable, realistic, and time-bound (SMART) tactics that must be implemented in order to make them come to pass.
  4. To insure that potential innovations in the marketplace, environment, and competitive space have a fair and reasonable mechanism for being considered.  This has the effect of creating the “funnel of ideas” that the IT group can use to feed in potential technical innovations.  The same funnel, BTW, is used to feed in information about competitive threats, new business opportunities, product innovations, etc.
  5. To insure that appropriate models of information are created and to guide the development of information management practices that help to reduce the complexity and difficulty of managing the consistency, accuracy, reliability, timeliness, and security of the information assets of the business.
  6. To insure that the processes used by the business to perform core or supporting business functions are well understood, and that the appropriate attention is paid to making process improvements when the business strategy demands it.
  7. To guide the development of business services that allow for the simple composition of business processes in such a way that the information can flow readily to the appropriate people, that the customer is delighted with the result, and that the stated business goals of the business can be met in a timely and agile manner.
  8. To guide the development of technology services that allow for the simple composition of business systems in such a way that the systems are reliable, available, and appropriately manageable, so that the business is not constrained in their ability to create the necessary compositions implied in the prior step.

Depending on how you define SOA, Service Oriented Architecture is either item 7 (business services) and 8 (information services), or just item 8 (information services), in the list above.  Occasionally, SOA is used to describe a mechanism that can support item 5 (information quality mgmt), but recognize that SOA, at its finest, is not sufficient to actually accomplish that item.  It is useful, and very valuable, but not sufficient. 

So, David, the answer to your question… what does EA do that rises above and beyond SOA?  The answer is listed in items 1, 2, 3, 4, 5, 6, and depending on your definition of SOA, 7 above. 

Other than that, SOA is really great.

Service Oriented Architecture Conceptual Model

By |2010-04-16T23:49:12+00:00April 16th, 2010|Enterprise Architecture|

Almost two years ago, I described some of the key concepts of service oriented architecture, including the distinction between a canonical model and a canonical message schema.  Since that time, I worked on a wide array of models, including Microsoft IT’s Common Conceptual Model.  That model (CCM for short) is the metamodel for IT concepts in Enterprise Architecture.

I received a note recently about that old post, and how valuable it was to the reader.  It occurred to me that those concepts were part of our Common Conceptual Model, and that a “SOA-Specific View” of the metamodel may prove interesting.  It was. 

I’ve attached the SOA view of our Common Conceptual Model for this post.

SV - SOA Concepts

A quick set of notes on my conceptual models.  

  1. Two terms separated by a slash (Application / Installable Software) are distinct terms for the same concept. 
  2. The associations are all “arrows with verbs”.  Read the associations along the direction of the arrow.  (e.g. “Application Component implements a Shared Service”).  I intentionally avoided using UML notations like “Composition” and “Aggregation” because they are difficult for non-technical people to read.
  3. Many verbs on an arrow – The stakeholders couldn’t agree on which verb best described the relationship, so we used multiple distinct verbs. 
  4. In my prior post, I used the term “Canonical Schema.”  Our stakeholders preferred “Shared Message Data Contract.” 

 

The key relationships to notice in this model are the ones that may be the least expected ones: the connection between the obvious SOA concepts like “Contract” and “Canonical Entity” and the not-so-obvious SOA concepts, like Business Process and Business Event. 

When SOA is done well, it is part of a seamless ecosystem between the processes that people perform and the systems that support them.  Being able to see the parts of the system and how they connect is key to understanding how to build a truly effective service oriented architecture.

The cost of “SOA-fication”

By |2010-02-15T17:54:19+00:00February 15th, 2010|Enterprise Architecture|

No, Virginia, there is no SOA Santa Clause.  SOA is not free.

That said, if I’m changing a system to meet new needs, and I’m substantially refactoring a section of the code to deliver to those needs, SOA doesn’t have to be wildly expensive either.

The myth of “expensive SOA” is just that: a myth.  If you have an existing system, and you are refactoring it anyway, it may make sense to spend a bit more money for “SOA-fication” (the process of turning a “closed system” into a system based on Service Oriented Architecture principles… if you can’t get all the way to “based on SOA,” I’m happy with “plays well with SOA.”)

Of course, that cost is not trivial to estimate.  This is where I’d like to ask the community for your input.  What drivers have you seen to drive the additional cost of “SOA-fication” of an application?

Here’s what I’ve observed…

  • “SOA Security” (Auth/Auth) – to insure that there are no additional risks to the enterprise through the availability of a previously inaccessible system feature.
     
  • Design and Design Review – to insure that the services being developed are truly worth investing in.  That means that the services exhibit good behavior for enterprise SOA services wherever possible (non-chatty, independent, transactional, reliable, traceable, etc.)
     
  • Testing – to insure that the interface is stable, meets the stated business needs, and can realize the stated design goals.
     
  • Monitoring – to insure that the service can be discovered / tested / validated and, depending on the service, that it correctly and reliably participates in Business Activity Monitoring and Workflow scenarios.

 

Gentle Reader: Do these “dev cost drivers” coincide with your experience in turning legacy applications into SOA applications?  What “pitfalls” come to mind that are not listed?

(I’m interested in actual experiences.  Please share.  There are no wrong answers here… just good people helping one another.)

SOA Optimistic Data Synchronization considered harmful

By |2009-09-04T20:58:00+00:00September 4th, 2009|Enterprise Architecture|

Let’s say that you have two systems: Adipose and BellyFat.  They both need the same information.  Adipose handles customer transactions, so it needs information about customers.  BellyFat handles the long-term management of customer information, like what products they have purchased and what rights they own.  It also needs information about customers.

How do we keep the customer information, in these two systems, in sync?  SOA offers three answers: Call-and-Response, Optimistic Data Sync and Pessimistic Data Sync.

  • Call-and-Response says: One system holds the data.  The other does not.  The system that does not hold the data calls the one that does, on a real time basis, and gets the data back quickly and efficiently.
     
  • Optimistic Data Sync says: Both systems keep a copy of the data.  If an event happens in either system, drop it on the bus.  The other system will get the event, interpret it correctly, and update its internal store to reflect the change.
     
  • Pessimistic Data Sync says: One system masters the data, but the other system keeps a local copy.  If an event happens in either system, drop it on the bus.  The other system will get the event, interpret it as best it can, and update its internal store according to its own business rules.  On a periodic basis, the ENTIRE data structure will be copied from the master to overwrite the data in the local copies (data refresh).

Each of these three has its own strengths and weaknesses.  One of them, Optimistic data sync, is so bad, however, that I’d like to call it out for special ridicule. 

Type Advantages Disadvantages
Call and Response
  • Any data entity exists once
  • Less duplication of data, lower physical storage
  • One view of the “truth”
  • Diminishes need for ETL capabilities
  • Easy understanding for software developers that are familiar with relational database design
  • Consistent inter-system data models not requireed
  • Constrains architecture – provider and consumer must be “close”
  • Requires highly available data providers
  • Cumulative negative impacts on overall ecosystem reliability
  • Requires highly scalable messaging infrastructure
  • Fosters point-to-point messaging
  • Leads to rapid increases in complexity and management cost
Optimistic Data Sync
  • Allows multiple systems to “master” a data entity
  • Diminishes need for ETL capabilities
  • Encourages loose coupling between systems
  • Supports systems that are wide distances apart. 

 

  • Requires highly scalable messaging infrastructure
  • Requires highly reliable messaging infrastructure
  • Assumes that data updates are always consistently applied
  • Data gradually gets out of sync, with no recourse to get it right.
  • Consistent data models are a necessity
Pessimistic Data Sync
  • Does not require expensive messaging infrastructure
  • The amount of “correctness” between systems can be carefully managed
  • Encourages loose coupling between systems
  • Consistent inter-system data models not required
  • Requires system architects to agree that one system is a master
  • Data gradually gets out of sync, but administrator can use refresh mechanism to resync
  • Requires ETL capabilities

 

You will choose one over the other depending on your tolerance for the “disadvantages” and your preference is for the “advantages” of any method.  However, and this is the focus of this blog post, one of these three is really not workable in the long run: Optimistic Data Synchronization.

The reason for my enmity for this approach is simple: this approach uses an underlying assumption that is poorly considered.  That assumption is that it is fairly easy for two systems to stay in sync simply by keeping each other abreast of all of the events that have occurred in either one.

The problem with this assumption is that it is NOT easy for two systems to stay in sync.  If the two systems don’t share an IDENTICAL data model, then each system has to interpret the messages from the other.  The rules of that interpretation have to be coded in each system, and that code must stay perfectly in sync.  Plus, there can be no errors in interpretation, or variations in the way that a change propagates throughout the recipient’s system.  There can be no variations in the event model between the systems.  No bugs either.  Sure…. if we can get to the point where no humans are involved in writing computer applications, then this might make sense.

Not long ago, I used to think that Optimistic data sync was a good idea, and that SOA developers should assume that their data exists outside their systems.  Heck, I used to think that call and response was a good idea too.  Both are bad in practice, with Optimistic sync being by far the worst.  There are just too many common scenarios (like one system going down for a period of time, and coming back up after missing some messages) that drives down the overall integrity of data managed in this way.

While I’d go so far as to say that Pessimistic and Call-and-Response are reasonable patterns for a SOA architect to select, the optimistic data sync method should be considered an anti-pattern, and avoided as much as humanly possible. 

Creating a distinction between business services and SOA services

By |2008-11-30T17:19:00+00:00November 30th, 2008|Enterprise Architecture|

I’m always a bit dismayed when I hear the following terms mixed up, or combined: SOA service and business service.  In my mind, these things are different.  In one sense, they are related, but indirectly.

A business service is a function (or capability) of the business that is offered to one or more customers.  Those customers are often  internal, because this scenario is often applied to corporate supporting functions. For example, the accounting business unit may provide “accounts payable” services to every business division of an enterprise.  Those divisions are internal customers.  The business unit is accounting, and the business service is “accounts payable.”

In some cases, the customers of the function may be both internal and external.  Many years ago, the Carlson company took their marketing division and not only made it into a shared function, that their various internal divisions could use,  but that division was able to offer their services to the general market as well.  They provide a list of shared business services used by both internal and external customers.

The people who use shared business functions are “businesspeople” of all stripes.  They have work to do, and a business service is simply a way to do it.   A shared business service includes responsibilities, and therefore people who are responsible.  It is a kind of “sub-business” that has customers, and processes, and capabilities, and information.  In many companies, IT is run as a shared business service, providing technology services to many areas of the business. 

A SOA service is a different animal altogether.  Service Oriented Architecture (SOA) is an architectural style.  That means it is a set of software design patterns.  These patterns are united in their support of a basic set of principles.  The people who use SOA are people who write software.  (If you compose an application, even if it is simple to do, you are writing software.)

The logical data model that encapsulates this concept is below.  This is a very tiny part of the data model derived from our traceability model, which allows us to recognize the interdependencies between business processes, applications, and business units.  At the top of the image you see business services.  SOA services are on the lower right.  (click the image to enlarge)

A business unit may provide zero or more business services.  Not all of the capabilities required by a business unit may be involved in a business service. 

SOA provides the ability to share features.  Those features may provide information, or calculations, or data manipulation.  They may also include the limited automation of some elements of a business process.  SOA services are provided by “installed software” (we use the term “application” many times for this entity… a different blog post someday…).

Business-vs-SOA-Service

(note: I updated the image about 12 hours after posting this blog, due to an error in the original image -ANM)

The point of this post is to provide sufficient context to challenge the notion that SOA provides shared business services.  It does not.  SOA provides shared features that many business units call upon.  Those features are required by the business processes within those business units. 

Note to responders: before you flame me, take the time to try to map your concepts to the diagram above.  You may find that if you look for your concepts, and not your words, that you are simply using different words than I am to refer to the same concepts.  Disagree with me about concepts and I’m interested.  Disagree with me because I don’t use a word in the same way that you do, and we will probably not have a very interesting discussion.

Understanding Governance as Decision Rights

By |2008-10-04T04:13:39+00:00October 4th, 2008|Enterprise Architecture|

Todd Biske, whom I respect for his writings on SOA, seemed to miss the mark in his recent blog post about SOA Governance and Decision Rights.  In that post, he said:

if you focus on education, you can allow individual teams to make decisions, because you’ve given them the necessary information to make the right decisions. If they don’t have the information to make decisions that will lead toward the desired behavior, it turns into a guessing game. Not only can there be confusion in what the correct decisions are, there can also be confusion on what decisions should be escalated up the chain. If we instead focus on creating policies and making those policies known so that anyone in the organization can make the right decision, we’re in a much better state.

Was Todd wrong?  Nope.  He was right on. 

In describing governance, Todd just described a useful, and workable, set of decision rights.  I don’t think he realized it, because the blog was essentially trying to refute the concept of Governance as decision rights!

What were those decision rights he describes?

  • "You’ve given [individual teams] the necessary information to make the right decisions" — Implied in that statement is the notion that the individual teams, having made the right decisions, will not have those decisions taken away from them by someone else.  In order for a team to make a decision, they must clearly have the right to make it and, here is the kicker: Management Must Respect That Right.  Want that to happen?  Be explicit.  Tell everyone what decisions we will let the team make, and then hold them responsible for making them.
     
  • "There can also be confusion on what decisions should be escalated up the chain" — to avoid that confusion, we must be clear, as Todd correctly points out, about who has the right to make which decision.  Where does a decision stop?  That clarity avoids red tape.  That explicit clarity is called "decision rights."
     
  • "If we focus on creating policies" — And here is really the confusion.  What are those policies called?  They are called "decision rights." 
     

SOA Governance is a subset of IT Governance, on all three aspects of Governance: design time, deploy time, and run time.  SOA decision rights are a subset of IT decision rights, which are a subset of overall IT policies. 

What we have here, is a failure to communicate.

Malik's Laws of Service Oriented Architecture

By |2008-08-21T12:46:00+00:00August 21st, 2008|Enterprise Architecture|

  • No one but you will build the services you need in time for you to use them
  • If you build a service that no one else asked for, you will have built it for yourself
  • If you build a service for yourself,  you will optimize it for your own use
    • It is therefore the optimal service for you to use
    • It is very unlikely to be the optimal one for anyone else to use
    • No one besides you will use it
    • You will not use anyone else’s

Implication

Therefore, any team building reusable services must build each one only after two or more people have asked for it, with full knowledge that the resulting service will almost certainly be available too late for any of them to use it.

Therefore, no team should intentionally build reusable services.

Additional Laws and Corollaries

  • If you invest in improving someone else’s pre-existing service, you will create a reusable service.
  • Creating a reusable service, be improving someone else’s service, will cost you more, up front, than writing a completely new one.
  • The cost of maintaining a service increases proportionally to the number of consumers that use it.