Have you ever woke up in the morning with an idea in your head that you simply have to write down?  I just did.  Here’s the idea: Everyone talks about how important the catalog (or repository) is to Service Oriented Architecture.  It isn’t.

The reason everyone wants a catalog is simple: If I create a uniquely valuable service, and I want people to use it, I need a place to advertise it.  So I put it in a catalog.  The catalog contains useful information like what the service is, and what it does, and who made it, and how to call it.  Useful stuff.  Sears Roebuck, circa 1893.

So how can that be unimportant?

Because this is a case of ‘doing a really good job of solving the wrong problem.’

A friend of mine and fellow architect here in Microsoft IT named Mohamed El-Ghazali changed the way I think about service contracts.  And Gartner changed the way I thought about “what makes adoption work” and together, there’s a powerful brew.  It took me a while, because these ideas are “just different enough” to make me pause, but between these two sources, they had the intended effect, and now I can say, without blinking, that the catalog is not the high order bit.

Why? Because the catalog is not an IFaP.  It is a list of chaos. 

If you have 20 services, or even 50 services, a catalog is really useful.  I’m looking at an architecture that will require something around 500 enterprise information, activity, and process services, about 200 infrastructure services, and countless ‘point solution’ services.  There is no way a list will do.  No human can remember it, or use it.  Duplication and overlap will prevail.  Face it, the catalog doesn’t scale.

So where does the solution lie? 

How about looking to the past to find the future. 

I call your attention to the history of the Periodic Table of Elements.

If you are not familiar with the history of the creation of this simple yet extraordinarily powerful concept, you should read this page.  Two key concepts I’d like to pull out:

First off, by creating the periodic table of elements, Mendeleev created a situation not only where elements could be classified, but where missing elements could be predicted.

Between 1868 and 1870, in the process of writing his book, The Principles of Chemistry, Mendeleev created a table or chart that listed the known elements according to increasing order of atomic weights. When he organized the table into horizontal rows, a pattern became apparent–but only if he left blanks in the table. If he did so, elements with similar chemical properties appeared at regular intervals–periodically–in vertical columns on the table.

Mendeleev was bold enough to suggest that new elements not yet discovered would be found to fill the blank places. He even went so far as to predict the properties of the missing elements. Although many scientists greeted Mendeleev’s first table with skepticism, its predictive value soon became clear.

This meant that not only did Mendeleev help to understand the list of ‘needed domain knowledge’, he actually created boundaries that empowered other people to focus their efforts and deliver incredibly quick innovation.  This innovation came from people he had never met.

The second thing I’d like to highlight is that the original table was useful but it was changed as knowledge increased to match a more modern understanding of chemistry and modern techniques for measuring atoms that was not available when it was developed.  In other words, the concept is good, even if the implementation is iterative.  (19th century agility).  The boundaries remained, and the table stands today as a fundamental artifact in the understanding of our natural world.

What does that have to do with SOA?

I am creating a similar table of services based (loosely) on the layers defined by Shy Cohen, message exchange patterns defined by the W3C, the work on Solution Domains that my team in IT Enterprise Architecture has started, and the business behaviors that I see as necessary to accomplish a partitioned design.  The goal is to create an all-up IFaP of services based on multiple spanning layers. 

Unlike the periodic table, this will not be bounded by physics.  Instead, it will be bounded by the data elements and solution elements defined by performing a Solution Domain mapping exercise against the enterprise.  Your organization will have different elements, but either way, there will be boundaries, and that will, I believe, foster organized and directed effort, creativity, and discoverability.

I believe the value will be clear.

  1. We will know what services we need to develop to meet the needs of the enterprise.  We can even prioritize the list and create a roadmap showing the CIO when we will be “done.”
  2. We will have basic patterns already established for how they will be called and what they will return.  This reduces a huge amount of churn and will give brave developers the ability to resist the “not invented here” plague.  The patterns can be designed to include all the needs of the test and support teams that are normally ‘left out’ of application specs but are ever more critical to the success of SOA.
  3. We will have generic test harnesses in place to test them before they are written, allowing test architects to build reusable test value, while at the same time relieving project teams from writing difficult and complex test software to support SOA.
  4. We will have sufficient information to estimate their cost by the team that must build and maintain them, providing some visibility to the cost of developing an integrated application.  This gives us the ability to seperate out the incremental cost of SOA from the cost of application development in general.

I’m pretty excited about doing this, and I think it is a strategy that can work. 

So what part of this kills the catalog?

The catalog helps a programmer to find the name of a service that performs a specific purpose. 

However, if I know the purpose, and the list of activities is a constrained list (as is the list of data subject areas), then I can create the name of the service and just hit the infrastructure up for it.  If it exists, the service will respond with details.  If not, the infrastructure can respond with information on what is needed and where it should live.

It really is that simple. 

We go from this:

The catalog describes the service infrastructure (bad)

to this

The catalog is the service infrastructure. (good)

And in this world, the catalog is informative, but not required.

By Nick Malik

Former CIO and present Strategic Architect, Nick Malik is a Seattle based business and technology advisor with over 30 years of professional experience in management, systems, and technology. He is the co-author of the influential paper "Perspectives on Enterprise Architecture" with Dr. Brian Cameron that effectively defined modern Enterprise Architecture practices, and he is frequent speaker at public gatherings on Enterprise Architecture and related topics. He coauthored a book on Visual Storytelling with Martin Sykes and Mark West titled "Stories That Move Mountains".

15 thoughts on “The Unimportant SOA Catalog”
  1. You have a good idea for your periodic table for identifying services to be developed, but this doesn’t remove the need for a catalog/repository. I think you’re trying to be too revolutionary.

    I like your idea of the infrastructure as the catalog, but a service is more than just an interface.

    A repository provides all the service metadata including: SLAs, security, data classification, versioning, vocabulary, profile info, and other useful info.

    A good repository is more than just a list. If your catagory is just a list, then yeah get rid of and make product marketing maintan a word doc.

    And yes, a repository isn’t necessarily IFaP, but that’s ok. It is a tool that allows discovery and governance.

  2. Hi Bruce,

    Your response is altogether fair.  However, if I put that data in the service endpoint, as part of an interface, allowing the services to simply declare that information, and then query it when I need it, then I solve the problem of having two sources of information for service information: the catalog and the service itself, and keeping them up to date.

    Therefore, once we get rid of the ‘ability to find’ the service, we can ask the service itself for the rest of the data according to a shared and common interface / schema.  The information is still needed, but as Harry and Buzz (two other IT architects) like to say, “Truth is at the edge.”

  3. I worked with someone who always referred to catalogs that didn’t work as ‘dogalogs’ – replacing cats with dogs doesn’t make it any more useful

  4. Hi Nick,

    I like your rebellious mind-set. You are very creative, not afraid of kicking holy houses, and above all: you publish your ideas at an understandable level of abstraction. I appreciate your renewal thinking avoiding the pitfall of defining a new world by rehearsing old tricks.

    The idea of the unimportance of a SOA (services) catalog is appealing. I took it the other way around: the importance of a business events (canonical messages) catalog. Here you see the magical inversion of SOA and EDA (http://blogs.msdn.com/nickmalik/archive/2007/06/25/enterprise-it-integration-and-data-security-in-the-canonical-data-model.aspx).

    I think it is not primarily the reusable services we must focus on, but the reusable messages.

    The big trouble is that many think this is to simple to be true…

  5. I completely agree, Jack.  As the service catalog fades in importance, the business event ontology rises to its rightful level of importance.

    Nothing is more necessary than a business event ontology.  

    That said, we don’t need to get agreement on the terms.  We can map the terms.  Its the timing and the contents of the Enterprise Canonical Data Model to the messages that makes this taxonomy so important.

  6. You and Jack are spot on and I just want to add that to make the business event ontology relevant to the different stakeholders accross the enterprise, it must provide semantic relationships thus creating the notion of a Semantic Business Event Ontology.

    Taking a page from best practices in Eneterprise Knowledge Management from the aspect of search metadata, one apprach to do this is to implement the technique of faceted-search to associate business events semantically. Of course, I’m not suggesting a full-blown EKM metamodel-drive, faceted search infrastrcuture…just the concpet of faceted-search and apply it to the Semantic Business Event Ontology notion.

  7. Hi Gabriel,

    Please comment on the distinction between a business event ontology and a semantic business event ontology.  Wouldn’t a business event ontology that didn’t create semantic relationships simple by badly created?

    — Nick

  8. Yup, we are in agreement. I just called it out to be clear that the semantics of business events (ie the semantic association between ontology entities) should be captured in the business event ontology that you are describing. I guessed that you implicitly meant this but wasn’t sure if others did so I thought I’d call it out.

    Because ontologies are geared to domain-specific logical heirarchies, the danger I wanted to highlight is the impression that providing an ontology of business events will get you cross-domain semantics. Including cross-domain semantics of business events will help with cross-domain understanding and usage.

    Think of this as describing business events within context or information about the association between them (as well as metadata such as alias’ and subtypes).

    I’m by no means an ontology expert. Most of what I’m describing comes from experience with knowledge management metadata management challenges and lessons learned.

  9. We have come to a similar position but from a BPM starting point. You can map a service to an activity in a process. The "catalog" is the business process library. Each activity can then have a matching service description (WSDL etc) and can be marked as abstract (defined but not built), concrete (built) or theoretical (not yet defined). Alternatively an activity may just raise an event and have a description of its message (semantics) and be marked similarly.

    PS I assume that your "catalog" can query your services for the "non-functional" service attributes?

  10. @Peter,

    Assumption is correct.  (Theoretical at the moment, though.  Still building in this space).

    — N

  11. Nick, as a person fanatic about systematization I immediately fell in love with your idea about a Periodic Table of Services.

    Could you please make a follow-up post on this and perhaps show an example of what it looks like in practice?

  12. @Andreas,

    Unfortunately, my work requires me to put together some other SOA artifacts before I create the Periodic Table of Services.  I expect to pick up this endeavor again soon and have some traction against it in a month or so.  I will definitely share.  It is an exciting idea.  

    To be honest, I have no real idea if it will work.  I’m just determined as heck to try.  😉

    Thank you for your patience and your support.  I will report back as I discover pitfalls or missed assumptions.

    — Nick

Leave a Reply

Your email address will not be published. Required fields are marked *

four − 3 =

This site uses Akismet to reduce spam. Learn how your comment data is processed.