Taking your name off of a project

By |2006-12-22T10:22:00+00:00December 22nd, 2006|Enterprise Architecture|

You cannot often choose the projects that you work on.  However, the ones you work on should reflect your principles, your asthetic sensibilities, even with all the compromises that necessarily have to take place between conception and production.

But when does it get to the point where, like Daniel Libeskind (who removed his name from the Freedom Tower project), you are prepared to take your name off of a project whose demands and compromises alter the design so substantially that it is not your creation any more?

This is certainly not new.  In the film industry, for many years, when a director wanted to take his name off of a film, it was credited to the psuedonum “Alan Smithee.”  That name has racked up quite a career! (see Wikipedia article on Alan Smithee).

So, what is the line that you won’t cross? 

Perhaps we don’t have enough software projects where the architect is well known.  Does anyone care who the architect is?  Honestly, who remembers the name of the architect of the colosseum in Rome?  What about the architect for your home… do you know his name? 

On the other hand, I can easily imagine a day where software architects compete as individuals, just as building architects and movie directors do, for the right to work on a project of high visibility and importance.

And I can easily imagine a day when a software architect will want to take his name off of the system. 

Internet-wide Services Integration Architecture needed now!

By |2006-12-20T10:32:00+00:00December 20th, 2006|Enterprise Architecture|

As I mentioned in a prior blog entry, the lack of a single consensus mechanism for different Software-as-a-Service apps to integrate with each other and with enterprise-oriented software applications (like SAP, Dynamics, Baan, Siebel, Oracle, Clarify and others) is a clear obstacle to the success of Software as a Service.

I would say that it has always been an obstacle to the success of the software packages themselves, but the SaaS viewpoint adds more fuel to the fire.

It is expensive and time consuming to create point-integration between every application and every other application.  In Enterprise Architecture, we attempt to address this by creating a canonical mechanism for communicating a single business event, and a shared enterprise service that serves as the interface for that event. 

In the Internet, we need the same thing.  We need the ability for a SaaS app to generate an event that an internal SAP instance can consume, including the ability for the SAP instance to call back to the SaaS app to get further data about the event or the data owned by that app.

Of course, if the SaaS vendor is Dynamics Live or Salesforce, then the SAP company has a conflict of interest.  After all, they have their own CRM solution.  Why would they want to integrate with a service-based solution and potentially create more competition for themselves.  Dynamics faces the same conundrum with respect to Dynamics AX and Salesforce.

To overcome this conflict, it is imperative that we begin, now, to embark on a new approach.  We need a single canonical mechanism for all enterprise app modules to integrate with each other.  If done correctly, each enterprise will be able to pick and choose modules from different vendors and the integration will be smooth and relatively painless. 

Why?  because if SAP uses ‘Mechanism X’ to communicate between their financials and CRM, and Dynamics uses ‘Mechanism X’ as well, then an organization can move from using Dynamics for both modules to using Dynamics for one and SAP for the other, or the other way around, readily. 

App developers: Think of it this way: The SQL Server management environment has been using a published API to talk to SQL Server for a long time.  (DMO).  Everyone knows that the API works, because SQL tools use it. Therefore, if you want to create a tool for SQL, just write to DMO.  You can replace one component and leave the other, because the interface is public and defended.

I would like to see a body be created to develop the public interfaces between the major functions in enterprise software, that the body doesn’t move forward until a majority of major players in enterprise software sign up (including all three of these: Microsoft, SAP, and Oracle) and commit to supporting it, and using the interface for their own product modules to speak to one another.  The interface has to allow for SaaS modules as well as installed modules to both produce and consume events and data.

The lack of this consensus creates unnecessary inefficiency, vendor lock-in, and huge levels of waste in the IT business. 

It’s the elephant in the room, folks.  Rosetta is a great start but they didn’t have buy-in that the products would change to match the interface.  In effect, they created an expensive ‘bag’ to attach to the side of any product, but nothing that the products themselves were committed to adopting to communicate between the modules internally. 

Perhaps we need to threaten with legislation… hmmm…

The roadblock to Software As A Service

By |2006-12-19T17:16:00+00:00December 19th, 2006|Enterprise Architecture|

Prediction for 2007: The market for Software as a Service is going to peak soon, and then fall off.  In a year, existing players will stabilize and consolodate and one or two players will be profitable, while the rest fall away.

The promise of Software as a Service (SaaS) is the same as that of the web: no more installation and management of client-pc software.  You could deliver the software as a service, pay for it when you use it, and cancel your subscription at any time.

So SaaS vendors (a.k.a. Application Service Providers) provided a series of hosted solutions.  The commercial public was expected to sign up, relinquish the desktop app, and agree to be billed for using word processing, e-mail, financials, customer service, and other systems.

Many folks have.  Goodness only knows that SalesForce.com is a major player in CRM software, so much so that Microsoft has gone directly into competition with Dynamics Live. 

The problem is that you get the user interface along with the package, whether you like it or not.  Along with that, you may, or may not, get a proprietary interface for integration with your corporate systems.  That limits the ability of the organization to truly integrate the ‘service’ tool with their infrastructure. 

Why does this matter?  Because the advantages of SaaS are that it is easy to implement and that you can quit at any time, which puts pressure on the provider to do a good job and constantly innovate.  However, if you cannot quit at any time, because your integration layer doesn’t work with the competitor’s service offering, then your service provider has no incentive to improve. 

In addition, any solution, these days, that doesn’t come with ‘out-of-the-box’ integration between major business functions, like financials to CRM integration, is dead on arrival.  However, you cannot expect all financial system vendors to work to integrate with all CRM service-providers.  That would be complex and expensive.  This is no better than expecting the end-customers to figure it out.

What is needed is a standard, demanded by the consumers of the service, that all service providers in a field MUST sign up for.  To do that, we need a consistent architecture, across ALL SaaS vendors, so that they can write to a defensible interface and so that a SaaS vendor can integrate directly with another SaaS vendor, allowing a company to outsource both CRM and Financials yet have them work together.

That means Architecture at the Internet level.  Architecture that vendors sign up for.  OASIS is a great place to put it, but the TC for this need doesn’t exist there, at least not yet.  Doesn’t have to be OASIS.  Could be any other group willing to drive the standard architecture that allows Software as a Service to integrate across the corporate landscape.

Either we do this as a community or one vendor will do it and force it on the rest of us.  Personally, I like community.


Declaring architecture in the code

By |2006-12-18T03:44:00+00:00December 18th, 2006|Enterprise Architecture|

Code sits below the architecture.  It is not an expression of architecture.  Code realizes architecture, but is constrained by it.  Therefore, it is entirely possible to declare the architecture in the code.

For example, let’s say that we are building a distributed system composed of a user interface and four services.  The user interface calls the services only.  One of the services recognizes a business event and transmits it to an EAI system for distribution to subscribers.

The services, architecturally, should remain isolated from one another.  We want to declare that… to constrain our code.  We also want to make sure that the user interface is not dependent upon any interface to the services except a declared public services interface.  This dependency needs to be declared and enforced.  In other words, no other dependency is allowed, especially the sharing of class across the service boundary.

So how would we declare these constraints in our code?  I would prefer to do this visually, because I don’t think that architecture can or should be described in some distorted text typical of most 3GL programming languages.  I have no problem if the visual image has an XML representation, or some other computer-readable standardized mechanism, but the developer should access the architecture visually.

In addition, the architecture should be a starting place for the code.  I would envision that when a developer opens the code for his app, he sees the diagram and uses it to navigate through the project or projects.  Since this is a distributed system, compilation of each component must be fairly seperate, since each component could potentially reside in different systems.

Unfortunately, it is 1am and I don’t have the patience for writing up the diagram at the moment.  I’ll slap together an example tomorrow.  I envision the code representation to look something like this (note: this is a revision of this post.  My first attempt was in XML but in hindsight, that wouldn’t be part of the code, it would be part of the IDE or config, so I rewrote to something that looks like C#):

architecture SCRIBE // system name is SCRIBE in this example
      endpoint CustomerEnterpriseService;
      endpoint InvoiceEnterpriseService;

      //by declaring DataLayer to be InProcess, that means that I intend
      // to have the users of this component call it directly in a DLL rather
      // than through service calls.  This is important to SOA.  
     component DataLayer(InProcess)
          layer DataLayer;
          DataLayer.namespace = “contoso.scribe.service.data”;

      component UserInterfaceModule
           layer front, service, composition;
           front.namespace = “contoso.scribe.interface.front”;
           service.namespace = “contoso.scribe.service”;
           composition.namespace = “contoso.scribe.interface.proxy”;

      component CustomerServiceModule 
                       delivers CustomerEnterpriseService
           layer facade, business;
           facade.namespace = “contoso.scribe.service.customer.facade”;
           business.namespace = “contoso.scribe.service.customer.business”;

      component CustomerServiceModule 
                       delivers  InvoiceEnterpriseService
           layer facade, business;
           facade.namespace = “contoso.scribe.service.invoice.facade”;
           business.namespace = “contoso.scribe.service.invoice.business”;

In this example, wildly oversimplistic but illustrative, I have described two services and a user interface layer.  The u/i layer declares its composition layer and the fact that it is calling services.  (.channels.add)  The interactions can then be limited.  If a bit of code in the interface were to directly call code in ‘contoso.scribe.service.data’, then the system could warn the developer that the interface layer can only call the composition layer or the service layer.  The service layer can call nothing else.  The composition layer is allowed to interact using a service model.

Perhaps this ties in to Domain Specific Languages somewhat.  My problem is the notion of considering this to be something seperate from the C# or VB.Net languages themselves. 

We keep building things AROUND our programming languages, but I think we should not omit the potential gains by building WITHIN them as well, so that our compilers can warn a developer when they exceed the architecture. 

It also means that the architect can do a code review on a single document, the declared architecture, and feel reasonably assured that the document that he or she is looking at actually reflects the intent of the system, because the compiler is working for the architect, to enforce and reassert the architectural intentions if the developers make a mistake.

A SOA Infrastructure Reference Model

By |2006-12-14T20:58:00+00:00December 14th, 2006|Enterprise Architecture|

I realized that I had been using a reference model, in my head, for a long time when describing the ‘moving parts’ in a Service Oriented Architecture.  It was time to put those items into a diagram and share it around.  So, here’s the current diagram.

In this diagram, I illustrate the distinctions between:

Transitive Transaction Store and Services

Domain Data Management, Repository, and Services

Thin Consumer, Rich Consumer, Extended Consumer, Proxy Consumer

Floating Services

Large Object Messaging Service

I don’t go into a lot of detail outside the diagram (here in the blog post).  I may add another post to follow up.  Comments are very welcome. (click the thumbnail view to enlarge)


Pay Attention to the Application Portfolio

By |2006-12-14T09:40:00+00:00December 14th, 2006|Enterprise Architecture|

One nice thing about being in Enterprise Architecture: you get to watch as the infrastructure changes.  When you are working on one big project or another, or even a dozen different projects over the course of the year, you only see the changes that you helped to create, but in EA, we have to take a regular inventory.  It is then when you realize all the things that have actually happened the year before.

And it is then that you sometimes find that the data about your application portfolio doesn’t match reality. 

In our area, we had a small exercise last fall where we were classifying the applications according to whether they were Strategic, Core, Maintain, or Sunset.  It’s a part of Application Portfolio Management called Application Segmentation.

The practice allows the organization to inspect the investments across the portfolio (Project Portfolio Management) to see how many of the investments, or how much of the money, is going towards strategic needs vs. core needs vs. maintenance, etc.  You could even make portfolio decisions, like “of our total budget, we want to spend X% on strategic needs” which could move a few of the projects into, or out of, the Plan of Record (POR).

Anyway, while doing this exercise, one of my friends in a different group noticed that one of the “apps” in the portfolio wasn’t really an app at all.  The PM who had created the entry had been working on a project and thought that “all projects need their own application id” so he created one, even though the project didn’t create a new app… it modified one.

So my friend did the rational thing: he asked that the ‘entry’ be removed.  Our support manager agreed and the application entry was removed.  Unfortunately, I wasn’t aware of this. 

Now, the end of the year rolls around, and I’m taking my annual inventory and updating models, and I notice that the app is marked as ‘retired.’  I know that the functionality is still there.  Few people remember why the app was ‘turned off’ and my friend’s name is not on the entry’s log anywhere.  It took about an hour to figure out what was going on.  Once I realized what he had asked for, I agreed with the logic and removed the entries from my models. 

Point being: Enterprise Architecture is intimately tied to Application Portfolio Management as well as Project Portfolio Management.  Make sure that your processes for review and governance include process points when an application is decommissioned, not just when it is rolled out.  Otherwise, you, too, will spend a few hours, at some later date, trying to track down “what the heck happened to the SNAFU app?”

Requirements for an Enterprise Service

By |2006-12-12T09:35:00+00:00December 12th, 2006|Enterprise Architecture|

An enterprise SOA service is not just any old web service.  There are specific requirements that it must meet.  These are not optional, yet I have not been able to find many resources on the web describing these requirements.  (Reasons = different post).

In general, if you are developing a service that you want other folks to consume, you must follow these high level principles. 

  1. Your service must be simple to call.  With a coarse-grained service, there can be a lot of data to pass, and that data may need to be fairly complete.  The receiving end may need to perform quite a bit of validation on that data.  One tactic that I’ve seen to ‘optimize’ this is to write a DLL that creates the data and calls the service.  Then you just ship the DLL to your customers.  This is nuts. 
    • You customers still have a call-level interface.  Instead of it being the service, it’s the DLL.  Nothing gained.
    • EAI components now need an adapter to call your service.  Something lost.
    • Your service, if called from an external or cross-platform environment, it is not being called by your DLL.  In fact, you cannot really be sure that the service is called by your DLL even inside your environment… so you will have to validate the data anyway.  This means that the data, when it is called by your DLL, is validated twice.  This is a performance hit.  Do it once… in the service.
  2. Your service must be able to be called directly from EAI components, like Biztalk or your favorite ESB, without engaging an expensive or custom adapter.  EAI components have no trouble reading WSDL.  Requiring an adapter means that you don’t have a truly sharable service.
    • Heavily nested or embedded XML structures are unweildy in most tools.  Therefore, the fact that you can do something in XML, doesn’t mean that you should.  Avoid the creation of the “universal message” that can be extended 1,000 different ways.  Go ahead and create messages for specific business purposes.  Keep the vocabulary of messages from exploding by using optional attributes and structures… that’s fine.  But if you create an XSD that refers to 10 other XSD files, you’ve got a structure that no human can understand, much less code to.
    • Requiring some non-standard encryption to take place or some special mechanism to ‘encode’ the message, to make sure that it is coming from a known source, forces your EAI tool to use an adapter.  It also gets in the way of flexibility.  To remain flexible, you need the message to come from anyone without changing the code.  It is perfectly appropriate to encrypt the message or to sign the message.  There are standards for that.  Use them. 
  3. If you do want to make a DLL available for folks to call your service, perhaps to reduce the barriers to adoption, do so with open source.  If the developers using your code can see the code, and change it, on their side, then you have done them a favor.  If you simply substitute your web service call for an API call, you’ve done nothing positive at all.
  4. Services running at the Enterprise Level must have methods that can be called to validate (a) that the service is running, and (b) that downstream connections are valid.  Therefore, I suggest that every service should have standard methods for “ping” and “check.”  Note that, for services that have no downstream connections, they both do the same thing.  I’d suggest that the return for the call is simple connection information to the service (perhaps the URL of the service itself).
  5. Services running at the Enterprise Level should provide data about what servers, data centers, and support teams they are associated with, live and in real time.  Therefore, I suggest that ever service should have a standard method for “GetSupportData” that returns a structure containing all the info. 
  6. Services running at the Enterprise Level should NORMALLY be asychronous, with a very rapid return of an acknowledgement that indicates that the service itself takes ownership and responsibility for the message. 
  7. The sending side must place a unique id into every call.  For async message exchange, the receiving side must send back the caller’s unique id, along with any unique id it has created for it’s work.  This allows message correlation. 
  8. The call-back structure, for calls that require a later return, must be standard, allowing any caller to easily construct the data needed for the receiving system to understand how to call the sender back with the response package. 

Of course, putting requirements like this on Enterprise Services means that not every service can partake.  On the other hand, it means that every service in the Enterprise catalog can be verified automatically, and basic information can be collected and reported.  You can go further, of course, and place a great deal of stuff in a standard interface, but this is the basic set that I would start with.

Encouraging Enterprise Services

By |2006-12-11T09:46:00+00:00December 11th, 2006|Enterprise Architecture|

  1. Enterprise Services save money.  When done well, they create discoverable, supported, consistent, and reusable end-points for functionality. 
  2. Enterprise Services cost money.  They need constant monitoring, upkeep, maintained standards, and systems of financial connectedness that the business is not used to providing.  These costs can overwhelm the benefits of some services, but other services can clearly pay for themselves.

If you accept both statements, then there is an obvious conclusion: we must encourage services to develop as described in the first statement, while ferreting out services that are too expensive to own or incapable of playing nice, as described in the second statement.  This means that some services should exist, while others should not.

Fascinating problem, because a lot of the discussion around the generation of SOA architectures has focused specifically on the first statement.  However, an unbalance approach to SOA will generate many of the services that it should not develop.

I want to add a third statement: Services that are ‘correct’ today may be ‘incorrect’ tomorrow.  Here’s why:

In the early days of retail, and still today in the smallest of towns, a single store provided for all the needs of the community.  In the USA, the “general store” had already been replaced with boutique retail as of the late 70’s when Walmart started to emerge.  We’ve since swung back, with the small retail “downtown” of most towns going quiet while the Walmart on the edge of town draws away the business.  Whatever you have to say about a disappearing lifestyle, or the poor wages offered by these megastores, the fact remains that retail goods are distributed more efficiently and at a lower cost now than ever before.

The point is that a model of service (rural retail) that was appropriate at one time is not appropriate at another.  Just as retail has moved through this cycle over time, without central planning, we should expect that some areas of our SOA will do the same over time.  There will be cycles between the use of centralized services and the use of distributed (boutique) services. 

Right now, this is not possible.  Right now, most enterprises have very nascent models that either rely deliberately on one or the other.  That deliberate voice comes with a price: the price of control.

So, my third statement goes like this:

  1. Enterprise Services need to have a model where they may NATURALLY migrate from central to distributed and back without the central architects either driving or resisting the change.

Note that the city planning of most small towns did not change during the time between distributed retail and central retail.  The investment by a clever company changed the landscape, to be sure, but the city planners neither planned for Walmart, nor did they prevent Walmart (in most cases).  The growth was natural, not centrally planned, and organic.  The result was displacement, but also efficiency.

So, we need models for details that we haven’t really thought about:

  • How do you “lay off” a service? 
  • How do you determine that a service is not worth keeping open?
  • How do you compete in a services marketplace, to get consumers to come to you (we are talking IT SOA here).
  • How do you create a ‘data supply chain’ that makes the distribution of data efficient and well run?
  • When is it the right time to create a ‘boundary’ that encourages very specific services (think “boutique retailers”) while discouraging consolodated service providers (like a “Walmart”) from providing those services.

In city planning, any company wishing to provide services to the public has to apply for a permit.  The rules for permitting are complex, and often a ‘zoning board’ will set rules that the permitting process needs to respect.  This level of sophistication needs to be reached in IT, and fast, if we are to create SOA governance that hits on all three statements above.  Otherwise, we risk creating the situation where we either stifle services, or create a rigid set of services that cannot adapt over time.  That isn’t better than COM.  It’s worse.

Should SOA be Top Down or Bottom Up

By |2006-12-09T12:52:00+00:00December 9th, 2006|Enterprise Architecture|

It’s the age of the mash-up and mix-in and composed service… yet I continue to wonder if we shouldn’t still be developing the SOA, at least seeding the initial SOA framework, in a top-down way. Just as the Internet blossomed only after the standards were created, and a mechanism evolved to create decentralized ‘control’, SOA still needs the standards to be set… not just technical, but policy, payment, and support.

In any sufficiently large organization, it is impossible to imagine a SOA emerging where the systems of one part of the business wouldn’t need to call the services developed in another.  That’s good.  Who pay for supporting those calls?  Until the question is answered, you have a roadblock to SOA adoption.

I support a part of the business that brings in about $1B per month.  I work with another part of the business that brings in far more.  But if I want to integrate with services developed by other parts of the business, I will get asked questions, valid ones, not only about cost… Questions include:

— if a service goes down and a consumer application is thus affected, how do we get the support staff for the consumer application to quickly identify the problem?  If that service is delivered by another support team, how do they make sure that the service is brought back up and that the consumer app is then diagnosed for lingering effects (like dropped or incomplete transactions).

— if a new consumer wires up to a service, and that linkage causes the service to be overwhelmed with calls, how do we charge back the cost of scaling that service up?  What if the service cannot be scaled up?  What if the architect says your app MUST use the service, but the cost to upgrade it is prohibitive?

I think that architecture has to address a number of issues: standards, economic models, support models, service level agreements, tracking and managing of end-to-end transactions, data consistency, reliability, throughput, performance, compatability with EAI systems like Biztalk, and more.  I don’t believe this is possible without some top-down constraints (a.k.a. Internet Standards) that place some requirements on a service that “may” be shared.

While the architects do not need to know about every moving part, they DO need to be aware of the largest of those parts, and make sure that they are managed well. 

This is similar to city planning, where the city needs to work with a large employer or a large retailer (like Walmart) to make sure that roads and parking and congestion issues are managed, without having to worry about cafe and card shop that are also employers, but have a minimal impact on the infrastructure.

So top down or bottom up?  I say: standards are top down, as are the connections between large systems.  The rest is bottom up in a fair and free manner.