Coding Dojo Suggestion: Template Method Kata

By |2005-10-28T21:32:00+00:00October 28th, 2005|Enterprise Architecture|

As I’ve blogged before, a Kata is a practice exercise.  I’m attempting to brainstorm out a kata for each of the main GoF design patterns, partly as a mental exercise, and partly as the first step in producing a course that I can give to developers so that they will truly “get” how design patterns can be used.

Template method is a simple pattern, really.  The idea is that the base class provides code for a method that calls abstract methods to do some of the work.  The concrete children implement only the abstract methods.  Users of the concrete methods call the base class implementation which does all the work.

For a kata to work, I need to create a DLL that behaves in a particular way, and a set of unit tests, all of which fail.  The student is responsible for creating a class or multiple classes in such a way that all unit tests pass and any non-testable constraints (like the choice to use a particular pattern) can be verified.

I’m thinking that the base class should have methods for depth-first-search and breadth-first-search, but should delegate the movement from a node to its child to the concrete object.  That way, a developer who creates a tree structure in memory can use either mechanism to search the tree simply by implementing the code to move from one node to another.

If you can think of a better Kata for Template Method, jump in.

Application Adolescence

By |2005-10-28T20:05:00+00:00October 28th, 2005|Enterprise Architecture|

As my kids have grown (I have a 12 year old, a 10 year old, and a 7 year old), it is easy to watch as they become more mature and more independent.  They are growing towards maturity and independence.  Perhaps it is inevitable that I, as an architect, find myself comparing this growth to that of application design.

When my kids were small, I had to tell them everything that was happening, and what they should do.  “It’s raining, Max.  Put on your coat,” or “We need to get ready to leave, or we will be late to see the movie.”  Every step was reminded, every consequence laid out.  They were responsible primarily for listening and doing what we told them to do.

As they’ve grown, my wife and I have worked hard to teach them to look for the consequences themselves.  “If it is bedtime, you don’t need us to remind you to brush your teeth.  You should do it on your own.”  The goal is to have a household that runs pretty much on its own.  They get up when the alarm goes off.  They contribute to the household chores without the need for step-by-step oversight.  They are rewarded for being good household “citizens.”

The other day, I was describing autonomous services to a developer and my mind wandered to thinking about my kids.  It hit me: an autonomous service is like a child that is growing up.  For the service to be truly autonomous, you don’t have to tell it what to do with information.  You just provide the information that it is interested in.  It is responsible for handling it.  If it needs help, it should speak up.

Look at the “normal” design between the application layer and the business object layer as an example.  The app layer creates the business object, tells it to fetch data, updates each field independently, adds any detailed objects into it, and indicates that the data should be saved to the database.  “Collect your homework.  Put on your shoes.  Put on your coat.  Grab your backpack.  Get in the car.  Buckle up.” 

Now look at an automous service.  I am the user interface.  I ask for data about an “invoice” and the service hands me a complete document (Data Transfer Object) filled with the data.  It then goes away.  I know what to do.  I display the data and modify the document with updates.  I then call the service and say “Here’s the document.”  I don’t say “save it in a database, or check for duplicates.”  I don’t sit waiting for error messages.  I’ve checked the data and it is right.  

If there is a problem on the service’s side, it doesn’t tell the me (the user interface).  It copes.  If the database is unavailable, the data is queued.  If the “invoice” doesn’t already exist, the service creates one. 

Errors still happen, of course.  If the user doesn’t have the right to create an invoice, it responds with a “Negative Acknowledgement” and logs the event.    If the invoice is for an unknown product, the invoice may be entered as “partial” and the document could enter a workflow for later resolution. 

The point is that a service doesn’t know, can’t know, that its consumer is a user interface.  It simply has rules, and follows them, without being told every little step.

The point is that the service acts more like an adult.  The document that I give to the service is a statement of fact (“It is raining.”) and not a command (“Put on your coat”).  The document simply provides information.  The service may already know the information, but that’s OK.  The service will be polite and say “thanks.”

That’s how adults behave.

SOA: The data aggregation layer

By |2005-10-19T08:50:00+00:00October 19th, 2005|Enterprise Architecture|

When developing a Service Oriented Business Application (SOBA), you want to seperate your user interface from your services.  Services present chunks of data in a manner that is tied to the underlying business rules, but in order to keep your objects fairly independent, you will want a thin layer that the U/I classes use to call the services.

The aggregation layer is responsible for the following things:

– Knowing that the data is coming from services, instead of local configuration or direct database calls.  The U/I layer should know NOTHING about which service is used.

– Knowing how to compose the objects from various services.  More on this second point below.

The driving reason to create a layer of code here is that you want it to be easy to replace a service.  This allows not only the easy change in business rules, but also changes in underlying systems, to occur independently.  However, to make that work, you may need to partition your services so that one service does not refer to a partner service unless it absolutely must. 

For example, let’s say that I have a service that returns information about an invoice.  I can ask that service for all of the data items on that invoice.  One of the data items is the customer id that I sent that invoice to.  However, the invoice service should NOT return a customer object.  Instead, the data aggregation layer, if it needs to get data about the customer, would turn around and call the customer service, using the customer id returned in the invoice object.

It is perfectly appropriate to optimize this a little if the invoice has some fields that are rightly “customer” fields, like the name of the customer company, the address the invoice was sent to, etc.  In a real sense, an invoice is a legal document, and these are data fields that would appear on the document, and therefore should be kept as part of the invoice anyway.  However, if the front end needs data that would not reasonably be kept with the invoice (like the current list of shipping addresses), then the aggregation layer needs to get that information from a different service.


SOA: Good fences make good neighbors

By |2005-10-17T12:36:00+00:00October 17th, 2005|Enterprise Architecture|

One of the tenants of Service Orientation is that the boundaries are explicit.  Boundaries are usually expressed in the form of the data that is passed back and forth, but I challenge all designers to exceed that simple level of understanding and to make sure that the real boundary is one of responsibility, not data.  Make a service responsible for all cohesive methods that fulfill a single responsibility that can be expressed for the business

Yep… Responsibility Oriented Partitioning is alive and well in the SOA world.

For those of you who think that services solve everything…

By |2005-10-15T10:51:00+00:00October 15th, 2005|Enterprise Architecture|

A few days back, I blogged on “how to design a service.”  It has become increasingly clear that I answered the second question first.  The first question is “why in the world would I want to create a service?”

Isn’t it going to slow down the code?  SOAP is slow, right? 

History lesson:  Why services?

Many years ago (around the time Object Oriented Programming first came out), a real conversation began in software surrounding reusable chunks of code.  Before then, the only rational ways to reuse code was to copy a project from one place to another and modify it, or to integrate (system to system). 

The problem with this kind of reuse is that the granularity is too large.  If you want just a little bit of a systems functionality, you either had to go through a large amount of code to dig it out, or you had to put in a point-to-point integration, requiring changes to TWO systems, with regression testing and integration testing.  It was cost prohibitive.

The “object oriented” conversation was about reusing smaller bits of code.  The proponents claimed that much smaller bits of code, even parts of an algorithm, could be shared.  Different folks could write bits of complimentary code without ever speaking to one another.  And it worked, in a way.  The success of this line of thinking comes out in frameworks like the .Net Framework and, yes, even Java’s class library. 

We got a lot of reuse, and we got a growth in frameworks.  With OO, apps became more flexible, easier to modify.  Finally, the IT department could put out an app in 8 months when previously it took 18!  Wow!

Only problem is: business changes faster than that.  It didn’t matter in 1970, because no one expected computers to really change things, so changes in programs or products were done in the “people-side” and the accounting system was told after the fact.  Over time, however, most companies adopted computers into every function, and that made them more competitive due to removing the inefficiencies in those “people processes.”  Now, however, we are victims of our own success.  Every time we encapsulate a people process in code, it becomes tough to change. 

So the IT department is squeezed.  Where IT departments think of a project lasting 8 months as “short,” business thinks of that timeframe as eternal.  Frustration abounds.  No matter how fast the programmers go, it just isn’t fast enough.  We’ve developed whole new software development processes to speed things up by cutting out some of the documentation requirements, and the time delays for code inspection.  This is all because business needs more speed.

Just look at what we are doing!  Instead of building systems where we only have to change a few people’s jobs (the “people side”), we are building systems where a lot of people have to coordinate a lot of activity very carefully in order to change anything (IT project).

This is systemic.  It is fundamental, and it has nothing at all to do with whether the code is in C# or Java or whatever.  It has nothing to do with the technical details.  The problem is fundamental: how technology is used to solve business problems.

We have to get back to the place where a few business people decide to change things, and they do, without requiring an army of IT workers to make it happen. 

If we seperate the things that change from the things that don’t (Commonality Variability Analysis) we can build our nice OO objects to withstand change.  That works well at the design level, reusing frameworks.  We’ve made it easier for the programmer.  However, if we want to take IT workers completely out of the picture, we need to drive things in data.  That means we need simpler interactions with more constraints, so that the data can actually encapsulate the logic. 

We need “coarse grained” frameworks, where components can be interchanged, one for another, with business users making the connections using configuration data.  The configuration data needs to be in a rich format that covers the connections but does so in a simple way.  (XML is a start).  Simple interfaces are needed to wire up these interactions, interfaces that a business person can use.  I don’t want a business user defining parameters and data fields.  They should simply say “The shipping department needs to send a copy of the completed ship order to the accounting department so that the customer can be billed.”  The system should know who the shipping and accounting departments are, what’s in an order slip, and how to send it. 

Anything less and we have an army of IT workers to make it happen, and that is what is wrong with the system.

We need simple and clean.  We need chunks of logic and functionality, in runtime, that neither know nor care about the interaction logic.  This component, at runtime, that presents functionality but allows the business to change easily, is called a service.  The logical dissonance needed to abstract it and communicate with it is built in to messaging systems like BizTalk.  The abstraction needed to make this easy enough for your average business user to change things… is not there yet.

There is hope.  We are starting to get closer.  We have a common code framework with the .Net Framework. 

We are growing some proprietary business frameworks, in which any enterprise of a particular size and type and legal framework and market expectation can create the correct services, and they will be preconfigured to work.  These are called an ERP systems.  If you are lucky to have enough time, and money, and an army of IT workers, you can get ahead of the game.  Microsoft Dynamics has made some great strides.  We are down from an army to a platoon now.  If you are not a huge company, but want to remove the inefficiency from your systems, take a look.

Large enterprises use SAP.  Same family, different species.  An army is still required for this one.

So, the reason for services isn’t technical.  It isn’t because we want to make things faster or use the hardware better or improve the runtime throughput.  It is because we want to get out of the business of changing every frigging detail of every frigging program when the business decides to do something radical… like make money.

SOA: How to design a service

By |2005-10-07T09:49:00+00:00October 7th, 2005|Enterprise Architecture|

I find myself in the situation of having to set down a design standard for SOA services.

The thing is: SOA Services are much more than a technical artifact.  They are a solution to a business architecture requirement for shared services, and the business architecture needs to exist first, or at least be tacitly understood and described, in order for the service to have any lasting value.

So, first step in designing a SOA Service is to refer to the business process diagrams that your business architect has produced.  For each interaction between roles, look for the data needed to make decisions.  For example, if you are looking at the process flow for a new purchasing agreement, you may see steps where three different collaborating systems need data.  Consider, for each system, what data they have and what data they need to get, from the flow, in order to do their work.  Consider the timeliness of the connection, the direction of the flow (event-push, or call-pull).    

You are collecting data for your service contract.  Before you write the contract, however, you need to write out the “high level service design.”  This design starts with some key design questions that you must answer:

  • What is the responsibility of the service?  This provides the basic description of WHY the service should exist.  Services are expensive.  If you cannot stand in front of an executive and make a purely business-oriented case for the existence of a service, you need to rethink it.  You are doing something wrong.
  • What rules does the service own?  This provides the basic scope of the service.  In a well devised, Enterprise SOA, you will have a rule implemented in a relatively small number of services (hopefully in one service) which makes it easier to change that rule.  This means that you need to describe the collection of rules owned by a service.
  • What style of EAI are you implementing?  I’ve written about trusted services in the past, which is part of the “Messaging” style of loosely coupled services.  The alternative is the “RPC (Remote Procedure Call)” style.  If you are passing a block of self-describing and complete data to a service, and all that returns is “thanks… got it,” then you are using the messaging style.  If you are passing a command (with or without parameters) and are expecting either a set of data in response or an “OK… operation complete” message back, then you are using the RPC style.  Note that RPC style services are more typical but, IMHO, less powerful because they assume a real-time binding between the interacting systems.  In other words, with an RPC style interaction, if either system is unavailable, then the channel is not useful.  With messaging style, both systems do not have to be running at the same time.  Let me be clear on this point: it matters!  If you are presenting an RPC-style service, you are not achieving the full benefits of loose coupling because of the real-time binding between systems.
  • Is the service itempotent?  In other words, if a call to the service is duplicated, and instead of being called once, the service is called twice with identical parameters/payload, will the service detect the duplication and prevent any effects on the underlying systems?  This is very important in messaging style services, but it turns up in RPC services as well.  A service that provides idempotency is more loosely coupled than one that does not, but it also adds to the complexity of the service implementation.
  • What preconditions and postconditions apply to this service?  Just as in use case development, you must be able to describe the factors that must be in place for this service to be used.  Unlike typical use case development, however, you must describe the behavior of the service when these preconditions are not met.  You also must describe the limitations and constraints of the service.  For example, if a service is designed to be used only during a specific business process, then this must be described and included in the service design.
  •  What actors may use this service and how will they be authenticated?  There are two questions here.  First question: define the actors and/or roles that a calling system may play or represent when calling this service.  This is an optimistic statement, because you (a) may not know, and (b) may not want to limit your implementation.  However, you need to consider all of the actors who can use the business rule that you are encapsulating.  If one of those actors cannot use your service, you need to either find a suitable interaction where that actor can use the rule, or create another service that meets that actors needs (I prefer the former). 

The second question above: “how will your actors be authenticated,” must be answered.  Too few people will take this into consideration in service design and that is a huge mistake.  Even within the firewalls of the data center, it is imperative that the communications between systems be understood to be secure from mal-intentioned people.  If your answer is “pray,” then you may want to consider a new line of work.

  • What data elements will be required in order to call the service?  Do not define the format of the calling sequence.  Define the semantics of the data element itself.  Is there an Enterprise-wide Unique identifier for the data item you are submitting?  If there is a numeric value, what does it mean?  At what point in a process is it meaningful? 
  • What data elements will be returned by the service in its acknowledgement / receipt / return?  Answer the same questions as the preceding question.  Think of these questions as the services “data dictionary” but with more constraints.  Data dictionaries describe data a rest.  These points describe data in process.  It’s tougher.

OK.  Now it is time to formalize your service contract.  This is the more technically oriented part of the design.  You will define the rules for interacting with the service.  All of the points above were more concerned with the behavioral aspects of the service.  Now you come to the technical aspects.

The data points you need for the service contract are:

  • What message format will the service use?  Typical answers here include SOAP, Proprietary XML, and various RPC mechanisms.
  • What protocol will the caller use to communicate with the service? Typical answers here include HTTP and various RPC protocols.
  • What level of availability should the caller expect for the service?  Will the service be available 24×7?  Whats the service level agreement that the hosting team is willing to sign up for?
  • What is the expected interaction on the part of the caller when the service is not available?  Should he call again later?  Try again right away?  Discard the call?  Return an error to the calling system?
  • If the service call is part of an expected sequence of service calls, define the sequence.  This may include call-back mechanisms.  Note that call-back mechanisms require the reflected service to be as well defined as the initial service, using all of the rules described here.
  • What latency should the caller expect for the synchronous return from the service call?  (In messaging systems, this is the time that the caller should expect to wait for the acknowledgement or receipt.)

These are the technical aspects of the service.

Once you have defined all of these points, you have the high level design of your service.  You need to publish the service contract and all of the rest of the design elements above in a place where the calling systems (and their programmers / administrators / support teams) can get access to them.

And that’s how you design a SOA Service.

Futurist wanted

By |2005-10-06T03:09:00+00:00October 6th, 2005|Enterprise Architecture|

I have the best job in the world.  I am paid to think about the future.

Not in the sense of writers for Futurist magazine, or Science Fiction, but in the sense of the “next” Microsoft… the one that will be born out of the success of the coming product wave, and the ability of the company to capture new markets. 

I think about that new Microsoft, and the capabilities that the systems need to have to empower it, and I write down that “future” and deliver it.  That’s my job.

Dang I love this job.

If you have what it takes to be a futurist (aka Enterprise Architect), send me an e-mail using this site.  We are looking for good people.  In addition to all the good descriptions, I’ll add one condition… you must not be afraid of the future.

Measuring EAI Maturity

By |2005-10-04T17:20:00+00:00October 4th, 2005|Enterprise Architecture|

Hohpe and Woolf identify four basic integration styles.  I am considering the enterprise as a whole and asking “what is the right mix?”  In other words, in an ideal world, how much of the data should move by file feeds, how much through shared databases, how much through RPC and how much through messaging?

What leads me here?  There are dozens of applications in the area that I work in.  I’d say that if you measure the EAI mechanism by virtue of the amount of data moved, file feeds win.  On the other hand, I don’t consider that to be a good measurement, because most of that data is domain data, which really should move through file feeds.  So, if I measure the connections between “logical” systems as “channels” and I literally classify and count the channels, I’d estimate that 80% of the connections are flat-file feeds, 10% are RPC (including web services), 5% are shared database, and 5% are messaging.

So what are the correct numbers?  Flat file feeds are not cheap, but neither are web services.  You have to have good subscription management with everything except shared database (where subscription management is often not done).  You have to have good monitoring with all four.  Cost these out.  I find their costs to be quite similar, and that these aspects account for the majority of the cost of maintaining an EAI infrastructure.

So it isn’t really a matter of pure day-to-day operational cost.  I don’t think I can use opereational cost to justify moving from one EAI style to another.  The strengths and weaknesses of each style are in other areas.  Things like operational efficiency, network load, data velocity, brittleness, and adaptability. 

So do we say that large data sets of data that change no more frequently than daily should ALWAYS be in a shared database or a flat file feed?  Do we exclude this kind of shared data from web service or Messaging infrastructures?  If so, why?  If not, why not?  I can make an argument for and against, but I don’t have numbers to back me up. 

I’m so tempted to say that this is the point where “art” replaces “science.”  However, I have promised myself, and my customers, to resist flippant answers like that and keep working on the “magic formula” that will help determine how we can measure our EAI maturity.  The goal: know what “good” looks like.