/2004

C#: a way to get around the lack of multiple implementation inheritance

By |2004-12-31T14:03:00+00:00December 31st, 2004|Enterprise Architecture|

I run across this question from time to time: why is there no multiple inheritance in C# like there was in C++.  Personally, I’ve never needed it, but I do see a value to it, and there are some times when it would appear to be handy.

There is a workaround to this problem that is not difficult to do.  You get some of the same abilities as multiple inheritance, with a few structural advantages.  Before I describe the solution (below), let me frame the problem so that we are all using the same terms.

We have two concrete classes.  They are both derived from different base classes (not interfaces).  You want to give both of them a set of common methods and properties that meets a defined interface or base class (multiple inheritance).

Note: if you just want to inherit from an interface and implement the properties and methods directly in the class, you can do that now.  That does not require a workaround.  In other words, it is perfectly acceptable to do this:

   public class MyClass : System.Web.UI.Page , MyNewInterface
   { … }

So the problem only really arises if you have two or more BASE CLASSES that you want to put on that line… something you cannot do in C#.  I will treat one base class as “given” and one as “add-on”.  It really doesn’t matter, structurally, which one is which.  There is only one “given” class.  There can be as many “add-on” base classes as you want.

So, you use a simple composition pattern (that I cannot find the name for… if someone knows the name, please send it to me).

Step 1) you need an interface.  This defines a single getter property with a name derived from the base class name of the class you want to add on.

interface IAddOn
{
    // define a getter to return one “AddOnClass” object
    AddOnClass GetAddOn
    {   get;   }
}

Step 2) insure that your concrete object inherits from IAddOn

public class MyMultiConcrete : MyBaseClass, IAddOn
{ …. }

Step 3) Create a factory object that will return a class of type ‘AddOnClass’  (optional, but good practice). 

public class AddOnFactory
{
   public static AddOnClass NewAddOnObject()
   {   return new AddOnClass();   // factory method
    }

}

[edited] I want to add one comment here.  You don’t have to return a type ‘AddOnClass.’  In fact, if the add on class is an abstract class, you cannot.  You would need to derive a class from AddOnClass and then instantiate one of those types.  If you created this class specifically to be called from your new type, then you have a pair of classes that work together.  The derived add-on has access to the private and protected members of the add on type. 

In this case, you can pass in a reference to the calling class:

    public static AddOnClass NewAddOnObject(IAddOn Caller)
    {   return new ConcreteAddOnClass(Caller);   // factory method }

This gives the concrete add on the ability to directly manipulate the ‘container’ when you call a property or method on it, as shown below.  [end edit]

Step 4) Declare, in your concrete classes, a private property to hold the reference:
      private AddOnClass _AddOn;

Step 5) In the constructor for your concrete class, call your factory object to return an object of type AddOnClass.  Assign the reference to the private _AddOn property.

   public MyMultiConcrete() : base()
   {    // do normal constructor stuff here…
         _AddOn = AddOnFactory.NewAddOnObject();
   }

Step 6) Define the property that returns the add-on object
     public property AddOnClass GetAddOn
     {   Get { return _AddOn; } }

/// you are done ///

Now, every place in  your calling code, where someone needs a method or
property from the add-on type, they will reference it this way:
   MyMultiConcrete.GetAddOn.MyAddOnMethod()

One nice thing to consider, we can do this for as many types as we want within a class.  Therefore, we could theoretically inherit from dozens of base classes. 

I hope you have as much fun using this pattern as I have had describing it.  I doubt that I’m the first person to identify this pattern, so if someone can send me a link to another name or description, I will be grateful.  If not, perhaps I’ll go to ChiliPLoP and present it :-).

How is workflow different from a Finite State Automata?

By |2004-12-28T13:35:00+00:00December 28th, 2004|Enterprise Architecture|

After showing a workflow diagram to a co-worker, he asked me if I could tell him how this is any different from basic Finite State Automata (FSA).  To be honest, I had to think about it for a few minutes to get my thoughts around this, but there is a fairly big difference.

For those of you who aren’t familiar with FSA theory, this is a segment of computer science that goes back to the earliest days of computing.  The idea is this: arrange your input into a stream of tokens of the same size.  Then, keeping a state, read each token.  The token will dictate the state you move to next.  Side effects could be added to a transition.  These side effects, taken together, were the actual functional code of the system.

In a strange way, we’ve all moved to FSA programming when we moved to event driven application programming.  The event handler is essentially a place to put logic for reacting to an input (the event) in a state (the global environment).  It’s a bit different, though, in the sense that our input isn’t in a stream.  We can’t look ahead at the next token or move in the other direction over a series of tokens (important parts of compilier design).

In that respect, Workflow modeling is closer to event driven programming than it is to FSA theory, because we don’t have that input stream.  We can’t look ahead. 

On the other hand, unlike event driven programming, most workflow systems use the FSA approach to modelling, where you look at the behavior of the system by creating a graph showing the stages of work, and the transitions from stage to stage. 

However, what really distinguishes Finite State Automata from workflow programming, in my mind, are the three layers of abstraction inherent in Workflow analysis. 

Finite State Automaton development requires a pre-processing step, where you take the input and interpret it as a series of tokens in a language.  In compiler theory, we call this lexical analysis.  This analysis happens at only one layer of abstraction (usually at a very low level: character sequences).  Therefore, the structure of computer languages has to be represented as a parse tree: a complicated heirarchical structure that “represents” the analyzed token stream.  The FSA is done when the parse tree is done.  It isn’t involved in actually using that tree to create the target code.

With workflow analysis, there are three layers of abstraction: Business unit level, Business process level, and Workstep level.  All three are distinct (but related).  All can be described with rules and constraints.  All have a specific purpose, and each one can be represented as a state graph.  The mathematics are considerably more complex.  (Only the lowest level has to be deterministic).   There are many PhD level practitioners of workflow modeling who can attest to the fact that workflow is much more complicated and complex than the fundamental concept of a Finite State Automaton.

Why?

Because FSA’s interpret logic… nothing more.

Workflow modeling deals with human behavior.

On XML Models of Process

By |2004-12-27T09:56:00+00:00December 27th, 2004|Enterprise Architecture|

XML is an interesting language, but is it a useful one for describing a process?

We have multiple competing standards for workflow and collaboration.  We have BPEL, XPDL, SWFL, XRL, XScufl, and custom XML workflow models developed for the XFlow, AntFlow, Agile, YAWL, and OpenWFE tools.  (If anyone is looking for a good idea for a masters thesis in Workflow, they should create a comparison of these different languages, catalog features, and create a roadmap for the rest of us).

Just to add to the fun, rather than learn an existing modelling language, I wrote my own for an internal tool I’m working on.  Wise?  Probably not.  In keeping with the philosophy of the project?  Yes.  Most of the languages I mention above are the creation of committees and have many features designed for multiple vendors to extend the core set.  I needed less (features demanded by existing Java projects) and more (features specific to my Microsoft-based solution).

I also needed a language feature that I didn’t see anywhere else, including on the workflow patterns homepage: native support for ad-hoc workflow.  This means allowing a user the right to change the routing rules in the middle of a workflow process, while the engine maintains managability.  No mean feat. 

So, inspired by YAWL, and angry at the limitations of the competing partner products that we evaluated, our team wrote another XML based workflow model structure. 

I learned a few things that I want to share:

  1. XML is an interesting language, but not a forgiving one.  It is easy to create bugs by making small errors in the specification of a schema, where the downstream ripples can be quite large.  If I had to do this all again, I’d better appreciate the time it takes to create and debug the schema itself.
  2. I am far from the first person to tackle the idea of Workflow.  Perhaps it would have been better to start with XPDL (or a subset thereof).  My customers would have a better ability to switch away from my project later, which is one of the stated goals of the project.  I, on the other hand, could have leveraged the built-in workflow experience that comes from leveraging a schema that comes from workflow experts.
  3. XML is an old-fashioned declarative language.  It is about as advanced as C (not C# or Java).  Therefore, while there are many things you can do in XML, you have the freedom to do some pretty stupid stuff.  In addition, you don’t have the constructs to do some elegant stuff.  By comparison, XML is an infant.  The effect: the resulting code is difficult for a human being to create, read, follow, analyze, debug, test, or support.
  4. XML parsers are touchy.  They remind me of Fortran-77 compilers.  First error and they are done.  You can’t count on an error message from a parser to be all that helpful. 
  5. Tools for generating XML are new, but getting better.  Two commercial tools worth mentioning: Microsoft Infopath (the most underrated, creative, well-built, xml forms system I’ve seen), and Altova Stylevision (an interesting product that suffers primarily from the lack of imagination of its original designers, not the details of the implementation).  Add Visual Studio for Schema generation and you have almost everything you need.
  6. Automatic mapping between XML and databases: a new and immature field.  The current bits in SQL Server 2000 are OK, but I’m looking forward to better capabilities in Yukon and other tools.  Right now, I wouldn’t count on using automatically generated or automatically parsed XML as a way of reducing struggle and pain on a development project.  You will only replace one kind of agony with another.
  7. Like any code-based method, process declaration in XML inherently describes only one aspect of a process: the static connectivity between pre-declared states.  The dynamic aspect is not well described or modeled when you focus on the static.  Some folks have tried to focus on a dynamic model exclusively, but the resulting description was even harder to understand (refer: Biztalk HWS).  In other words, the model, in XML, isn’t clean enough to provide to business users.  A LOT of translation is required.  XSLT comes in very handy.
  8. Even with these drawbacks, I can’t imagine a better way.

So, XML it is.   And for now, I’m still using my proprietary schema for workflow models.  Perhaps, someday, I will switch over to BPEL or XPDL.  But not this day.

steps to transition a client server VB6 app to .Net

By |2004-12-26T22:46:00+00:00December 26th, 2004|Enterprise Architecture|

Every now and again, I get a question that I’ve seen many times before.  I saw this one recently and thought I’d record the answer, so I can refer to this in the future.

Question: Our company develops and markets a client/server application which is
written in VB6 as a rich Win32 Client/Server application. For a variety of
technological reasons we are looking to migrate toward the .NET environment.  How can we best migrate our VB6 app to the .Net.  We are planning to add a new feature.  Can we plug in the new feature into existing code and maintain a seamless U/I.

Answer:

When it is time to add a new module, this may not be the best time to move
to .net.  I guess it depends on how independent the new module is.  If you need a new module, see step 4. 

  • Step 1: Spend a few days getting used to your code again.  Like an old
    friend that you’ve lost touch with, code tends to deviate from the design
    during development and maintenance.  Take a fresh look.  Consider this
    question:
    If I was to write an altogether new interface, could I use the same objects
    I currently have?  Would I?
  • Step 2: refactor the existing code, in the existing language.  Clean up any
    of the partitioning problems you saw in step 1.  Some things will still
    creep up in the development later, but taking a little time now to clean up
    the objects can really make a difference in your ability to deliver on a
    predictable basis.
  • Step 3: Create an ENTIRELY NEW USER INTERFACE.  This is not for the faint of
    heart.  However, VB6 has a few advantages that your ordinary application
    doesn’t have.  For one thing, all of it’s objects comply with the COM
    interface.  Therefore, you can create an entirely new interface in .NET
    without actually removing the old one.  You can pop back and forth, validate
    functionality, and even refactor the old code while writing the new
    interface.  Here is where the work done in Step 2 really pays off.  If your
    business object layer was actually designed to be independent of the U/I
    layer, you will appreciate it today.  If not, you will need to make it
    independent.  There is no way around it.
  • Step 4: retire the old user interface and release the code to business users
    or beta testers.  It takes time to create the new interface.  Someone always
    adds an “innovation” or two.  Start collecting feedback now.  Depending on
    your business, it is perfectly acceptable to actually release the code at
    this stage.  If there are new features to add, do it using the OLD code base
    and the OLD language, called from the new interface.  In other words, keep
    the layers intact.
  • Step 5: Create a parallel data access layer.  This will be used by the .NET
    business objects that you write.  You will continue to use the VB6 Data
    access layer while you still have VB6 components in play.  You will use the
    .NET data layer for .NET objects.  Don’t plan on mixing these two… it
    isn’t worth it.  (take that from experience and pain… trying to use the
    VB6 data layer from .Net, or vice versa, is a doorway to undue delay and
    gnashing of teeth.)
  • Step 6: Pick a relatively self-contained set of business objects.  Something
    achievable.  Don’t go to the biggest, but don’t pick the smallest either.
    Give yourself a chance to succeed and still learn.  Recode those business
    objects to .Net.  Have your GUI call them instead of calling the VB6
    objects.  Have the .Net objects call the parallel data access layer.  Learn
    from the experience.
  • Step 7: refactor everything.  Literally, take an entire dev-test cycle to
    just fix the stuff you need to fix.  This could be a month or several
    months, depending on the size and complexity of your app.  Be specific.
    Plan your refactoring.  Do this only to improve structure.  Apply lessons
    learned to the new code, as well as fixes to the interface into the old
    code.
  • Step 8: Release.
  • Step 9: Recode another set of business objects.
  • Step 10: Repeat steps 8 and 9 until all business objects are recoded in
    .NET.  Retire the old VB6 data layer.  Release.
  • Step 11: buy champagne.  Give everyone a week of skiing or windsurfing.

On being offered MVP status

By |2004-12-21T06:53:00+00:00December 21st, 2004|Enterprise Architecture|

Back when the MVP program first started, I was a developer at American Express.  I was writing code in VB1 and VB2, and the forum was on Compuserve.

I answered questions regularly, and earned an MVP award.  I kept it up, and remained a VB MVP for three years.  Then, when the dot-com explosion came, I stepped out and took my shot, first at fine.com and then at Acadio.com (which I co-founded).  I simply didn’t have time to participate in the forums, although I did miss the interaction.

Earlier this year, I started to contribute again to the newsgroups, this time to the C# forums.  It is fun and I enjoy it.  I also applied to work at Microsoft.  The dot-com bust had crushed my former employers and I had grown tired of the politics surrounding the government-sector consulting that I was doing. 

Well, Microsoft took me on, and I’ve been with one of the IT groups for a number of months now.  However, I did not reveal this fact on the forums.  I didn’t want folks asking me questions about their MSDN accounts, or asking me to justify a language feature in C#.  Microsoft is a big place. 

However, I got a call about a week ago.  Apparently, I had been nominated for MVP in C#.  What an honor!  However, since I was an employee of Microsoft on the day I was nominated, I had to refuse.  Microsoft employees are not eligible for this award.

Such is life. 

But it is time to reveal my employer.  So from now on, my signature line on the newsgroups will have the characters [MSFT]. 

Let’s see if I start getting the PSS questions…

How to learn Object Oriented Programming

By |2004-12-21T06:42:00+00:00December 21st, 2004|Enterprise Architecture|

Do you understand what encapsulation means, but don’t know why in the world you would want to use it?
 
Do you see examples of inheritance in the frameworks and libraries, but have only just dabbled with inheritance in your own code?
 
If you answered yes to these questions, you want to begin to learn the patterns literature.

Object oriented programming gives you the tools, but doesn’t tell you how to effectively use them. 

We all just rummage in the dark sometimes, but after a while, some techniques started showing up over and over.  A few years back, a group of clever researchers decided to ask some of the best developers using OO languages what solutions they were coming up with.  If a solution tended to appear over and over, the researchers wrote it down, gave it a name, and described when it would be good to use.

These recurring solutions are called “design patterns.”

The landmark book that opened up design patterns to the world is called”Design Patterns, Elements of Reusable Object Oriented Software” by Gamma,Helms, Johnson, and Vlissides (a.k.a. the Gang of Four).

This book is an excellent reference, but not an easy read.  I do not recommend that you start there.  I recommend, instead, that you start with a slim and readable volume called”Design Patterns Explained” by Shalloway and Trott

These are not code snippets per se, although many books illustrate their examples using bits of code, and I’ve seen at least one patterns book come with a CD illustrating samples of each pattern (Design patterns in C# byCooper).   Design patterns are not libraries, either.  They are techniques and solutions to the problem of learning and understanding Object Oriented programming.

Take a look at the Shalloway book.  If you aren’t a better programmer after reading that book, I’ll eat my hat. (I read it in two days… it’s an easy read)

A list of useful books in the patterns space can be found here:

http://www.amazon.com/exec/obidos/tg/listmania/list-browse/-/2U7CN56RFJYU3/ref=cm_aya_av.lm_more/104-0692560-9975161

Good Luck

(this blog posting was copied from my prior blog.  I find myself referring to it in the forums from time to time, so I wanted to keep it around.)

Storing configuration settings for your DLL to use

By |2004-11-25T11:32:00+00:00November 25th, 2004|Enterprise Architecture|

One common complaint about the .NET framework is that there is only one config file for the application, even if there are many assemblies (exes and dlls).  This post contains advice for how the author of a DLL can keep the configuration settings for that DLL seperate.

The config file gets its name from the .EXE, not the DLLs.  Therefore, if your U/I is the EXE (which it usually is), then your dlls will be getting their settings from the config file for the EXE.

It is often the case, however, that the DLL provides services that can be configured, and that you would want to configure those services differently for each application that uses the DLL.

On the other hand, if you want to provide a set of “common services” using your object layer, then you can create an XML file that the DLLs will use. So, how does the dll find it’s XML file?

I’ve answered this question many different ways in different apps, trying things out to see what works.  I’ve got three answers:

  1. Put the name of the XML file into the registry during install.  The DLL looks to the registry to get the config file name.  This is useful if you are writing a DLL that needs to run from Biztalk or Sharepoint, since you cannot control either their executable or the directory in which they are installed.
  2. Give the XML file a fixed name, hardcoded into dll itself.  Have the DLL look into the application directory (where the EXE lives) to find the XML file.
    • Variation: in the app.config for the EXE, provide a seperate section for the DLL to use.  That section will contain the name of the XML Config file.  If no name is given, use the hardcoded name.  If neither is found, use basic default settings or raise an error in the constructor of your classes.
  3. During install of your app, create a specific directory where nothing but the XML config file will live.  When it comes time to go looking for the file, take the first XML file that you find that lives in that directory.
    This is convenient when transferring your app from dev to test to production, because you can have three files: dev.cml, test.cml, and prod.cml (I renamed xml to cml on purpose). 
    When you install the app, all three are placed in the directory.  The next step in the install is to ask the person doing the install “what environment is this” and, using their response, rename the proper file to the “xml” extension.

In all three cases, loading an XML is not as difficult as it first appears.

Take a look at the XSD.EXE tool that is delivered with the .NET SDK (a free download from Microsoft).  Using the XSD tool, you can point at an XML and the tool will generate a class that the XML will deserialize into. 

Using this class, and the XML deserialization methods built into the framework, you can very easily load the entire XML file into an object that contains other objects.  Now, you can use that object “tree” to inspect your settings very easily. 

In fact, if you change the values in the settings, it is very easy to save those changes back to the XML config file by simply using the serialization functions (something that is a bit more difficult with the config files).

I hope this provides useful information.

Three levels of abstraction in BPM – Part 1: Business Unit Level

By |2004-11-16T18:43:00+00:00November 16th, 2004|Enterprise Architecture|

I identified, in an earlier post, that I believe that there are three levels of abstraction in business process modelling.  The highest level (what is often called the 30,000 foot view), is the Business Unit Level.

This blog entry will discuss some of the attributes of models at the Business Unit level and some of the rules that should be followed to keep your models clear, consistent, and useful.

To see the overview, visit:
http://blogs.msdn.com/nickmalik/archive/2004/09/14/229641.aspx

At the business unit level, you are modelling the interactions between business units involved in a set of related business processes.  A business unit may be departments within a company, or may represent partner relationships, or customer relationships.  One example I am somewhat familiar with is the flow of medical claims data from a medical provider to one or more insurers who may (or may not) provide some payment against the claim. 

It is important to note that different departments in a single organization may very well be part of the model.  This level is not restricted to showing only the boundaries created by actual business ownership.  However, to keep this level of abstraction clean, you need to know the boundary for where to start, and where to end. 

The boundary for the business unit level begins when a business document is completely defined.  For example, a medical claim may contain information about the ambulance charges, room charges, durable medical equipment, and doctor’s fees for procedures provided through the institution.  An entire process may take place to create the medical claim (it usually does) and an ongoing process may take place to validate, modify, and/or supercede it.  However, this process is outside the scope of our business unit level model until the business document is created.

In some scenarios, the fact that a document has been communicated to a partner actually defines its existence.  For example, if I send a claim to an insurer, I will assign a claim number that is unique to me.  In some systems, the claim number is not assigned until I send the claim, and at that point, it becomes a claim.  Before that, no claim exists.  Additional charges for the same patient would be sent on another claim.  In this sense, the fact that the document needed to be transmitted stands as the “moment of creation” for the document.  When defining the business unit level processes, this is a boundary.

As I mentioned in my overview, the business unit level of workflow description is characterized by a single set of steps, one “thread” of interaction.  Of course, business doesn’t really follow a single pathway.  However, the interaction will follow a single “typical” thread which can be modified by the addition of messages.

So, in our example:

    • The hospital sends the insurance claim to Blue Cross. 
    • Five days later, the hospital sends a message to Blue Cross asking them for status of the claim. 
    • Two days after that, Blue Cross sends a message asking for more information. 
    • The next day, a reply arrives with the results of a particular test (for example). 
    • A week later, a message is sent from Blue Cross to the hospital providing information on a payment that the insurer is making to the institution to cover the charges.  Blue Cross sends a document to their bank instructing payment to be forwarded to the hospital.

Some definitions:

Business Document: A business document is a self-contained, uniquely identifiable document that exists as a snapshot of data in existence at a point in time.  

Message: A message is a self-contained request or command that is passed from one party to another with the expectation that the message itself, or the response to it, will affect the business processes in one of the parties involved.  Messages refer to documents.  They do NOT refer to people, or entries in a database (both of which change over time).  Messages refer to a document that does not change. 

Some considerations:

The life-span of the workflow model matches the lifespan of the document.  If a document continues to live, on one side or another, after the primary interaction is through, then the workflow model continues to exist as well.  As a result, a great deal of document management issues, as well as information archiving and retrieval, can creep into workflow.  Remember that, at this level of detail, we deal with documents and messages.  The methods that a business unit may employ to fulfill their committment to respond to a message are not useful details at this level of abstraction.

One document may give birth to another document.  In our example, a claim gives birth to a payment.  Note that a payment is not a message.  It stands alone from a claim. A document may be sent from the insurer to the hospital informing them of the existence of a payment, along with messages that tie the payment to specific claims. These two documents may (normally will) have different lifecycles.

So, in conclusion, the Business Unit Level illustrates the interaction between business units, is tied together with the creation of a document and continues through it’s lifespan, and describes the interactions in terms of the document and messages that refer to it.

[minor edits made in Aug of 2008]

SOA and BLOBs — using SOA principles for block-oriented data transfer (Updated)

By |2004-11-01T18:41:00+00:00November 1st, 2004|Enterprise Architecture|

Abstract: What happens when a business transaction, in Service Oriented Architecture, is too big to fit into a simply SOAP transaction?  This (updated) article describes a problem of this nature and the solution that allows block-oriented data transfer to work in an SOA-based application.

Introduction

Some things should simply not be done. 

If I have a large batch file, with ten thousand records in it, and I want to transfer it from point A to point B, using an SOA model to transfer the each record, one at a time, is really dumb.  The folks who believe that “all things must be done in XML” will not gain any points with me on this.

On the other hand, sometimes a single record (a single business transaction) is big… big enough to consider block-oriented data transfer.  This article is about one such situation and how I am proposing to address it.

The big business transaction

I deal with documents.  Some of them are in “source” format (word documents, powerpoint presentations, Infopath forms, even PDF documents), while others are simply scanned images of a document (in TIFF mostly).  These documents, the metadata that describes them, and the relationships that bind them, can form the basis for a “set of documents” that business can understand.  A good example would be the papers you have to sign when you buy a house.  There are deeds and warranty trusts and loan papers and all kinds of stuff.  I don’t know what they all are, but I do remember that it took hours for my wife and I to sign them all.

Together, these documents make a package.

And now for the problem: we want someone to be able to submit all or part of a package of documents, from one party to another, over the web. 

Doesn’t sound so hard, does it?  Surely, we aren’t the only folks to deal with something like this, but I haven’t seen many examples of how this is done in other XML-based solutions.  Not even legal e-filing, where this seems a natural requirement.  Perhaps I just missed it. 

A “business document package” contains header information and many documents.  The list of documents changes over time.  In other words, I can create a set with four documents, add a fifth, then replace the third.  Each document can be a TIFF or another large-format document file (too big to fit in a SOAP message on HTTP). 

The SOA Mismatch

Service oriented architectures usually present the notion of a “business document” or “business transaction.”  For the sake of clarity, I will use “business transaction” since my transactions themselves contain binary objects that just happen to contain documents… it would be too confusing to describe any other way.

So we have a business transaction.  This can be implemented in many ways.  SOA says that a document is self-contained and self-defining.  Therefore, the document set must be self contained and self defining.

Normally, in the SOA world, if a business transaction is updated, we could simply replace the entire transaction with entirely new values.  So, if a transaction is an invoice, we would find the existing invoice header, delete all the rows associated with it, replace the values in the header, and add in the rows from the document.  All this is done as “data on the inside.” 

The problem is that the entire contents of the business transaction are huge.  Our self-contained transaction contains the header information and all of the scanned documents.  If each document is 2Megs, and we have 14 of them, then a 28MB SOAP message starts to seriously stretch the capabilities of the protocol.  It is literally too big to fit into a SOAP message without serious risk of HTTP Timeouts. 

So, we need the concept of an “incomplete” sub-transaction… and that’s where the solution lies. 

(Note from nick: we decided to go a different direction: I’ve added details at the end of this posting).

The SOA solution

In our interaction, we have two computers.  The sending side, where the transaction originates and the receiving side, that needs to end up with all the data.  Both sides are computer applications with database support underneath. 

The new transaction is created by the sending side.  It will send the document header and enough information for the receiver to know what documents survive into the final form.  Any existing documents that changed will be deleted from the receiving side.  All documents that don’t exist on the receiving side, when this process is done, are represented as an “incomplete records” in the receiving end’s database, along with some size data.

Now, the sending side asks the receiving side for the id of a document that is marked as “incomplete”.  The receiving side responds with a message stating that “SubDocument 14332 in document set AB44F is incomplete.  We have block 9 of 12”.

The sending side will then go to the database and extract enough data to send just one block… in this case block 10.  That could be simply 100K in size.  Wrap that up in a SOAP message and send it.  The receiving side will get the message, which contains a complete header document and the contents of this block.  The interaction is done, and will start over with the sending side asking for the id of a document that is marked as incomplete.

The conversation

So, it look like this:

Sender sends:

<MyDocumentSet id=”849C751C-FF5C-4438-A3F0-055B9EE786E3″ >
   <Metadata Filer=”Nick Malik” CaseNumber=”ABC123″ —other stuff — />
      <Contents>
         <Document id=”EBDE445D-5C26-43da-A142-E12A350EC1B6″ name=”MyDocument1.pdf” — other header info — />
         <Document id=”9E4F8C83-B2D1-4aee-8C53-B235D026CD1E” name=”Document2.doc” — other header info — />
         <Document id=”05B10DAA-2A01-406b-AAB0-6BAEEF98F7A8″ name=”MyDocument3.ppt” — other header info — />
         <Document id=”7135612A-CE48-4371-ABFC-F8EF70DF76CF” name=”MyDocument4.pdf” — other header info — />
      </Contents>
   </MyDocumentSet>

Sender gets the message and checks to see if that document set already exists.  If it does not, simply create the document set on the receiver side with four incomplete documents.  A much more interesting case happens if the document set already exists on the receiver side… so let’s look at that.

The receiver looks up document set 849C751C-FF5C-4438-A3F0-055B9EE786E3 and sees that it currently contains five documents.  The first three documents in the existing document set are named in the list above.  The fourth document above doesn’t exist in the existing document set, so it is an addition.  The other two documents in the destination document set must be deletions.

So we delete the two extra documents on the receiver side and add a document for MyDocument4.pdf, and flag it as incomplete.

Now, the sender asks the receiver for the id of any incomplete documents.  The sender replies with the id of the fourth row above: 7135612A-CE48-4371-ABFC-F8EF70DF76CF and the fact that no blocks of data have been successfully stored.

The sender side gets this response and decides to send block one of that document.  It goes to the data
base, gets the first 50,000 bytes of data, encodes it with Base64 encoding, and sends it back to the receiver as the following:

<MyDocumentSet id=”849C751C-FF5C-4438-A3F0-055B9EE786E3″ >
   <Metadata Filer=”Nick Malik” CaseNumber=”ABC123″ —other stuff — />
      <DocumentBlock id=”7135612A-CE48-4371-ABFC-F8EF70DF76CF” name=”MyDocument4.pdf” — other header info — >
           <Block totalblocks=12 thisblock=1 size=50000>
FZGl0OzI0NTk2MDs+Pjs+Ozs+O3…a really long string of base64 characters … 
           </Block>
      </DocumentBlock>
   </MyDocumentSet>

The receiver now appends this data to the current document on the receiving end.  Note that the receiver “knows” that, even though this message is complete, the document is not complete, because this is block 1 of 12 (see the <Block> tag above).

The sender then asks again: what documents are not complete.

The receiver responds again: 
Document 7135612A-CE48-4371-ABFC-F8EF70DF76CF is not complete… we only have one block of 12. 

The sender sends block 2… and on it goes until the last block is sent.  At this point, the reciever gets the final block, marks the document as complete, and appends the last set of data to the database.  The next time the sender asks “what is not complete” the receiver responds “everything is complete”

The loop terminates.

The motivation for doing block-oriented data transfer this way

Certainly, we could use FTP or some other mechanism for file transfer.  This method, though, has some characteristics that are interesting.  First off, this protocol is stateless.  That means that, at any time, the sender could stop asking about the status of documents on the receiver side, and nothing is lost.  The sender can go offline, or go to sleep, or lose connectivity, and nothing bad happens.

Secondly, because the block sizes are relatively small, SOAP doesn’t time out.  We can handle extraordinarily large files this way (theoretically in the terabyte range).

Thirdly, the sender doesn’t have to know much about the receiver.  It doesn’t have to know if the document set already exists in the database on the receiver side, because the header data is sent with every block.  Therefore, no Commands are being sent.  (See my previous blog on “commandless” documents).

Pros and Cons (updated)

At the time of my first posting, this idea was being floated to our development team.  There are pros and cons to this solution that I can discuss in more detail now.

The advantage of this model is that the receiving side is not getting any data that it doesn’t want or know what to do with.  The sending side asks “what do you need,” and the receiving side responds with “file X Block 10”.  However, this is still a communication protocol.  If the sending side decides not to ask, the receiving side has no option but to leave the content of its database incomplete. 

This is (a) counter-intuitive, and therefore hard to explain to business users and the development team alike (as I have discovered), and (b) we have mixed the details of data transmission with the details of data representation.  I hadn’t thought carefully about this when I first wrote it, but, on hindsight, it’s a bad idea.

An SOA transaction should be complete, self-describing, and self-contained.  The process above saves us from sending the same bits more than once over a wire.  That’s its biggest advantage.  But that’s not our biggest cost.  All of the wires that I care about, in my application, are owned by my company, and they are utilized at a fairly low rate.  Therefore, we don’t save any measurable dollars by making the data transfer process efficient.

On the other hand, if we seperate out the data transmission from the data representation, then we can test each seperately.  I can test data transmission of a 20 GB file by transmitting any 20GB file and comparing the results with the original.  I can test data representation by creating the business document on one end and copying it to the other using sneaker-net (walking it over) from one dev machine to another.  This test isolation is important for reducing complexity, and that will save measurable dollars… real money from my bottom line.

The forces that led us to SOA still exist: we want to decouple the sides from each other and we must transfer these large transactions over HTTP or HTTPS connections.

The new interim solution

We decided to seperate the data transmission from the data representation.  Therefore, we will create an envelope schema that simply provides a transaction id, the current block number, to total number of blocks, and a data field. 

So a transmission could look like this:

<Transmission id=”39B2A4DD-AD68-4ae9-AA68-FCC6A48A0FFA”>
           <Block totalblocks=12 thisblock=1 size=50000>
FZGl0OzI0NTk2MDs+Pjs+Ozs+O3…a really long string of base64 characters … 
           </Block>
</Transmission>

What goes in the Block?  A base-64 encoded form of the entire business transaction itself (possibly compressed).

The receiving side will collect together all the block, assemble the actual stream, decode it, and load it into an XML object.  From that, we can extract embedded documents.

This data is not optimized for transmission

We get a lot of data inefficiency in the data format here.  If we haven’t thought seriously about compression before, it’s starting to become important now.  Here’s why:

Uploaded document, in PDF form, is a page of text.  Notepad would represent it as about 1K.  In PDF, it would be about 5K because PDF includes things like fonts and formatting.  That’s fine. 

In our business document, that 5K becomes 6.7K, because, in our business document, we are embedding it in Base64 text.  Base64 is a format that represents three bytes (24 bits) as four characters of six bits each (24 bits).  Add about 2K of header information (to make our document complete) and the business transaction size hits 8.7K.  At this point, we take that 8.7K transaction and encode it, again, as Base64 for the sake of block transfer.  We now get 11.6K.

Our PDF went from 5K to 11.6K.  That’s double it’s original size, and that’s assuming UTF-8 encoding in the XML.  If we go with UTF-16 encoding, the XML files can hit 20K. 

On the other hand, if we compress just before we pack the data into blocks for transmission, we can take that 8.7K document and compress it down to just over 5 K, (even though it is character data, it is not going to compress further, because it is randomized, which removes the advantage of compression).  We take that 5K document and encode it in Base64, we go back up to 6.7K.  Now, that is efficient for data transmission.

The receiving side has to decompress, of course, but this may be worth it.

Conclusion

After reviewing the initial proposal to embed the data transmission mechanism directly into the data representation structure, we rejected the idea in favor of a mechanism that wraps the data representation structure with a data transmission structure.  This allows us to test data transmission seperately from data representation.  It also allows us
to stick to the original idea of keeping all of the business data together in a single business transaction, regardless of how large it grows to be.

Stateless is to service as commandless is to data

By |2004-10-25T13:57:00+00:00October 25th, 2004|Enterprise Architecture|

Abstract: This article provides a counterpoint to Pat Helland’s most recent article by describing the concept of “commandless data documents,” and how they are logically used in Service Oriented Architecture.

Introduction

I was re-reading Pat Helland’s latest SOA article “Data on the Outside, Data on the Inside” recently.  It is a good step forward in the discussion of how data is managed in an SOA environment.  I believe that one aspect of the discussion is lost in this article and many of the others that I have read: Something I like to call “commandless” data.

In Mr. Helland’s article, he states:

In service oriented architecture, the interaction across the services represents business functions.  These are referred to as operators.  The following are some examples of operators:

  • Please PLACE-ORDER
  • Please UPDATE-CUSTOMER-ADDRESS
  • Please TRANSFER-PAYMENT

I don’t get a chance to disagree with his articles very often, but this time, I must.  The above statement is often incorrect because it assumes that the document represents a command, and not a statement of fact. 

To whit: When a purchase order passes from a buyer to a seller, I assert that the document is NOT a request. 

A statement of fact

The sending system has already entered the document as State information.  It is not, in all liklihood, waiting for a response before booking the purchase order as an outstanding obligation.  No, the purchase order is not a request to purchase… it is a statement that the buyer expects the seller to follow through a contractual arrangement that will result in the transfer of goods and funds.   This statement of fact is already on the balance sheet of the buyer.  It already affects financials, projections, and, in all liklihood, customer-facing statements about the availability of products further along in the supply chain. 

Viewing a purchase order as a request assumes that the sender knows what must be requested, which means that the sender is sending a command to the receiver.  Viewing the purchase order as a statement of fact means that the sender cannot know what must happen from a system standpoint on the receiver’s end.

This is a minor shift in the point of view, but a very important one.

The fantasy of “Once and only once”

The first time the seller gets the document, the seller may (or may not) be surprised by it.  “Look, ma, someone wants to buy something!”  However, the behavior of the receiving systems is not “add” the purchase order!  On the contrary, it is “Does this order already exist?  If not, add it.”  That is because the order may very well already exist.  In fact, how can the sending system know that it doesn’t?

The systems of business have to allow for information to come from multiple sources.  There is no concept of “send it once and only once.”  People use telephones, fax machines, e-mail, online ordering, snail-mail, calls to the customer service rep, and indirect purchases (e.g. from a dealer) to indicate that product needs to move from place to place.  I call this an “open channel” because the same message may arrive, in different formats from different sources with the same content.  The sending system cannot assume that the receiving system does not already know about the transaction, nor should it presume to dictate what the receiving system should do with it.

Never a command?  C’mon…

That is not to say that documents must never contain a command, but they would be commands for information, not normally commands for functionality on the receiving end.  By this, I mean that some messages are, of and by themselves, useless without referencing an ongoing transaction.  For example, if I ask for the status of an order, I am referring to an order number for an order that already exists.  The receiving system would respond with information about the status of the order.  A list of response codes may include…

  • Already shipped
  • Shipment planned on schedule.  Expected ship date: 11/01/04
  • Shipment planned outside of expected schedule: Expected ship date: 12/01/04
  • Shipment accepted and vendor commits to fulfillment, but ship date not known.
  • No shipment planned.  No committment is made.
  • Problems detected in processing the order.  The order is on hold until manual resolution.
  • I’ve never heard of that order.

On the other hand, if I want to transfer payment from one account to another, my document does not contain a request for the transfer… it simply states that the sending system considers the transfer to be a fact, and would like the receiving system to confirm that it, too, believes the same to be true.

This way, if the same document is sent from the sending system to the receiving system 25 times, the transaction still occurs only once on the receiver.

Note: Acknowledgements are a special case.  If I send a document to a system, and get an acknowledgement back, I do not expect to get back the entire submitted document.  I would expect to get back my sending identifiers and the receiving systems unique identifier for the same document.  I would want to ASSUME that the document that I referenced as “Contoso transaction 12” and which was acknowledged as “Contoso transaction 12” was not changed by the receiving system before being acknowledged.  If this is not the case, then the protocol would need to handle these situations. 

I can respond to a question… how do I respond to a fact?

This ability to respond to a fact, rather than a request, is essential when creating a new system in the world of SOA.  This is because we need to be able to make statements that are detached FROM EACH OTHER.  Sure, some documents are going to cross reference (like my status request).  However, the system should function, and function well, without having to coordinate many of the incoming documents.

For this reason, if a purchase order arrives from a new buyer, the purchase order doesn’t need to be preceded with a “add new customer” order.  The purchase order establishes the customer if he or she doesn’t already exist. 

This is the concept of “commandless data.”  This is as powerful a concept for integration as “stateless transactions” are for scalable web page requests.  With commandless data, we get simplified interactions, scalable messaging, and protocols that are reliable, even when our networks are not.

This is why I love this stuff.