Document Management as a foundational technology for Workflow

By |2004-09-27T21:49:00+00:00September 27th, 2004|Enterprise Architecture|

I like the term “foundational technology”.  It sounds so much cooler than it is.  I have no idea if there is one and only one definition for it.  My definition, in this post, is this: A foundational technology is a technology that provides infrastructure for another technology.

In this context, we would say that a document management system could require a relational database system as a foundational technology, because all of the document management systems that I know of store some data in a relational database.  (I’m sure there’s an exception to that statement somewhere… it’s an example… please don’t flame me).

And now to the meat:  In my opinion, document management is a foundational technology for most workflow systems. 

This means that workflow systems are NOT part of document management, but good workflow requires at least some element of document management in order to function properly. 

This means that workflow should be built on the concepts of document management, but workflow does not, of and by itself, represent core functionality of document management.  It is in addition to it.

In an application where the records are actually in a database, and not stored in stand-alone documents, this means that the workflow process will treat the records as documents.  The workflow process should not be dealing with relational intricacies like many-to-many relationships. As far as the workflow process is concerned, the data that it deals with is self-contained and self-descriptive.  The fact that this data may be managed in a relational database system is entirely unimportant from a workflow standpoint. 

(In the engine I am working on, this is enforced by requiring the application to extract the data from the database and pass it to the engine in the form of an XML document.  The same document comes back from the workflow engine, potentially modifed.  The app is free to break it back out into records and update the database.)

So, what elements of document management are required for workflow to “work?”  I refer to a “managed document” to mean a document that is within the operational boundaries of a workflow management system.

Managed documents are self-contained and self-describing.  This means that any relationships to outside data (vendor numbers, product codes, country codes, etc) are not important for the processing of the document through the workflow itself.  If data is required for the processing of a document, then the related values should be brought into the document, and decisions should be made not on the coded foreign key, but on the business value from the lookup table itself.

Managed documents are persistent.  Their lifetime is determined by business value and business rules, and they nearly always remain in existence throughout the life of the workflow process (at least) and often much longer.  This stability is expected for a system that needs to manage the information about an item over potentially long periods of time. 

Managed documents are individually addressable.  While the name of a document does not need to be unique, the name of the document within a “directory” (or document collection) would be unique.  The system has one clear way to refer to the current version of each document.  (Prior versions are perfectly normal.  However, managing prior versions is not a workflow function, and the workflow system is not normally required to be aware of it).

One version of the managed document exists at any one time.  This is an important one.  Most document management systems do a fairly good job of allowing a user to check out a document for modification, and then check it back in with changes.  This can occur in “groups” of documents (where all documents changed should be treated as a single change for the sake of check-in), or it can occur for single documents.  However, there is one “master copy”.  After each operation, the system reaches a stable state by ending up with one master copy (even if a merge operation is required).  In a RDBMS system, we say that we want our transactions to be atomic.  In a DMS, we want to have only one version of the document “present.”  (This is why relational DBMS systems have proven so useful for document management… they dovetail nicely).

In an environment where these rules can be followed (most applications can be fit to these criteria), workflow systems are fairly simple to implement.  The engine that I am working on would have no difficulty tying to an .NET application that can meet this notion of document management, even if it doesn’t actually manage documents.

YAWL — must see TV for Workflow practitioners

By |2004-09-24T14:08:00+00:00September 24th, 2004|Enterprise Architecture|

YAWL – Yet Another Workflow Language

I suppose the name would be funny, if the idea for the name wasn’t so ingrained in Unix culture, with such landmark tools as YACC (Yet Another Compiler-Compiler – a BNF language tool available in some of the earliest releases of Unix from Bell Labs).  Aside from the cute name, though, there’s a real conceptual basis for this particular open-source product.

YAWL grew out of the research described on the Workflow Patterns homepage (http://www.workflowpatterns.com).  This remarkable research and development effort, performed by Prof. Wil van der Aalst, Assoc. Prof. Arthur ter Hofstede, Lachlan Alred, Lindsay Bradford, and Dr. Marlon Dumas is an exceptional leap forward in the growth and maturation of workflow as a basis for collaborative applications in practical uses.

While I haven’t actually implemented any of the YAWL engine code in the workflow engine I am creating, I freely admit to having read some of the research literature that Dr. van der Aalst has placed into the public domain.  (I have not even downloaded the code, although I hear that Yawl now has a SourceForge site where the code is available.  I don’t want there to be any finger-pointing about “who owns what code” if my project is ever posted to the public domain.)

Splits and Joins

Some of the most compelling work went into creating a structure for understanding how process flows from task to task, and the kind of rules that can be embedded into the junctions between a task and the link that flows out of it.  This was a new concept to me when I first heard it, and I suspect that it may be new to others as well.  I will try to distill it here, but if I get it wrong, please tell me so I can correct my thinking.

A “split” is a condition on the exit for a task, where more than one link can emerge from the task.  In other words, if there’s two roads out of the state you are in, you have a split.  There are three kinds of splits: AND-split, OR-split, and XOR-split.

A “join” is a condition for the entrance to a task, where more than one link merges in to the same task.  In other words, of two different flows can come together in a single place, you have a join.  Once again, three types: AND-join, OR-join, and XOR-join.

The logical operators are similarly named for a reason.  The operations of the workflow engine are driven by the type of split and the type of join that the system encounters along the way.

XOR Split and XOR Join

An XOR split will activate one outgoing link from the task.  Only one.  Therefore, it doesn’t create a situation of concurrency.  An XOR join will activate the task each time an incoming link is activated.  Therefore, if two concurrent processes are winding their way through the workflow, and they both come to the same task, the task will be triggered twice.

OR Split and OR Join

An OR split will activate one or more links from the task.  As many as the conditions will allow.  Therefore, an OR split can easily create a situation where concurrent operations begin to come into play.  An OR join is useful for making sure that the task is only fired once, regardless of how many of the inbound links eventually fire.  Essentially, the first one fires the task, and the rest are discarded without firing anything.

AND Split and AND Join

An AND split will activate all outgoing links from the task.  Think of this as a constrained OR split… you are definitely creating a concurrent situation.  An AND join will wait until ALL inbound links are activated before firing the task.  In other words, if three links come together in an AND join, the task will not fire unless the processes that feed all three links are completed.  This encapsulates a fairly powerful concept that is sometimes lost on workflow system developers.

There are other operators as well: Composition, Multiple Instance, Cancellation, etc.  I don’t have time, or room, to go into them here.

I encourage any student of collaborative workflow to read the material on the workflow patterns site and to look into YAWL.

Is workflow best represented by something other than a graph?

By |2004-09-20T13:55:00+00:00September 20th, 2004|Enterprise Architecture|

A few days ago, I blogged about the three layers of abstraction for workflow items.  Now, I’m going to think aloud about a theoretical problem… is workflow best represented by a single graph? 

Is the analysis of the problem leading to another problem?

Most workflow processes are described using one of two types of visual paradigms: “flowchart” style graphs and petri-net graphs.  Both are directed graphs that start in one place and follow to another.   Every workflow that I’ve seen attempts to capture 100% of the workflow steps in a single diagram.  Even those folks who attempted to be “sophisticated” would limit themselves to only two levels of abstraction (a high level process, and individual low-level processes that “plug in” to the high level process at specific places).
The problem I’ve been running into is that these abstractions are extraordinarily leaky.  The graphs usually combine details from more than one layer of abstraction.  Error handling is described in different terms depending on your location within the workflow, and the ability to modify the workflow in an ad-hoc manner is not covered at all.   This is “directed workflow” and, honestly, doesn’t model the interactions that people truly expect to perform in a business environment.
While analysts have no difficulty describing systems from the standpoint of a high-level graph, it does not appear to be a good implementation paradigm.  As my former coworkers in the hardware space might say: there is impedance in the system.


The concept of Impedance comes from Electronics, where it is a carefully defined measurement.  In Computing, in recent years, this term has been used to describe the “resistance” between a relational database modeling system, and an object oriented middle tier in an application, requiring complicated logic in the data access layer to map data from one storage paradigm to another.

In my context, I am using this term to refer to the fact that humans follow processes very well, as long as they make room for human activities in a particular human manner.  Our methods of creating a workflow diagram, on the other hand, do a poor job of actually capturing the requirements of human beings, working with individual capacities and on individual cases.  

To overcome this impedance, we either create overly simplistic workflow designs or overly complex designs.  The simplified designs simply show assignments and require that the individuals interacting in the workflow would simply “know” the steps that they need to follow when work has been assigned to them.  This is perfectly human, but not terribly flexible from a business standpoint.  The overly complex designs attempt to model every possible exception or flow.  The problem is that people, in the act of using the workflow, will inevitably choose a path that was meant only as a very rare exception… leaving some work item stranded down a rabbit hole.  Overcoming this requires extensive training… and once again, limits business flexibility.

Sometimes, our systems attempt to overcome the impedence for us.  A few workflow systems attempt to hide this impedence under a layer of compiler transformations that seek to take the flowcharts and petri nets and convert them to executable models that can be interpreted at runtime.
I’m beginning to wonder if this is an effective approach. If the analysis method doesn’t match the area of human activity, let’s stop trying to force it to fit.  Let’s just change the analysis model.

Adapting the “multiple views” concept to Workflow

Just as software development requires multiple views in order to describe a design, I believe that workflow analysis should require multiple views in order to describe a process. 

The static view

The static view would be a fairly simple Finite State Automata diagram, where work items remain in a state until an event occurs to move it to another state.  This view type is well understood, so I won’t describe the idea in detail here.  My only addition would be that one or more diagrams may appear at EACH level of abstraction, and that an analyst should limit themselves to describing only a single level of abstraction in each diagram. 

The dynamic view

Another view would be to approach the model as an activity stack.  Essentially a “porous” LIFO stack, a workflow would start with a set of activities, a series of evaluation expressions, and a collection of variables in their initial states.
The each activity would define behavior for pushing messages out.  After initialization is complete, the each activity is evaluated, and any outbound messages are sent to a partner (usually an application, a managed component, or a transmitted form like Infopath). 
The system then waits for zero or more messages to arrive.  (Note: the object may subscribe to one or more time-based events).
When an event arrives, predefined behavior will dictate that the event will either perform an operation or modify one of the state variables.  (If it performs an operation, the operation may indirectly modify one of the state variables).  When the event is complete, each of the conditions on the stack is evaluated.  All conditions that are true are dropped from the stack.  (This is the porous part… all items are evaluated.  Therefore, on a seven element stack, items 1, 4, 5, and 7 may all be satisfied by a single event, leaving only items 2, 3, and 6 remaining on the stack).
The top activity would be re-evaluated.  Any outbound messages will be transmitted, and the system returns to waiting for an event.
This is not altogether dissimilar from petri nets.  The difference is that Petri nets use mathematical weights to determine paths and outcomes, making them inherently difficult for a business analyst to construct.  Difficulty of use dramatically reduces the effectiveness of a solution (Assembly language is fast, but very few programmers use it… same reason).
The conditions on the stack would represent various patterns of interaction.  These exist at the pattern level (what I call the “business process” level of abstraction).  The interaction conditions would include things like various combinations of approvals, weighted voting, time and termination conditions, etc.  Dr. Van der Aalst produced an excellent paper on workflow patterns that could serve as an initial inventory of primitives for this stack object.
The collection of state variables must be queryable at all times from the outside.  This is a major deficiency of some workflow systems (that will go nameless)… that the data that is used to derive decisions or track the status of an item is not visible by a program that wishes to find the item on the basis of that information. 
For example: if a workflow item manages the process of approving an invoice, and the invoice has some fields like quantity, supplier, requestor, cost center, total value, and even business priority, then shouldn’t these fields be visible to anyone who wishes to query the list of workflow items to find, sort, and report on the items in the queue? 

That’s a lot of work… why do it?

e advantage of looking at workflow as a stack: it is fairly easy to move a workflow item from one process to another.  The constructor for a workflow would define the necessity for a set of variables and initial conditions.  If a variable already exists, no need to create it again.  The existing stack is completely replaced with the new one, or stacked on top, and the process begins as described above… in many circumstances, there may not need to be any data conversion or gyrations at all.
Another advantage: ad hoc events could ADD items to the stack.  This would allow someone to add a primitive that could seek special case approvals, for example, or interact with a system that is only used occasionally.  New variables could be added as well.  This would allow a workflow item to truly change as it moves, yet remain deterministic. 
Of course, when a condition is “removed” from the stack, it isn’t really “removed” but placed into a bin of conditions.  The events are also stored in the bin.  That way, if compensation is needed to roll back the workflow, you can walk the bin, finding the events in reverse order and executing the compensation on each event.  Also, the bin can be used for audit tracking and generic time-based reporting. 

I’m not quite sure what diagramitics to use to model a dynamic view, or even if this view is a useful description from the design standpoint.  I can say, however, that I would have found it useful if this kind of concept were built-in to workflow systems, because I could use it to reduce the amount of code I’d have to write in order to implement a solution.


Messages vs. Documents in a Workflow System

By |2004-09-16T14:42:00+00:00September 16th, 2004|Enterprise Architecture|

Using workflow systems can be interesting.  Everyone has a good idea of how to solve the “workflow problem,” and it usually involves systems with various different basic features… however most are similar in that the problem doesn’t allow for a lot of variety in how it is solved.

In fact, a rather nice reference (if you can tolerate the formality of the language), is a dissertation that compares the features of various commercial workflow systems from a standpoint of the which workflow patterns are supported by the system.  A summary and the link to the original document can be found at http://tmitwww.tm.tue.nl/research/patterns/products.htm

One thing that is missed in a discussion of patterns, though, is the distinction between workflow messages, workflow documents and target documents.  I find this troubling, because this is one of the key points that caused a rather embarassing project failure (for me personally), when I was attempting to create a workflow solution using Microsoft Biztalk Server.

Definition: Workflow Message

A workflow message is a relatively small data object that encapsulates the “command” pattern.  The object contains enough data to associate the command to an existing data item in another computer system, and then issues a command against that data item. 

For example: in Healthcare, if you visit a hospital, the institution will send a claim to your insurance company asking to be paid.  This transaction is called the HIPAA 837-I.  This could take anywhere from a few hours to a few months for the insurance company to respond with payment.  At any point along the way, the hospital can send a “request for status” to the insurance company, to find out what is going on.  This request is called a HIPAA 276.  The response is a HIPAA 277, and provides some information about what the status is. 

Both the 276 and the 277 are workflow messages.  They contain enough information to refer to both the healthcare claim and the patient.  However, the message itself is not the business data.  It never has to be maintained.  It exists at a single point in time, and represents a command (in this case: a query for status).  To summarize, a workflow message is:

  • comparatively small in size
  • contains information that relates the message to a business object
  • expresses information about a business object or encapsulates a command to be performed on that object
  • is immutable… it exists at a single point in time, and it does not change.

Definition: Workflow Document

A workflow document is a data object that contains enough information to trigger or drive a business process.  While it is similar to a message in that it usually refers to other data objects, the workflow document can be changed during the course of the business process.  In fact, it nearly always has to be maintained in some fashion during the course of the workflow, and is often kept as a record of activity after the workflow process has completed.  This document may have attachments placed in it, and the list of attachments may change during the course of the workflow process.  A workflow document is a construction of the workflow process itself.  If there was no workflow process, this document would not come into being.  The format of the workflow document is nearly always constrained by the workflow process, and is often expressed as a form with fixed fields.

For example: let’s say that you work in a busy PR department, and one of the department’s responsibilities is to generate press releases.  You will usually have a workflow process to generate the press release.  In our example, the press release is generated by one of the company divisions and sent to a PR coordinator in your group.  That coordinator will then behave like a reporter, calling the source contact and collecting information about related products and detail on the information to be released.  The coordinator may collect the original PR request, one or two supporting documents, and a spreadsheet containing a release schedule.  The coordinator will track the date that the release should go out, checkpoint dates when the business has an opportunity to change their minds, and signatures from various folks showing that they agree with the content.  All of this information may change periodically during the course of the process.  The coordinator may share this document with other members of his or her team, and even assign it to someone when they go on vacation.

In summary: a workflow document is:

  • A creation of the workflow process itself — it does not exist seperately
  • It is used to capture the data needed by the process to enforce business rules
  • It may change many times during the course of the business process
  • It may need to be saved when the process is completed as a record
  • It may be transferred from person to person
  • It may have attachments that may or may not be directly related to the final product of the process
  • It has a specific format or set of fields.  It is often a form.

Definition: Target Document

A target document is a document in any format (spreadsheet, word processing, picture, diagram, whatever…) that is consumed by, created by, or managed as a result of the business process.  In our example of the PR department above, the final press release itself would be a target document.  So would the initial request for a press release if it existed in a different format than the form itself.  So would any supporting documents collected by the PR coordinator. 

The thing to understand about a target document is this: a target document doesn’t care about the business rules that surround it.  It is not specifically aware of it’s own status in the workflow process, who it is assigned to, and how long it has been there.   A target document is nearly always the “product” of a workflow process, but is nearly never aware of the process itself.  (The exception to this rule is when an “indicator” is placed on a document to indicate things like completness (“Draft” vs. “Final”).

In summary, a target document

  • is of any format that makes sense to the business users
  • contains very few fields or indicators of its own status within the workflow process
  • contains little or no indication of who it is assigned to or for how long
  • is managed by a workflow process at some point during its lifetime
  • may exist before the workflow process begins.  May continue to exist, and be edited, modified, and managed, after the workflow process has completed.

Why create the distinctions?

So, I defined three terms and hopefully did so in a clear manner.  The terms are fairly clear, but why are they important?

Because some systems have support for messages, but not workflow documents.  Others have support for workflow documents but not messages, and still others are entirely focussed on the target document alone.  Very few have support for, and understanding of, all three types of documents.

This is what bit me when I attempted to create a document management workflow using Biztalk alone.  Biztalk is an excellent message processing system and EAI middleware product from Microsoft that can be used to coordinate collaborative processes both within an organization and between organizations.  Let me make it clear: I’m not dissing BTS.  (I have often used the handle ‘biztalkbum’ in the online forums).

On the other hand, Biztalk is not a document management system.  It has no concept of a Workflow Docu
ment and nearly no concept of a Target Document.  There is no central formatted repository of data that is maintained through the steps of the workflow, and which can contain the documents that are collected as artifacts along the way.  If you need that, you have to add it.  I needed it.  In fact, it was so critical to the success of my project, and would take so much time to add it to Biztalk, that it would have been easier to start with a system that had support for shared documents, and add a workflow engine to it.  (That’s exactly the decision we made, by the way, but it cost us a month of delay as we tried to solve the problem with Biztalk).

So, by writing this down, I’m doing a couple of things:

  • I’m collecting my thoughts in one place, so that I can have a common term of reference as I go forward,
  • I’m sharing a set of terms that I hope others can use and adopt, so we can scrutinize and compare workflow solutions using criteria like “do you support the concept of workflow documents?”,
  • I’m coming to terms with the mistake I made, and trying to understand the distinctions I need to make to avoid making it again, and
  • I’m sharing my experience with those of you in the outside world who stumble upon this article, to help you avoid this mistake.

More articles on collaboration, workflow, and representation of the same, all to come…

Three levels of abstraction in Business Process Management

By |2004-09-14T20:51:00+00:00September 14th, 2004|Enterprise Architecture|

I’ve been thinking about the levels of abstraction that Martin Fowler defines in software development: a conceptual view, a specification view and an implementation view. 

These levels of abstraction reach far above “workflow” into the realm of Business Process Modeling.  Only the lowest layer of abstraction can reasonably be described as workflow.

Business Unit View: at this level, a work item can have threads in multiple, cooperating, business units. Communication between the workflow threads may be occasional, and will occasionally be required. However, the most typical workflows through business units is a single (primary) thread that can be reasonably easily understood and diagrammed. For example: Customer company issues a purchase order. Supplier company issues a shipment and an invoice. The shipment goes to the Customer company’s receiving department. The invoice goes to the customer company’s accounting department… etc. It’s a simple graph with a few lines and coordinating components.  Swimlanes work well.

One key point for business unit collaboration is that the connecting lines are sometimes documents, but are often just messages.  In this context, the difference between a document and a message is that a document stands as a complete encapsulation of a single business request, while a message requires the context of the business request to already exist.  For example: a purchase order is, itself, a document.  However, a request for status of a purchase order payment is a message. 

Business Process View: at this level, a work item follows one and only one thread through a series of interacting steps. The process may begin with a message or document from another business unit, and may end with the generation of one or more messages to other business units. This is the level where audit requirements typically live, and is the primary focus of efforts at Business Process Reengineering. In this view, an item lives in a single stage. Events do not cause an item to move from one stage to another. Instead, the transition out of a stage is expressed as a boolean expression with a series of business rules. For example: the purchase order is approved for transmission to a supplier if it contains a signature from a member of the company whose purchasing authority meets or exceeds the dollar limit on the invoice and if the supplier is on the approved list. It is up to the process to meet the condition.  It is not appropriate, at this level, to describe the criteria as a simple event (Joe approved the document).

Work Step View: at this level, an item exists in multiple states in a state machine or petri net. Events serve to move the work item from one state to another, and “gates” can be expressed in the graph showing things like: compensation logic, combination logic (all theads must complete before the item proceeds, etc), and time-related exception logic. This is the level of abstration that most workflow systems, and too many software developers, live at. We get so wrapped up in the details of how a message is read, and what occurs when it appears, that we can quickly and easily forget the existence of all other views.

I plan to expand on each of these views in upcoming blog postings.

Note: this blog entry was first posted on July 13th on my blog on blogger.com, which I am retiring in favor of this blog site.  I’ve updated the content, though, and added a little detail.

Note: minor terminology updates made in Aug of 2008