The blurry line between event driven data management and grid architecture

By |2006-08-31T03:22:00+00:00August 31st, 2006|Enterprise Architecture|

Ever found yourself describing a design that you think is so ‘obvious’ that everyone must already understand it, only to find that almost no one understands it… and then find that there is a product that uses it?

That has happened to me more than once.  It is frustrating.

So, I’m at the local meeting of the IASA listening to a presentation on ‘grid’ computing, which is a distributed computing architecture that ranges from SETI-style multi-processing to wide ranging smart-cache technologies.  Interesting stuff.

Then it occurs to me that the simple model I’ve been describing for event driven cache management, for well over two years, is really a simple form of the same data replication that is enabled by these tools.  I didn’t think of it originally… I had heard an ESB vendor describe this design many years ago, but he didn’t give it a name.  The design stuck in my head but I didn’t have any good category for it.

Well, at least I now know what to call it.  Gigaspaces calls this the ‘Data Grid.’  Other vendors have products that compete, from what I’m told.  I’ll have to dig.

Liking it.  Now, I need to find it on the MS stack…

Unit tests do not solve every problem… but they do solve a few…

By |2006-08-28T03:23:00+00:00August 28th, 2006|Enterprise Architecture|

a few days ago, I blogged about “what is the right level of detail to describe an interface,” and one good soul asked me to include unit tests.

OK.  I like Test Driven Development.  I really do.  I think it is an excellent practice that forces you to think in “specification completeness” terms, which is not something that comes naturally… it is learned.

But honestly, if I am communicating with a person who has not learned to think this way, then a unit test is code.  It is not a spec.  And if I am communicating with a person who is a program manager, and not a developer, then a unit test is code, not a spec.  And if I am communicating with a person who loves unit tests, but only in their favorite tool, then a unit test may cause “more heat than light.”  None of these three people will gain much from a unit test.

When I am communicating, I don’t always know who I am communicating with (like right now).  So unit tests, while valuable and useful for the person who is writing the code, and valuable and useful for the people who need to validate the code and maintain the code and even understand the code, are not always the best way to communicate about the code to an unknown but hightly technical audience.

Pictures.  Diagrams.  The friggin’ UML, for goodness sake.  (High fidelity diagrams).  I can, in four diagrams, illustrate more about an interface’s dynamic behavior than I can in a long list of unit tests. 

Let’s use TDD.  Let’s promote TDD.  But let’s know it’s limitations.

Contribute to Event Driven Architecture

By |2006-08-27T11:15:00+00:00August 27th, 2006|Enterprise Architecture|

I personally believe that Event Driven Architecture (EDA) is the future.  I don’t believe that SOA can deliver many of the promises attributed to it without a heavy dose of EDA.  Of course, to make this work, we need standards.

And that is where the Object Management Group’s SOA SIG comes in.  OMG has put out a Request for Information (RFI) on Event Driven Architecture to get feedback from the general community.  If you are passionate about SOA, I strongly suggest you start with Brenda Michelson’s excellent blog post on this RFI and go from there. 

Time is short.   If you wish to contribute, start today.

Say "no" to the ever-creeping meeting request

By |2006-08-26T04:44:00+00:00August 26th, 2006|Enterprise Architecture|

I spend way too much time in meetings.  I’ve been working myself ragged trying to keep up with my deliverables, and not able to get any of them done, mostly because I spend my days racing from one meeting to the next, often in different buildings, sometimes in different campuses.  (Microsoft has more than one in the Redmond area, with over 120 buildings).

This week, I came down with something.  A cold, I think, but my primary symptom was severe exhaustion.  I spent the first day at home mostly sleeping, but the next day at home catching up on my architectural deliverables, even though I was missing about five ‘important’ meetings.

And that’s when it really hit home… the meetings just were not that important.  It is OK to skip a few, or turn some down, if it means I can leave myself time to actually do the work they pay me to do.

I asked and some of the other architects have a standing set of “meetings” for three hours a day, three days a week, that they set aside to get work done… time when they cannot be easily ‘scheduled’ by someone who believes, perhaps with all the best intent, that they need an architect in the room.

But, to be honest, sometimes it is better for me, and the room, if I skip meeting #5 on the process for figuring out how to handle the return of a volume license from a company that didn’t buy one. (I made that up, but it sounds like the kind of timekiller I’m talking about).

Saying ‘yes’ to another meeting request means saying ‘no’ to my son who wants to ride a bicycle with me in the early evening, before the sun sets.  That’s the perspective I need to keep.

Clear responsibilities, clear leadership

By |2006-08-25T09:19:00+00:00August 25th, 2006|Enterprise Architecture|

As a company grows, new situations arise, and a great deal of effort will emerge to cope with the “new new problem.”  After a while, it is not a problem anymore.  However, as the company continues to grow, that ‘problem’ becomes an opportunity.

Example: let’s say you are in a small firm that makes pens.  You are competing with major players like Bic, so you have a niche market.  You make pens that are delivered to companies that personalize them for corporate marketing programs and conventions. 

Along comes a ‘problem.’  A salesman figures out that you can make a custom pen designed for cars that come pre-installed in the manufacturing process.  It is a problem because now the pen itself has to be customized, not just the writing.  You develop a process to handle this one-off situation and make some money.  Your sales team takes advantage, and with the blessing of the CEO, goes after every car company in the world to make custom pens to fit in cars. 

And, thus, a market is often borne by the ‘sales door.’ 

Of course, someone usually forgets to tell IT until after the first 50 orders are already in the door.  Turns out the fulfillment system has a problem.  It invoices the wrong people.  You need to add some data elements and rework the business rules a little.  An IT team launches a project, and you design the fixes.  Maybe you use an agile process.

Then the sales team starts adding features.  You see, they’ve been hankering for a CRM tool for about a year, and have been frustrated with the slow pace of the CRM project, which you are not working on.  You are working on the fulfillment system.

But they come to you and say “please add these features” and you spot, right off, that the capabilities they are asking for are not fulfillment.  They want the ability to track sales contacts against the customers.  The sales team is asking for CRM features “through the back door.”

What do you do?

If you add a few features, just to make them happy, you may slow down the CRM project substantially, because they now have to integrate with your data… data that is in the wrong place.  If you don’t, you can make your customer angry, and they are rising stars.  The CEO loves them right now.

This is where Enterprise Architecture is paid to step in and stop the insanity.  EA can review projects, find suggestions for overlapping functionality, and redirect the efforts. 

But for it to work, the dev team has to respect EA enough to call a halt to those features.  They have to back off on those, even though the EA member is not actually a part of the paid project team. 

And that is where you need clear leadership.  Clear committment to quality and excellent operations.  EA needs to be able to talk to the right people and have the right decisions made, and respected. 

A decision to do the right thing may be a difficult decision.  If handled well, it won’t hurt.  But if leadership is lacking, or committment to architecture is lacking, then Enterprise Architecture will be labeled as a noisemaker and either the customer or the dev team, or both, will write EA off as a waste of time and effort.

Don’t let that happen.

When you join a team, find the lines of leadership and establish your role.  Explain that you may be called upon to change the scope of overlapping projects, and that you will need their support.  Prepare the leaders.  The day will come when you will need them.

Just how to best describe an interface

By |2006-08-22T02:51:00+00:00August 22nd, 2006|Enterprise Architecture|

We have a pretty good solution for portal code interface in our team.  We’ve been using a home-grown portal for about a half-dozen years and it has grown to be fairly sophisticated.  We have role-based-security, page auth, object auth, data item auth, row level auth, and user attributes.  It’s pretty sophisticated.

Now, we have to describe it to another team that wants to understand it.  Problem is… it’s homegrown and most folks just use the modules that someone else wrote to do most of the basic things.  So it is not that easy to find a concise and descriptive document already in existence.

So we write one… right?

OK… what do we put in it?  I’m serious. 

If you have a code interface, and you need to describe it from a simple API standpoint, it may be simple enough to extract the documentation using .Net, but if you want to actually ‘describe’ it to a dev team so that they can consider the features of your interface, and perhaps put a few into their product, you need a much richer description than MSDN.

Interaction diagram.  Class diagram.  Some text describing the use cases.  Notes on data structures, dependencies, and type inheritance. 

That’s about 6 pages of description for a simple interface… 50 for a complex one.  Is that the right level of detail?  If you were a developer, and someone else had created an interface and you want to evaluate your tool, figure out how hard it may be to add the features from that interface into your library… how much detail would you want to see?

Creating a set of shared principles

By |2006-08-19T16:54:00+00:00August 19th, 2006|Enterprise Architecture|

Sometimes, it is easy to get people to agree.  That most frequently happens when everyone agrees with the criteria for deciding “what is right.”  If two passionate people both believe they are right, but they are right for different reasons with different results, agreement is impossible.

You have to make sure that everyone sees the problem in the same way.  That requires everyone to bubble up their requirements into the common ‘basket’ and to see the value in each other’s needs. 

This is the essence of what we call the “Common Requirements Vision.”  This is a document created mostly by listening to the needs of teams that are measured differently.  In creating this document, you are, in effect, creating a balanced scorecard.  There has to be a way that everyone can see the measures that matter to them and recognize the measures that matter to others.

More importantly, things matter for different reasons.  Juan may care about a particular application that he built, and may consider it a “requirement” that other teams use it in exactly the way he intended.  Amhad may care about about a particular approach to using technology, because he believes that the philosophy reduces costs in the long run.  Sofia may care about the ability to perform a specific business function efficiently, while Tina may care about providing a feature to the customers that she believes they will value.

Getting these different voices to agree on a particular solution is the apocrophyl “herding cats” but if you are a cat owner, as I have been for many years, it really isn’t difficult to get all your cats to come to the same place… simply open a can of cat food.  You’ve found something they all want in a way that they all expect to get it.

The common requirements vision provides a way to give the different voices a chance to be heard and valued.

Of course, you need a lens to measure the relative value of each of these ideas.  Juan’s care about his application is fine as long as that care aligns with the strategy and operations of the company.  If he is passionate about his home-grown inventory package, and the company is merging with another firm that has demonstrated excellence in supply chain controls, his application may need to take a back seat.  Getting him to agree to that won’t be easy.  You have to provide him with the lens that he needs to fairly evaluate the strategic value of his application.

You need to show Juan that each of the applications in a solution domain, like inventory control, have a fair shot at being considered.  People tend to like the idea of fairness.  Now find a measure that everyone agrees will be good for the company, like providing specific supply chain controls and supporting multiple currencies or supply chain networks.  Then allow him to make the call: how well does my little home-grown application, right now, today, meet the needs of the sophisticated organization that my company has become?

Only then, will Juan agree to release his application and step back and see the value of moving the entire company to a shared inventory solution.  Of course, I made this example up, but change the names and the solution domain, and you have a problem that is faced over and over… getting people who are emotionally opposed to doing the right thing to see the error of their ways.

What is that lens that will help this team of differently measured folks come together?  It is business strategy.  You have to help everyone measure their values against the stated and implict strategies of the business.  We did that with Juan.  We showed him that the strategy of the business was to get the best of the newly acquired company… by adopting their excellent mechanisms for supply chain controls. 

Sometimes business strategy, the reason for the big decisions, is lost in the shuffle.  Perhaps that business wasn’t purchased because they are great at supply chain control.  Perhaps they were purchased because, by purchasing them, the company removes a competitor or picks up a mature product to fill a gap in their product line.  In these cases, the processes and techniques used by the other company did not drive the acquisition.

On the other hand, the strategy of the company may still include excellence in supply chain controls, and the company being purchased may have particular strength in that area.  You may think it is a good idea to bring that strong suit to your company.  However, they may have built that strength by creating a culture of values that conflicts with yours.  Their values may have aligned with central control while your company may have valued independence of different business streams. 

This is also a strategy, albeit a cultural one.  The respect for the independence of each business area is a corporate strategy.  The fact that it doesn’t show up in the executive memos doesn’t mean you get to ignore it.  Nor should you assume it.

Write it down.  It is part of your common requirements vision.  “Strategy 4: Empower business leaders to solve their own problems quickly and independently.”  Make that opening verb strong.  “Empower” implies both commission and omission.  Providing a data driven business rules interface is commission.  You are creating opportunities for agility.  Building systems with simple connections that are easy to manage is omission.  You are preventing the business from getting mired in red tape when they want to implement something new.

You cannot create a consensus on the right direction without these common principles, aligned to strategy, that show a business how they work with you to achieve common goals. 

Facing uncertainty in your architecture

By |2006-08-18T05:40:00+00:00August 18th, 2006|Enterprise Architecture|

On OO design, we try to encapsulate the things that we think may change.  Whether it is creating an adapter or implementing a strategy pattern or even using an Inversion of Control framework like Spring.Net, the real goal is to isolate the “stuff that will change” from the “stuff that won’t.”

I try to do this at the architectural level as well.  In a real sense, that is what SOA is all about.  We use messages and contracts as a way of seperating two things… assumably these are things that are likely to change at different times and for different reasons.  SOA is really just OO applied to distributed systems.

However, it isn’t always easy.  Right now, I’m considering a number of options for a single application that is, in fact, the ‘point of a spear’.  The app has to live somewhere, literally on a server or two or two-dozen.  But where this app goes, dozens’ of other apps need to follow because they will have the same customers and will consume the same services. 

On the other hand, this bit of the infrastructure is not built yet.  I’d like to say that it was, but the app is beating the infrastructure out the door.  That’s just the way it is.  The business is driving the dates.  More than that, we still have some fundamental disagreements about some very basic things that are being worked out, so I can’t assume even simple things, like the number of domains the app will operate in, or the security infrastructure it will leverage, or even what data centers it may live in. 

This is the uncertaintly that we need to encapsulate.  It is pretty pervasive stuff.  The potential for screwing with the application is substantial (and appalling).

So we are doing what we always do… we are defining an interface.  On one side is the stuff that won’t change… the application itself.  On the other side is the infrastructure that will.  Across the interface flow the services that the app expects from its infrastructure. 

Now, I can put in an adapter that isolates the app from changes in the infrastructure.  Hopefully, I’ll get the interface right (or fairly close to right). 

As for the rest… we will just have to wait and see.

Sitting, silently, spinning

By |2006-08-16T02:04:00+00:00August 16th, 2006|Enterprise Architecture|

If a group of people look busy, but the real value that they are supposed to be producing doesn’t appear, they are in a pattern I like to call ‘sit and spin.’  The pattern is named after a children’s toy but could just as easily refer to an activity that my daughter likes to do… sit on Dad’s office chair and spin around until she’s dizzy.

Nothing gets done, but just look at her go!

We have all seen this happen.  Could be a single person, a team, a department, even an entire division.  Everyone is busy but strategic goals fall by the wayside.  The words in the management team meetings seem ‘right’ but the right stuff isn’t happening.  Credibility falls.  Morale drops.  Team members find other opportunities.  Deliverables fall apart.  It is sad.

What causes it?  How do you break free?

I have an idea…

I was reading a magazine on business opportunities recently, and saw an article about overcoming fear.  It stated the obvious: people may avoid doing the things they don’t like, but they will really run from the things they are afraid of.  The article went on to say that the trick is to recognize the fear, face it, and reward yourself for it. 

I think this is one of the main reasons that an individual may find themselves in the ‘sit and spin’ pattern.  Fear of going forward. 

So when it is not a person, but rather a team, is it still fear that binds them up?  Teams don’t ‘feel’ fear in the same way that a person does, but I’d suggest that fear can still infect a team.  Fear of failure.  Fear of looking bad.  Fear of uncertainty.  Many things…

If if you want to know where this fear is coming from, look to the top of the hierarchy.  An old expression that I subscribe to: “The fish stinks from the head down.”  If the team’s manager hits a wall of fear in their own life, in their own capabilities, then their team will begin to either reflect that fear, or revolt from that manager’s leadership.  It is inevitable. 

So if you look at yourself and your team, and you see ‘sit and spin,’ look deep.  Perhaps there is fear infecting your team.  Find it.  Face it.  Overcome it.  Reward yourself. 

Your team will thank you.

The gift that keeps on giving

By |2006-08-14T21:14:00+00:00August 14th, 2006|Enterprise Architecture|

Every now and then, an odd thing happens: a manager takes on a tough business problem.  (not so odd).  But the problem comes with a team that is supposed to be hand-picked to solve it.  Literally, the project shows up, resourced and all, and gets handed to a manager like a gift. 

That should be your first hint. 

Then the project starts, and sure enough, stuff starts to slide.  The team is doing what they say they will do, but the customers keep making noise about ‘concerns’ and ‘requirements’.  Before long, all the estimates and planning assumptions go right out the window. 

What’s wrong?  I can spin up a lot of reasons, but the one I’ve run into the most is this: the team doesn’t understand the business problem.  They may have a really good idea of the technical problem.  They may even have great people with great ideas that are passionate about delivery.  But if they don’t align to the business problem, then they are toast.

An astute PM may notice this.  They may be able to correct course and get things back on track.

If the PM is not astute, or is too novice to question, or too stubborn to admit defeat, or is a contractor/hired hand, they may just let the train keep rolling and hope that their bravery and iron will can save the day.  (It can’t.  It never can).  Net result: the team continues to try to solve the wrong problem.

This is the hidden risk in all projects: that the team doesn’t really understand the requirements.  I don’t mean the DEV team… I mean the ENTIRE team, including the analysts, program managers, engineers, testers, and so on.  In short: everyone. 

Here’s an example.  Note: this is not indicative of any current MS IT project.  This is from past experience in my consulting days.  However, I’ve seen this behavior before and expect to see it again.

Let’s start with our team.  Just for fun, let’s call them Epsilon.

Team Epsilon is chartered to create a tool that many internal customers need to share.  (Marketing is a frequent victim of these kinds of things, unfortunately).   Let’s say that Customer X gives 30 requirements and that Customer Y gives 40.  They only overlap a bit.  So Team Epsilon creates a plan, and they agreed to roll out only the overlapping part in phase one.

Bad move.  Now, the overlapping part is not enough to satisfy either customer.  So Team Epsilon publishes their plan, and both customers balk.  Neither one wants to pay for phase one.

There are many different outcomes to this story. 

  1. The PM changes course, and they decide to build for Customer X first, and then add on Customer Y second. 
  2. A new approach gets Customer X and Y to agree on a subset that would actually be useful to one or both. 
  3. The entire project gets outsourced (which is another way of saying, “fire the team and start over.”)
  4. Here’s the one I started with: What if the entire team is moved to a new manager? That will solve the problem… right?

The new manager takes on the team, flaws and all, and agrees to solve the problem.  He doesn’t really know what the problem is, so he talks to the team.  They give him their view, but it is precisely that view of the problem that prevents its solution.

Of course, this little bit of deck-chair-shuffling doesn’t solve the business problem either.   The manager who got rid of the team solved HIS problem… he’s not getting noise from Customer X and Customer Y anymore.  The customers are still in the same situation. 

So if you are a manager, and someone hands you a problem along with the team to solve it, it is OK to accept the problem, but if you do, take my advice: Dissolve the team into your organization.  Form a new team, with a proven track record.  Then, start over. 

Most importantly, question everything.  Get customers in the room, without the old team present (or with the new team, sitting quietly).  Listen.  Ask follow-up questions.  Find out what is going on. 

Sometimes, the tree needs to be shaken before the fruit can fall.