I have to credit my manager for this gem. And it is easy to see why it is true. If you have an application, especially one so dependent on services as a SOBA, any problem in the application will first be blamed on the code that is ‘not invented here.’ It is sad, but true.
So what do we make of this… we who think that Services can provides some of the flexibility that integrated systems so desperately need? What is the action item?
All services must be instrumented to show when they are working, when they are failing, and what they are doing. There must be the ability to track a message, from end to end, in a process to show where the message sits, where it drops, and to provide the peculiars of the message itself (since it is quite normal for this behavior to only happen for messages that look like X, but not Y, even though X only happens once in a blue moon.
In effect, if you are a services developer, and you develop a service that you cannot PROVE is up and running, then your service will be blamed whenever anything fails. Services that cannot be instrumented are the work of hobbyists, not professional developers.
So what do you need to prove?
- Life – the service must respond to a ‘Alive’ poll very quickly.
- Health – the service must check downstream dependencies and return with a metric showing that it does or does not believe that it is functionally able to perform based on whether downstream resources are available.
- Throughput – the service must both log performance data as well as return performnce data for the last N transactions. This will allow a seperate system to watch a message go through and then ask, after the fact, how long that transaction took to complete.
- Message status – for each message inbound, the state of the message is returned. This allows an observer to determine if a message is in process or has completed. Status information needs to be detailed enough to determine which component the message was last sent to and, hopefully, if there are conditions that must be fulfilled for the message to continue moving.
Until you can answer these questions, the service is guilty of every sin imaginable.