Rob England: Incident Management at Cherry Valley, Illinois

It had been raining for days in and around Rockford, Illinois that Friday afternoon in 2009, some of the heaviest rain locals had ever seen. Around 7:30 that night, people in Cherry Valley – a nearby dormitory suburb – began calling various emergency services: the water that had been flooding the road and tracks had broken through the Canadian National railroad’s line, washing away the trackbed.

An hour later, in driving rain, freight train U70691-18 came through the level crossing in Cherry Valley at 36 m.p.h, pulling 114 cars (wagons) mostly full of fuel ethanol – 8 million litres of it – bound for Chicago. Although ten cross-ties (sleepers) dangled in mid air above running water just beyond the crossing, somehow two locomotives and about half the train bounced across the breach before a rail weld fractured and cars began derailing. As the train tore in half the brakes went into emergency stop. 19 ethanol tank-cars derailed, 13 of them breaching and catching fire.

In a future article we will look at the story behind why one person waiting in a car at the Cherry Valley crossing died in the resulting conflagration, 600 homes were evacuated and $7.9M in damages were caused.

Today we will be focused on the rail traffic controller (RTC) who was the on-duty train dispatcher at the CN‘s Southern Operations Control Center in Homewood, Illinois. We won’t be concerned for now with the RTC’s role in the accident: we will talk about that next time. For now, we are interested in what he and his colleagues had to do after the accident.

While firemen battled to prevent the other cars going up in what could have been the mother of all ethanol fires, and paramedics dealt with the dead and injured, and police struggled to evacuate houses and deal with the road traffic chaos – all in torrential rain and widespread surface flooding – the RTC sat in a silent heated office 100 miles away watching computer monitors. All hell was breaking loose there too. Some of the heaviest rail traffic in the world – most of it freight – flows through and around Chicago; and one of the major arteries had just closed.

Back in an earlier article we talked about the services of a railroad. One of the major services is delivering goods, on time. Nobody likes to store materials if they can help it: railroads deliver “just in time”, such as giant ethanol trains, and the “hotshot” trans-continental double-stack container trains with nine locomotives that get rail-fans like me all excited. Some of the goods carried are perishables: fruit and vegetables from California, stock and meat from the midwest, all flowing east to the population centres of the USA.

The railroad had made commitments regarding the delivery of those goods: what we would call Service Level Targets. Those SLTs were enshrined in contractual arrangements – Service Level Agreements – with penalty clauses. And now trains were late: SLTs were being breached.

A number of RTCs and other staff in Homewood switched into familiar routines:

  • The US rail network is complex – a true network. Trains were scheduled to alternate routes, and traffic on those routes was closed up as tightly bunched together as the rules allowed to create extra capacity.
  • Partner managers got on the phone to the Union Pacific and BNSF railroads to negotiate capacity on their lines under reciprocal agreements already in place for situations just such as this one.
  • Customer relations staff called clients to negotiate new delivery times.
  • Traffic managers searched rail yard inventories for alternate stock of ethanol, that could be delivered early.
  • Crew managers told crews to pick up their trains in new locations and organised transport to get them there.

Fairly quickly, service was restored: oranges got squeezed in Manhatten, pigs and cows went to their deaths, and corn hootch got burnt in cars instead of all over the road in Cherry Valley.

This is Incident Management.

None of it had anything to do with what was happening in the little piece of hell that Cherry Valley had become. The people in heavy waterproofs, hi-viz and helmets, splashing around in the dark and rain, saving lives and property and trying to restore some semblance of local order – that’s not Incident Management.

At least I don’t think it is. I think they had a problem.

An incident is an interruption to service and a problem is an underlying cause of incidents. Incident Management is concerned with the restoration of expected levels of service to the users. Problem Management is concerned with removing the underlying causes.

To me that is a simple definition that works well. If you read the books and listen to the pundits you will get more complex models that seem to imply that everything done until trains once more rolled smoothly though Cherry Valley is Incident Management. I beg to differ. If the customer gets steak and orange juice then Cherry Valley could be still burning for all they care: Incident Management has met its goals.

Image Credit

Do SLAs hinder collaborative relationships with our supply chain?

Pretty much all outsourcing contracts in the IT Service Management world rely on, or at the very least, utilise the Service Level Agreement (SLA).

Certainly they are important as they are the physical representation of performance of the contracting party and used as the measure by which trends in supplier performance is understood.

But is there too much reliance on SLAs as a measure of performance and are they often inserted by the eager contract or procurement manager to mitigate risk or provide a means for the insertion of penalty / reward clauses because “that is what is expected in a contract”?

In my personal experience SLAs are often poorly defined or their alignment to the realities of IT service delivery misunderstood.  Because of that, there has been many mitigating circumstances offered as to why an SLA has been failed by the contracted organisation, followed with significant discussion as to whether the mitigation can be accepted.  This has a tendency to suck up both time, effort and therefore money, from both organisations into managing the performance measures, drafting contract change notices and often not looking at the root cause of why SLAs are being missed or in one case I have dealt with perpetually exceeded (plainly in that case the SLAs were too generous or measuring the wrong outcomes).

Problem 1

The vendor will look to win the opportunity and subsequently concern themselves with making the delivery side work (especially when the bid team is not going to be involved in delivery).  They will obviously try their utmost to meet the targets set, but also expect to provide mitigations in the event of failed SLAs.  They have the experience of dealing with a number of clients and so have reference points to support them, whereas the contracting organisation does not have the same level of experience or number of reference points.

A common resolution is the instigation of a continuous performance improvement plan, and when that has been met redrafting of workable SLAs agreed by both parties, or if it fails penalty clauses or litigation.

Problem 2

Poorly defined requirements from the customer.  Either they are unsure what they want, have over / under specified the level of service really needed, they are looking to outsource a problem, or the business units have been poorly engaged, if at all, by procurement through the tender process.  In such circumstances the supplier is almost being set up to fail from the outset (which from their experience they will probably realise) and therefore they will look to manage their way around the issues as they arise.

The common resolution is a redefinition of the SLAs probably with an element of contract renegotiation once the customer has determined the service it expects or requires.

So what should a good SLA really be about?  A well constructed SLA should be seen as an important measure to support a positive contractual relationship, it should also be periodically reviewed for its applicability in light of changing business demand.  However, the SLA should not replace or overshadow the development of the relationship between the customer and supplier.  Rather, the SLAs in place should support a collaborative attitude towards delivering a contract outcome that benefits both parties; the customer receives the service they need, the supplier makes the profit margin they expected and the customer is satisfied with.

Neither party should be wasting time and money negotiating mitigations, instead the time saved can be spent on delivering future value.  Unfortunately developing proper, mutually beneficial collaborative relationships in a business environment is not easy where customer and supplier aspirations are not aligned.