“Mastering the Unpredictable” is finally out in its official edition – available in hardcopy and Kindle.Kudos to Keith Swenson who did an amazing job in getting the book out in record time.

Since I spend a lot of my time explaining to people what unstructured, unpredictable ad-hoc process are used for, I give a couple of simple everday examples trying to explain the importance (and ubiquity) of these processes:

1. Complex Project Management – there are many excellent tools out there to help plan projects and keep them on track, but in most cases for project execution, tracking and management everyone falls back on the tried and true methods of email and documents. This is true for every type of project – but especially those done by IT professionals. There is a project owner who defines specific project related goals and checklists, but that is it – everything else is managed on the fly by the participants themsleves and guided by the project owner.  Adaptive Case Management can help manage that part of the project management processes (and BTW this is a standard ActionBase usage scenario).

I think this is reason the BPM community is so “hot” on Social BPM – they’ve discovered the hard way that there a class of processes (in their case business process modeling) that is an inherently unstructured, unpredictable and hard to manage project.  What they don’t seem to understand yet is that there is nothing special about BPM implementation processes (modeling or otherwise) - the same issues hold for any project, but I won’t belabor this here since I have posted on it in the past.

2. Executive Management Decision Tracking – For example at a Board of Directors meeting any number of decisions are made that require a follow-on process. From the executive level it can seem like a simple request (”get us the information on XXX” – e.g. the state of our audit processes), but that request quite often turns into a full-fledged, ad-hoc process that can involve quite a few people and needs to be managed. Again the tools of preference for executing these processes are email and documents. No one would even consider modeling the process, especially since it probably isn’t completely understood until it starts executing (making it true “Design by Doing”). The best one can hope for is an owner, goal and maybe a dynamic checklist (and of course an owner). 

Many high-level, complex project management  scenarios are similar to unpredictable, ad-hoc process and would benefit from using adaptive case mangement. I wonder – Is this the generic use case for Adaptive Case Management which is then tailored to specific domain?

I read a really interesting article by William Gladwell on “How David Beats Goliath” which got me thinking about what would happen to business if we could actually accelerate the decision processes done by knowledge workers. Most analysts now agree that knowledge worker processes make up the bulk of processes for most businesses, and for the most part the only tools IT provides for them in their tasks email and documents (and maybe wikis and IM). What would happen if we could magically increase the speed in which this type of work could be done? What would be the ramifications for business? What would be the benefits that could be reaped over competition?

I think everyone would agree that a company that could speed up these critical knowledge workers decision processes would have an “unfair” advantage over its competitors. So what is needed to make this a reality?

A simpler version of this problem can be found in performance problem resolution for complex distributed computer systems. The problem arises because today’s computer systems are so distributed, with so many components that interact in unanticipated ways and with so many transactions flowing through them – performance problems are a bear to solve.

Finding the root cause of distributed application performance problems is a very, very hard problem – and that is for systems much less complex (and less dynamic) in their interactions than human processes.  Not surprisingly, it turns out that the first step in solving such performance problem is having the right data about the actual transaction flow. The premise is that if you collect all the information of how a transaction progresses along its path (both as it happens and a historical view) – then solving a performance problem becomes much, much  easier. The intuition behind this is simple – the best way to fix a performance problem is to understand its root cause, and the best way to find a root cause is to see the problem while it is happening. If you don’t see it “in-vivo” while it is happening – then it becomes a detective task – you only get to see the symptoms of the original problem, and need to derive from those the root cause. Like the TV show CSI – a show that derives all of its interest from how smart people unravel a mystery from its symptoms (or after effects) without the benefit of an eyewitness. It is so interesting because deriving a root cause from symptoms is hard (and the source of many a TV show – e.g. House MD, Law and Order, Fringe). The reason the problem is so hard (and the show so interesting) is that they are studying the after effect of an event, and trying to ascertain the root cause. This blood splatter meant that, this rip was caused by the other. Contrast that to an eyewitness report – I saw X pick up the knife and stab Y. Would make for a really effective investigation (and a really boring show).  Same is true for performance problem resolution in computer systems. If you have the right “eye-witness” data, then in many cases analyzing and fixing the problem is a piece of cake, without the right data – it is a bear.

Performance problem technology went through some interesting phases – first came the sophisticated analysis techniques to help solve the problem (the CSI approach) and now we are seeing new technology and approaches that are focused on collecting better, more effective data (the eye-witness approach) – since examining only symptoms causes root cause analysis to be very expensive, while an eye-witness approach is much more effective (and cheaper).

 The same holds true for knowledge processes. If we want to improve knowledge worker process – the first thing we need to do is collect the right data – not after the fact symptomatic, biased data – but true “in-vivo” data of how the process is actually performed. Only then can we be effective at resolving the problems and speeding up the process. If we could view knowledge worker decision processes as they are executed – then most of the difficulty involved with speeding up such a process (or fixing a broken one) would go away… It would be simple to see where any given decision process stands at any given time, and to understand whether it is healthy or not.

I have been doing an experiment lately. Whenever I talk to a group of people about Adaptive Case Management (or Human Process Management) with knowedge workers that are familiar with BPM  – I ask “How many people present use BPM or a system built on BPM for their everyday work?”. Everyone at these meetings is a knowledge worker and (some of them even develop BPM tools and systems) . Never has more than one person raised their hand. Then I ask who uses email and documents every day, and of course everyone raises their hand. If you ask them how many of them do structured, repeatable, routine work eveyday – then the usual answer is that is always a part of their job, but not what they are paid for.

I know this is kind of a stupid experiment, but it does point out (at least for me) the current state of tooling for most knowledge workers. They consider their work (at least the important part of it) as “Design by Doing” as Jim Sinur so nicely worded it in his blog. BPM as they understand it (and I’m sure that everyone understands it a bit differently – but I’ll guess the ability to model a process, most probably in BPMN, is part of what they think is involved in BPM) has nothing to do with their everday jobs.

Sure we could try and extend BPMS (the tool) to include the kind of work they do, perhaps even extend it enough to have it include email and document management systems (or is it that document management systems include BPMS, I get confused). But why should we? Most BPM systems have hooks that let the system invoke email (and so do document management systems) – but I would doubt most people would consider email as part of BPM. BPM and email are two separate systems which are sometime used in unison to solve a business problem – there is no need to pull them both under the same umbrella. I think the same is true of Wikis and other social technologies – they are useful tools – and there is no benefit pulling them under the BPM umbrella. So in the on going debate on ACM as part of BPM – I’m going to have to side with Keith and Max.

BPMS (the tool, not the approach) is for routine, predictable, structured work where people and documents may play a part – but are not central. ACM is for unpredictable, unstructured, ad-hoc work – where people and documents are central. The two are complementary, but not equivalent. To prove my point, I’ll go back to my old post - no one is building social BPMS tools using BPMS - not even the BPMS vendors.

My presentation from our chapter in the book “Mastering the Unpredictable”

Yeah, I know – it was the process.gov conference, but my real interest was the Adaptive Case Management day (and the launching of the “Mastering the Unpredictable” book). Many thanks to Nataniel Palmer and Keith Swenson for setting it up.

1. My first take away is that adaptive case management covers a pretty wide swath of  approaches. Looking back at Keith Swenson’s chart on “Four Process Trends and 1 Gap” – the gap between human workflow and email is quite large – and some look at ACM as closer to the human workflow side of the spectrum, and we look it at as closer to email. That is OK, and will make for a rich set of tools, each with it own focus – but can cause people to think that we aren’t all talking about the same thing. I believe that all of ACMers are addressing the same business issue (the need for managing ad-hoc, unstructured, unpredictable human processes) – we just are going at the problem from different angles. I think all of the authors agree on that.

2. People have no problem with the notion of ad-hoc, unpredictable processes - almost everyone  intuitively gets it right away.

3. The BPM folks sometimes have a difficult time with the notion that anything that has to do with processes may be outside the scope of BPM. I think everyone continually confuses business process management with what can be done with a BPMS. Since ACM handles business processes of a certain type, ACM is arguably part of business process management, but it  is clear that it is different than BPMS, and needs a different set of technologies and approaches. If you are interested in why – read the book “Mastering the Unpredictable”

4. The type of processes that lend themselves to ACM tend to be a lot like projects.

5. We need to find a wider audience to participate in the ACM community – and I am hoping that we can find business users that would like to become part of this community.

There is a lot of buzz around what is coming in business process management (BPM), a lot of it seems to stem from the social technologies that are starting to gain a foothold in business, and the desire to understand how they will effect BPM. I actually think that issues with the current state of the art in BPM go deeper – it is just that the emergence of social technologies is bringing them to the forefront, and giving people a common understanding to discuss these changes. I think the real problem is that BPM is becoming too complex and is on its way to becoming even more complex – people are even asking whether BPM is becoming the next programming paradigm, or whether it will be used as a basis to replace all applications in the enterprise.

The are reasons for the complexity, mainly as an attempt to make BPM applicable to more types of business processes. Examples of the increasing complexity are that BPMN has become more complex as a modeling language, vendors are adding rules, complex event processing and process simulators (just take a look at Jim Sinur’s list of the Top Ten BPM Technologies - all valuable stuff, but adding complexity). It seems to me that BPM as a technologyhas moved away from the  business side, and is going deeper into the technology side. I think that this growing complexity is what will eventually stop BPM from growing (Clay Shirky has a great post on why complexity destroys - The Collapse of Complex Business Models) – though I wouldn’t go so far as to claim the complexity will cause BPM to implode, it will just stop its growth.

That is why I am happy that the WfMC positioned Adaptive Case Management (ACM) as separate from BPM - trying to extend the BPMS umbrella to handle unstructured, unpredictabel human processes is a mistake. The two process types are very different, though complementary. They require a different understanding of the world, different tools and a different focus. Just like in Clay’s article the AT&T folks just couldn’t understand the messy world of web hosting, I think many BPMS vendors will find themselves unable to reconcile themselves with the messy world of ACM.

An interesting blog post by Keith Swenson on unpredictable processes got me thinking about the use of a model in BPM. People use the term “to map the process” as a way to describe what is done during process discovery, and as the result of the discovery process. After having thought about it, using the term “map” is misleading, since the result is usually a set of codified directions – not a map.

A map is used to show the lay of the land, and existing obstacles. It doesn’t give direction (or focus on a particular path) – just shows what is out there (for better or worse), at various level of detail. For any specific navigation task, most of what is shown on the map isn’t relevant and it is up the person using the map to find the appropriate path. The more structured the terrain (e.g. an urban area, versus out in the wild) the easier it is to use automation to find a path. But in many circumstances the map doesn’t contain all the relevant details – especially about movable obstacles (e.g. where construction is happening right now, where there is an accident, where the lions are located at this very minute) – so you can’t always rely on the directions. As the terrain becomes less structured – the less detailed the directions that can be given, and navigating the trail relies more on the knowledge and skill of the participant.

Most BPMN models provide something closer to directions (since they don’t map all the terrain, but rather the process as it should be executed), not maps – which is why they work for routine processes, but are inappropriate for unstructured, unpredictable human processes. For unstructured, unpredictable, human work the best you can do is a very high level description (Guidelines, Best Practices, Checklist) – not a detailed model.

When looking the emergent, unpredictable, human processes that lend themselves to adaptive\dynamic case management,  we have found that many of these processes get into trouble at the point of handoff between participants in the process. That is when instructions are misunderstood, deadlines are misinterpreted, follow-ups are missed, steps are skipped.

That is very different from BPMS managed processes – there the model can keep things on track. But for emergent, ad-hoc, unstructured processes there is no model to keep things on track. A guideline or best practice can help and provide a framework, but can’t ensure successful hand-offs. So can tracking, reminders, notifications and social conventions. Ensuring there is an owner also helps – as do the social conventions people adhere to in the workplace (especially when their actions are viewed by others). But in the end, it is up to the participants themselves (and mainly the process owner) to ensure successful hand-offs. A key key feature in any dynamic adaptive case management system is how it handles hand-offs.

I’d be really interested in hearing whether this matches with other peoples experience too.

I was reading an article in the Economist (Data, data everywhere) about the growing abudance of digital data, and the problems caused by the need (desire?) to analyze all that data and extract usable information. That got me thinking about why the capture of the data in digital format (the data existed before, just no one was capturing it – sort of as the same issue as with the email filter problem) is causing so much angst. Then it hit me – the capture of this data is missing something- the context of the process involved in creating the data. The data captured after the fact is a jumble of direct information, symptoms and side-effects caused by the process of interest. So looking just at the data is the equivalent of doing forensic analysis of the aftermath of an unknown process – and trying to piece together the process (this is very hard to do, and what makes CSI such a compelling show to watch). If there was someone watching the actual events as they unfold (or the actual “murder” process) they could just say – this mess was caused by A shooting B and then running away – and understanding the resulting data (e.g. blood splats on the wall, shell casings on the floor) would be much simpler (and CSI would be a very boring show).

Of course, we can’t track the process for many kinds of data (e.g. a star exploding in astrophysics, or customer buying patterns in a physical store) – so we have no choice but looking at after the fact data and doing forensic analysis. But for any process that you can directly watch and provide the eyewitness view – the amount of data analysis needed goes down drastically, since the context is known.

Bottom line,  if we know the context – understanding the process becomes much easier, in many cases almost trivial. Not having an actual birds eye view of the existing process one of the key issues in trying to model actual business processes in any organization (especially since most interesting business processes involve human participants). Modeling a business process is similar to forensic analysis – finding the participants, interviewing them, looking for clues and artifacts, creating tentative models, simulation.  Hard, interesting work. Maybe we could create a show “Business Process Investigation”…

This is where adaptive case management (ACM)  tools can be invaluable – since they can start managing a process without a-priori modeling. But how do you get people to start using an ACM tool – since they would need change the way they do the process first – not something they would embrace. This is where doing ACM in existing email and document environments can really make a difference – users can remain in their regular environment – with the underlying process system-of-record proving the “eye-witness”  birds eye view of what actually takes place. Then if is there is still the need to model the process – it becomes much, much easier.

When we talk about managing unstructured, ad-hoc process (or using the adaptive case management terminology – knowledge process), one thing we always discuss is the notion of a best practice – which is a framework which describes a general outline of the work to be done (what needs to be done, not how to do it). In many cases we have seen these best practices boil down to a checklist (or a series of checklists).

Turns out that the lowly checklist is a very valuable tool for managing complex, unstructured ad-hoc processes in many domains. I was reading an excellent article on the benefits of checklists in unstructured processes from an old article in “The New Yorker“ by Atul Gawande, that was referred by Jon Udell’s blog post on “Atul Gawande on why heroes use checklists“.