Tuesday, June 30, 2015

Security is the Boat Anchor to Digital Progress

Have you ever tried to row a boat whose anchor is still stuck on the bottom of the lake or  the ocean? It's really a fruitless and frustrating experience. This is how I would describe the effect that the lack of great and easy security is having on the whole digital experience today. In the coming weeks, I expect to dig a little deeper into the nature of the problems and write specifically how the lack of easy and great security has on our progress toward a better digital experience enabled by great process leveraging cognitive resources. Security is holding back a better customer experience, a more powerful and cognitive set of processes and process sharing to enable seamless value / supply chains.

Security and the Customer Experience:

Why in the world do we have to remember user IDs and passwords for each organization that we do business with these days? Every organization has a different set of rules for passwords ergo causing us to remember IDs and passwords galore. To fight this and weak memory moments, we create lists of IDs and passwords and squirrel them away in some remote spot on our computers or paper file drawers which have their own security risks. This is pure insanity and starts us off with a sour attitude towards the customer experience nearly every organization we do business with even though the processes might be OK. This sets a lousy tone for the whole customer experience. Add the fact the processes don't really know what we want and how we want to be treated. and you have a mess. You can make this misery mobile as a first step, but the point is that the customer journey starts off on a bad foot. It's time for a change !!

Security and the Powerful Processes:

Why in the world should we have to deal with dumb processes with a simple mobile access that knows nothing about us and our preferences? Processes should know who we are based on additional data, policies and rules we give organizations. Based on our history with an organization, the processes should orient themselves around us and give us the knowledge to service ourselves or be serviced by representatives that are not so specialized as to have us pick the right department to work with for results. This kind of powerful process will create a higher promoter score for the organizations. This again is dependent on better security of IDs, passwords, data and rules. It's time for a change !!

Security and Shared Processes: 

Why in the world do we have to stitch together sub-processes and the manage the result of each to contribute to a set of end goals and outcomes? As cognitive and intelligent processes get shared across organizational lines (internal or external), processes will have to have bullet proof security protecting all the policies, data, rules, constraints and authorizations. As end to end processes act in a seamless manner across legal entities, the value of air tight security raises to a critical level. When this includes shared resources such as intelligent devices like smart sensors,  robots or cognitive services (COGS) in the cloud, security is one big issue to wrestle with in the future. This kind of process is starting to emerge in adaptive case management and large scoped end to end processes.
It's time for a change !!

Net; Net:

We have been sweeping security under the rug like that fact that we left the anchor out on our new digital boat. We can't row our way out of this one. It's time to pull the anchor and make some progress navigating the new digital sea. Why can't security become easy and fool proof?

Monday, June 29, 2015

The ROI of Illuminating the Digital Darkness

Dark events, data, processes and patterns all have enormous potential to add new insights into how an organization approaches its mission. Spending some time finding dark events can pay back operationally in a quick fashion, but these events and patterns can help make better decisions and avoid risks. 

It may be But as with any source of information, the value it generates depends upon a number of factors that generally focus on the value of the end result of some desired action, outcome or provides a more detailed content about a known event.  In this way, it is revealing the “unknown within the known event” adding significant richness that can provide real value to decision makers as well as automated rule-based decision systems.   The example described below uses the depiction described in Dark Event Post # 2 and walks through a simple example of how unproductive time can be both discovered and re-covered. 

As mentioned previously, we can see in the Figure 1 below the known steps (shaded in light blue) in a typical transaction system.  However, between steps 4 and 5 (Transaction Paid) are a number of other steps that are happening (shaded in gray) that detail a customer service intervention process that corrects the data on the customer transaction so it can process to conclusion.

Figure 1

So, the question some questions we might ask here are:

·         How much time did each newly discovered event consume?

·         Were all actions taken in compliance with standards?

·         Were there other actions taken that are not part of the process?

The answers to these questions can be found by examining the new event log, either created from new collectors or through an “event mining” exercise, that includes the time spent on all events, including the newly illuminated dark events.  When we look at Figure 2 below we see that a total of three customer records were reviewed and updated for a total time of 30 minutes, however in that time-frame, a number of different issues were addressed, some with potential ramifications as described below.

Figure 2

Specifically, we can see:

  1.         In the first record addressed, the agent used Google Maps to look up and validate the address.  This may be privacy violation that can be quite costly to the organization. 
  2.         In the second situation, the agent used instant messaging to contact another unit within the organization validated the address.  This is the correct procedure; the communication remained inside the secure firewall however the action was not captured by the transaction system.
  3.        In the third case, the agent used the correct procedure to update the balance, however, the time was increased by 15 minutes due to time being spent on the ESPN NCAA Bracket Challenge. This is an instance of “empty labor” that would be normally be seen as active transaction time. 

Here we see that two of the three cases involve situations where the actions either expose the organization to some type of risk that can drain value from the organization.  I call these situations Value Leaks

As mentioned earlier, unless they are truly random, business events are encapsulated by business processes.  The processes in turn are designed to provide some type of value to the business to the organization that is quantified by some means (number of widgets per hour, etc…) and converted to a monetary format (Dollars, Euros, Yen, etc..).  Therefore, each process has an expected value it needs to deliver to the organization.  When it is delivering something less than that expectation, it is in essence “leaking “ some of that expected value and is therefore a value leak.  Value leaks can also be seen as waste (or in lean terms muda), which adds costs and drains value from a process.  A good example is the U.S. healthcare system which has an estimated value of $2.7 Trillion per year, of which $1.35 Trillion had been attributed to waste!  On a smaller scale, we can see in Figure 2 above that of the time between 8:00 AM and 8:30 AM, 20 out of the 30 minutes of work time were spent on activities other that those that add value.  While this is normal a rather trivial example, imagine that event occurring multiple time a day and not just to one agent but to 1,000 agent or more!  In Figure 3 below we can see the potential value of improvements in value leak reduction, not including the potential leak associated with the potential privacy violation uncovered with the first record being reviewed!

                                                                            Figure 3

Net; Net: 

Dark events can be costly in multiple ways.  The effort to understand dark events and the associated value leaks represent pays for itself many  times over and in some instances,  can be the catalyst for major productivity and profitability gains. If you have questions or a need to discuss this ROI opportunity further contact Dr Edward M L Peters whom I thank for sharing this series with me and helping folks understand the opportunities and threats involved with Dark Events.

Thursday, June 25, 2015

2015 BPM & Case Management Conference

There was a strong surge of energy in this conference which has grown 20% over last years inaugural conference. The content was rich for all in the audience. If your angle was linking new digital efforts to business architecture, there was a rich vein of content. If your interest was in modeling there was a strong thread of modeling sessions around process, case and decision modeling. If your interest was around emerging digital technologies such as big data, the IoT, robotics, process mining, machine intelligence and the man/machine interface, there were sessions to please. If you wanted success stories, they were there. If you wanted to understand the opportunities in case management & BPM, there were sessions to please. The content available at this conference was as rich as any I have seen to date. I will highlight some of my favorite sessions and hope you might find the descriptions useful.

Linking Business Architecture to Processes:

There was a keynote session presented by Mathias Kirchmer on his BPM - D method for identifying the processes that differentiate an organization and how to link them to results and outcomes. While there were many great sessions on business architecture, this session emphasized completing the architecture in a more complete way for business differentiation. The most interesting tool vendor, for me, was Trisotech as they had a dynamic integrating tool that leveraged a repository that had the links between architecture and execution.

Process, Case & Decision Modeling:

There were several sessions describing how to model and integrate standard modeling notations (BPMN, CMM & DMN). For organizations starting their BPM/Case journey, modeling is a common entry point and generally leads to a linking to business architecture over time.

Emerging Digital Technologies:

One of my favorite sessions was given by Dr Chuck Webster on the linkage of medical processes with the IoT by leveraging wearable technologies. Chuck went on to explain how devices could be attached to patients for real time status and how wearables + RFID could be used to manage hand cleaning and other disease prevention. Chuck the pulled out his pocket robot, Mr RIMP, to show how robots could be used in a pediatric setting to settle kids. He showed a video of interactions with Mr RIMP that was hilarious. https://youtu.be/ull1cojGouA

I really enjoyed the process mining session by Dr Anne Rozinart of Fluxicon on using visual and animated cases that run through a discovered process model by measuring actual paths of cases/ process instances. This way the real deviations from the planned model and actual execution can be tracked and choke points can be identified.

An enticing session was presented by Quantiply and one of their clients (to remain anonymous) where big data was leveraged to identify real time events and event patterns to react to problems and even predict where issues my occur. Interesting.

Personal Touches: 

I really enjoyed hanging with my son David, Tim Price and Chuck Webster who put together a mini synthesizer leveraging a little bits kit (a Lego approach to creating composite electronic devices). Tim tried to play "Stairway to Heaven" with some success :) See below for a pic of Dave and Chuck

I was also given the 2015 Marvin L. Manheim Award For Significant Contributions in the Field of Workflow. It was a very nice recognition of my passion to drive BPM & Case into the new digital organizations.

Tuesday, June 23, 2015

Blog Activity for 2Q 2015

I thought you might find it interesting what folks are interested in reading about process related topics on my blog. It is interesting to me to see what gets hit the hardest, so I can plan future postings for you. The top posts were the Internet of things and decision management, but the hot themes were around digital transformation, a better customer experience and swarming processes. The up and coming subjects were around dark events and digital platforms. 

Where in the world are the hits coming from?  The US is always number one by a wide margin and is not shown here. There is no surprise that France, Russia & Germany are at the top, but I am surprised that Japan is off the chart. 

Monday, June 22, 2015

Dark Patterns: Recognizing Them in Real Time

We all know that events are firing off in many directions at once and the amount of events to sort through is mind boggling. In fact for every event you see; there are, at least, an order of magnitude number of events occurring for every one you see and this does not take into consideration the events you don't see (aka Dark Events).  It's a bit of an iceberg phenomena. What can help sort out the importance is to look for patterns of multiple events occurring over time and leveraging those patterns for next best action based on opportunity or threat analysis? Until big data, in memory, this was a difficult, if not impossible approach to take. Ahhh, the advantage of emerging digital technologies for business.

Seeking Patterns of Interest:

There are many noise events and patterns occurring at the same time there are emerging patterns that may mean something to an organization. Just like scanning headlines in the news, you ignore many individual articles. However when certain trends start to emerge, new events and patterns of connected events now mean something. This is true of organizations as well. Identifying emerging patterns of interest or targeting the size and shape of patterns that will require an organizations attention will be a growing field over time as we get into a predictive mode of business instead of our reactive habits.  Mining patterns will be a new set of skills organizations will need to have over time.

Connecting Patterns to Context:

Collecting interesting patterns can be fun, but organizations need to have a purpose that directly or indirectly affect them. This means that patterns of events need to matched against known opportunities and threats at a maximum or analyzed for next best actions. Matching against known patterns is the easy job, which may require multiple patterns to present or emerging. The real challenge is to recognize the value or non-value of an emerging set of events or patterns fast enough to make a difference. Of course there will be the normal analysis after the fact, but the premium is on near real time. Matching patterns to opportunities and threats will also be a new set of skills organizations will have to cultivate over time. There will be noise patterns just like noise events.

Connecting Patterns to Smart Actions:

So a pattern not only looks interesting, but it will require action, specifically associating the right actions that correspond to patterns. In the case of expected patterns, actions can be inventoried and put into action quickly. Many savvy organizations do this today. It takes some scenario planners that can identify patterns that represent conditional action(s). While this is not easy, it has been done. It's the emerging patterns and their ability to affect the getting to the goals set by applications, processes and work forces.  Goals driven approaches with next best action features will lead to the kind of organizational learning necessary to make smart actions a reliable habit.

Net; Net: 

Dark events alone have significant impact on organizations. However,  when events are considered in concert with processes and data, the impact magnifies. The same can be said of patterns of events and the coordinated actions that can and should be made in a timely manner. This is where complexity science can contribute, but that is a topic for another day. Stay tuned for our last blog about dark events on the ROI of Illuminating Darkness.

Additional Reading on Dark Events:


Thursday, June 18, 2015

Art Projects for the Second Quarter 2015

It turns out that I gave my paint brushes a break this quarter after creating three paintings last quarter.
See last Quarter: http://jimsinur.blogspot.com/2015/03/art-projects-for-first-quarter-2015.html

I just puttered around with some digital art this quarter. Let me know what you think of the following pieces. Thanks to those who purchased pieces last quarter.

Apple of My Eye:

Infinite Love:

Wheel in the Sky:

Sleeping Dragon:

For more of my art see my website:
http://www.james-sinur.com/paintings/  for traditional paintings
http://www.james-sinur.com/digital-art/  for digital art

Wednesday, June 17, 2015

BPM at the Cross Roads: Digital Vs Rapid Development

There seems to be two major streams of activity in business process management today. One is aimed at delivering the new digital organization and the other is aimed at rapid development. It seems that the BPM vendors are aimed at one or the other these days. I had a CEO of a leading BPM vendor say that he was just aiming at super charging the developer and all his organization was oriented in that direction. I had another CEO say that he was aiming at transforming organizations to the new digital world while delivering incremental benefit. There are benefits of both approaches that I'd like to discuss here.

 Benefits Rapid Development BPM:

With the popularity of model driven approaches to creating processes and applications, BPM technologies have stepped forward to assist wagile approaches that are leveraging SCRUM cycles in a big way. Since most BPM technologies are built for change incremental they are idea for SCRUM cycles. Since the level of code in BPM platforms at a much higher level of coding metaphors, portions of processes and applications can be delivered for review cycles quickly. This allows both IT professionals and business professionals to work hand in hand in creating case management, processes snippets, code services, integration services, mobile APIs and cognitive services (COGs). This approach is usually operating in the cloud and development seats are rented at a low price point to get started without deep architectural study. This is a rich vein for BPM vendors to mine for revenue and there are a new batch of BPM players emerging with this approach.

See http://jimsinur.blogspot.com/2014/03/successful-digital-organizations-slay.html 

Benefits of Digital Transformation BPM:

Organizations are looking to transform to a digital organization. They are likely to start at the user experience and most likely a "mobile first" effort. Mobile, social and cloud is a baby step in this longer journey in getting to digital which may lead to changes in the overall business model over time. Studying the customer journey for competitive implementations is probably the first big step for for the digital journey. Then all constituents of your organizations end to end value and supply chains will need to benefit from process particularly shared processes and business policies. This is an evolving set of efforts that will require new and creative thinking considering likely opportunity and threat scenarios.

See http://jimsinur.blogspot.com/2015/06/digital-business-platform-what.html

Net; Net: 

Both of these approaches can justify and sustain a BPM effort, they sometimes end up looking like a "zero sum" decision. I would propose that they are literally better together and the most savvy organizations are seeing the opportunities to deliver speedy development and rich incremental benefits while driving to a digital organization. As the definition of digital will most likely evolve for most organizations, an agile BPM platform can be more than helpful. Maybe an essential?

Monday, June 15, 2015

The Trifecta of Digital Darkness

In recent posts Edward M.L. Peters has been illuminating one of the newer areas of unknowns in describing dark events and their potential impact, negative and positive, on organizations. This post we will concentrate on integrating the three major areas of darkness and their interactions. The three areas are dark processes, dark data and dark events.

Dark Processes:
A dark process is set of actions that are not visible to the owners or managers of a process for mass adoption, change, improvement or dissolution. Dark processes steal opportunities for more benefits. There are many reasons that processes are dark, but many are a result of just dealing with the emergence in business which will accelerate in the digital age. Almost all processes are subject to losing their effectiveness over time due to exceptions, dealing with customer differences and using processes in new contexts such as a piece of an new end to end process or a in a new legal framework. Process variation is the key driver for dark processes and variations will become the norm rather than the exception as we transform to digital.

Dark Data:
Gartner defines dark data as the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing).  Technopedia adds that dark data is a type of unstructured, untagged and untapped data that is found in data repositories and has not been analyzed or processed. It is similar to big data but differs in how it is mostly neglected by business and IT administrators in terms of its value.  They also add that dark data is also known as dusty data.  Whether structured or unstructured, analyzed or un-analyzed, a few thighs are true; dark data are often collected and are not seen as providing business value.  In terms of dark events, if the events are dark, then the data that describes their duration and existence is necessarily dark, whether it already exists in some form or is yet to be captured.  In other words, if there is a knowledge gap in what one knows about a process (e.g., dark event(s), it can be closed only by either providing a new capture method for the data that describes the event (e.g., new collector) or by analyzing the data to actually “discover” the events from the data themselves. This latter approach often involves analyzing log files and databases as the starting points for understanding how data are clustered around events and how events are similarly clustered into process activities or tasks.
Dark Events:
A dark event, as described previously, is an occurrence that happened inside or around a process that might make a business difference, if one knew about it happening. It could be an action outside a process, a trend, a shift, a change in measurement, a sensor trigger or a change in data. Knowing that something has occurred may or may not be of interest.  Knowing that an event has occurred can be something of note or just unnecessary noise. Establishing rules for expected events helps, but mining for the unknowns is usually the cure for dark events. Once known, then decisions or actions can be made and rules established / updated.

Net; Net: 

Dark Data are encapsulated within Dark Events that may in turn be encapsulated within a Dark Process.  While Dark Data are necessarily part of a Dark Event, the Process may or may not be “Dark”; it could simply be a case of determining missing information about a know process.  However, it would be a know process with unknown events and data that when illuminated add to its understanding. 

Additional Reading on Darkness:

Thursday, June 11, 2015

Digital Business Platform: What Technology Suite Works Best?

A goodly number of organizations are on a path to incremental transformations of their business to digital. There seems to be a consensus that the user experience is a great place to start. Some organizations are starting with a "Mobile First" approach while others are looking at customer journeys. See http://jimsinur.blogspot.com/2015/05/dangers-in-first-steps-of-digital.html

Still others are looking at end to end processes that usually include value or supply chains. Since there is pressure to get going, decisions are being made on technology bases that might create regret later. While technology should not be the driver, but looking at the opportunities and constraints that each type of platform brings should be an exercise that organizations should be doing. Also organizations will have to pick an approach for integrated versus hybrid. Most likely all platforms will support social, mobile and cloud approaches. If they don't, consider this a warning sign.

Application Platforms:

Business loves to have a complete application, out of the box to support their business. While this is the dream and might be a good place to start, watch out for picking applications with rigid platforms that are hard to upgrade and difficult to leverage business differentiation. Quite often you are locked into a business model, so these are great for standard business models and functions. Beware that these approaches solidify the silo approaches as well. Look for a liquid infrastructure and easy modification capabilities.

Process Platforms:

Process platforms are quite often quite capable of supporting multiple styles of processes (emerging or static) and have enough pattern recognition and analytic capabilities to be helpful. These are what I call the "Swiss Army Knife" approach. If they have smart process applications that come with the platform, they are more valuable. If they come with rich analytical capabilities like next best action and cognitive computing features, they can be even more valuable. Make sure there is enough additive business capability to make them desirable.

Analytic Platforms:

Analytic platforms come with a richness of pattern recognition and decision support and work with large amounts of "in memory big data" to help guide business decisions. The problem with this approach is that they are not linked to actions and rarely work at the faster speeds required by the "IoT" This is what I call "the brain on a stick approach"

Content Platforms:

Some platforms are great at the "first mile" and support smart capture and even support multiple content types including "micro content". Make sure there is an integrated analytic and process platform. It is a bonus if they come with smart process applications on top of the content.

Integration Platforms:

Some platforms are great as a data and complex event integration hub and can bring together multiple legacy applications and data streams. This is essential infrastructure for the digital world, but make sure they are more than a "bus stop" and can support being a digital nervous system that can support process and analytic capabilities as well.

Hybrid vs Integrated:

Organizations have to decide if they want individual platform stacks (like many individual oil rigs) that they transport between or a planned integrated approach that easily works at convenient speed and low transportation costs. Some integrated vendors are warning about "Franken Stacks" and others are warning about "Integrated Average". Either way, watch out for "lock in" and inability to play well with others. This is true when you pick a core that might need future augmentation. Some vendors can support multiple stack approaches like an agile process platform combined with applications and just enough analytic capabilities.

Net; Net: 

Organizations have to start somewhere and the decision on platforms will likely be quick. Organizations will have to decide is they want best of brand which means you have faith in particular vendor stack / platform or a best of breed approach that combines the best of optimized capabilities. Big decisions are looming right now that could create downstream opportunities or problems.

Monday, June 8, 2015

Pegaworld 2015: Deliver the Digital Enabled Customer Experience

Again Pegaworld has surpassed it's high water attendance mark this year with about 3300 attendees. The theme was about going digital and how the enabled customer experience was the tip of the digital sword and Pega has the "end to end engine" to drive the digital transformation for organizations. Alan Trefler started  the conference with his view of CRM evolved and how Pega leverage of "next best decision & action" powers the new customer cycle picture below:

Going digital is essential, so it is a mandate for all organizations. See http://jimsinur.blogspot.com/2015/03/do-digital-or-die-21st-century-mandate.html

Alan then proceeded to selectively put clients on the stage that were on their way to digital; most starting with the customer experience. The prime example was Cigna who was pursuing a digital vision that means redefining what a Healthcare organization should be in the new digital age. Dr Mark Boxer gave us all a view of their vision that they plan to roll out on top of Pega. The basic idea was to leverage evidence based medicine personalized for each patient leveraging real time measures, events and patterns to drive treatment and proactive behaviors tied to premium incentives.

Don Shuerman & Kerim Akongul demonstrated the power of Pega7 Express, a lite gateway version of the Pega Digital Platform applied to a delightful & amusing application of a proactive health monitoring of Kerim's physical condition and training program.

There were also some client stories that focused on ROI and Speed that were highlighted for those not quite ready for a digital transformation yet. This proceeded breakout sessions.

IsBank showed great numbers

The State of Maine delivered an integration of silos through their "One Maine" efforts


United Healthcare showed great results over the years while growing to 200,000 users. Now that's hyper volume. Name a model driven approach with that kind of volume !!!!!!!

General Motors showed the power of emerging telematics. Updates to my car to optimize it's performance and my experience. Nice !!!!

I can't wait until tomorrow as the first day blew me away. I also can't wait until Pega 7 Express adds more feature over time, but it is poised now to help new users and light weight processes today though the release date is in the future. Low end BPM vendors watch out !!!

Thursday, June 4, 2015

Supporting Hyper-Speed in the New Digital World

In the new digital world. processes / applications will be combining the physical world through leveraging the IoT with the logical world in a process. This means that speeds will have to ramp up to nano-seconds for the physical world combined with long shelf times for adaptable case situations. Combine cognitive computing assists to speed up knowledge based tasks and the overall speed is increasing and the variety of speeds managed in a process / case will become more varied. 

Nano-Seconds Count:

Processes / applications never really had to deal with nano second responses, except for dedicated manufacturing and distribution domains. Now all processes and applications will be asked to respond with actions in a nano-second fashion. Responding to events and patterns from the IoT, complex event processors and social feeds will require more than a subtle increase in acceleration. This will require more intelligence than the simple rule based approach as used in straight through processing and exception management. It will require cognitive responses that may include self driving processes that leverage swarming behavior.

See http://jimsinur.blogspot.com/2015/05/self-driving-processes-are-built-to.html

Faster Decisions: 

In the past, because of performance issues, decisions were determined in a separate environment and in a slow analytical way. Today with big data in memory and in context, there is an opportunity to make faster and better decisions. Now decisions can be made in real time and in-line with emerging processes. Also the variety of data types is expanding to audio and video thus kicking off semiotic(shapes) and in context meaning. 

See http://jimsinur.blogspot.com/2014/02/business-survival-skill-ii-making.html

Faster Case / Process Instance Closure:

Today processes can only go as fast as the slowest resource involved with processes or case actions. We sped up processes in the past by replacing straight forward human judgement with explicit business rules and I expect this to continue. Now with cognitive computing we can replace wide search and parts of human judgement and replace humans for more investigative and complex judgement tasks. As we add more COGs to processes / applications, speed will increase.

Net; Net: 

Speed is increasing all around us, but the leap to the digital world really puts the "pedal to the metal" Digital is going to have as big of an effect on speed as well as adaptability and change frequency. As the digital age emerges we will fully understand the real impact, but I'm betting that it will be bigger than we all think collectively.

Additional Reading on Digital:


Tuesday, June 2, 2015

Dark Event Discovery and Analysis- Illuminating the Unknown(s)

Sometimes organizations think that they can plan the typical business and process scenarios and even have planned responses, but sometimes the "The Black Swan" shows up. No organization is omniscient, even if they have some of the best minds and experience. Dark events are bound to happen even though these dark events are more likely than black swans. In the last post, I briefly discussed the concept of Dark Events as being something that belonged to the area of the unknown.  

“There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know.”

As described in the quote above from former US Secretary of Defense, Donald Rumsfeld, this can be either a known unknown, like a dark spot where you know something is going on and are aware you don’t have the data or something that is totally unknown.  In the latter case, it is an unknown, unknown or something that is totally surprising.  In both cases, we need some way provide transparency into an area where the activities are not readily observable. 

For our purposes here, let’s focus on business events.  Most business events are part of business processes and the information is captured through business IT systems.  Normally, business executives rely on business intelligence (BI) solutions that focus on schema-driven reporting on transactions and documents or from business activity monitoring (BAM) solutions that provide pre-defined dashboards and metrics. Unfortunately, these and other silo-centric analysis type solutions provide only a partial view of what is really occurring.  What we are interested in exploring are the spaces between the actions we currently “know” exist, the “known knowns’” and determine if there are any events that are occurring but are not being observed
In the figure above, we can see the known steps (shaded in light blue) in a typical transaction system.  However, between steps 4 and 5 (Transaction Paid) are a number of other steps that are happening (shaded in gray) that detail a customer service intervention process that corrects the data on the customer transaction so it can process to conclusion.  These steps are part of a “Dark Process” that encapsulates the “Dark Events”. 

Dark Events are Temporal

Most organizations have events logs where they track and store data such as a record that a customer was on the self-service portal, an order placement or a call record. However, as in the example above, these logs are usually information poor. Dark Events, by their very nature, are time-based and found in the spaces between “known events”.   Each event is an instance of task execution, comprises a definitive unit of work and has a time-based start and stop tag associated with its existence.  While they are additive to “known events” they are do increase their overall time but rather provide descriptive detail about how the duration of time is being used.  In this way, understanding the number and duration of Dark Events provide the intelligence required for making decisions about their improvement.

Dark Event Analysis (DEA)

 Dark Event Analysis focuses on the time-based reporting of process execution information, differentiating it from BI and BAM.  DEA provides this insight because it analyses are at the ‘event’ level. Events are a new source of data for most organizations. DEA provides analytics and drill down capabilities to the details of the process execution in order to understand both time gaps that identify Dark Events as well as their effect on the root cause of process variations. Dark Event Analysis focuses at the level of the work steps as they occur, no matter where they occur.

Dark Event Analysis Techniques- Mining vs. Discovery

There are a number of ways to approach both the discovery and analysis of Dark Events.  The two most commonly used techniques are Automated Business Process Discovery and Process Mining.

The concept of Automated Process Discovery was first discussed in 1996 by Cook and Wolf in an attempt to understand and improve the software development process through the use of algorithmic techniques (e.g., neural networks, Markov models, etc.).  This differs from conventional process analysis techniques (such as Six-sigma and Lean Management) that are based on a-priori biased models. While these representations are good for detecting high level and readily observable issues, the data they capture is insufficient to understand the essential time spent between observable events as well as the nature of variations, exceptions and deficiencies that are being promulgated during event, task and process execution.  Automated Process Discovery often involves the use of platform specific data collectors that are designed to capture the complete picture of what occurred during a specific time period.  These data are then stored, structured and analyzed to reveal the entirety of the event record as the basis for improvement.

Process Mining is a statistical approach to understanding the "general behavior” in processes, in the way they are observed in the working environment. Process Mining looks to interrogate business system event logs as the basis for understanding the behavior of business systems.  The statistical techniques in Process Mining often concentrate on finding the “main” or “average” process that is being executed.  This information is used to build a model of the way the system is actually operating as a basis for comparison against expectations and a starting point for discussions about improvement.  Further, the models produced through Process Mining can be used for simulation and other predictive modeling techniques providing the basis for an advanced analytical capability.

Both of the approaches described above can be quite useful in both understanding business events and analyzing how to improve them.  When it comes to Dark Events, Automated Process Discovery tries to go further than Process Mining by capturing more detailed data than usually exists in business system event logs.  It is by capturing and analyzing this data that we illuminate the “un-knowns” and increase our ability to make meaningful improvements.

Dark Events; next topic: 

In the next post, Jim Sinur and I will discuss Dark Events and their relationship to Dark Processes and Dark Data.