Why Is Organisational Transformation so Difficult?
Why?
I was chatting with a senior leader from large enterprise about their frustrations around leading organisational change. Their core question was: No matter how much conversation took place and no matter how many people outwardly agreed the change was required, why was it still so difficult to execute? It’s an interesting question and the answer will benefit any Individual, Team or Organisation seeking to launch a transformation. I’ve written several case studies on why transformations fail by deconstructing specific examples, all from skills & capability perspective: leadership, strategic thinking and influencing, etc. This time we will investigate organisational change in terms of individual and organisational psychology, adding further insight as to why 70% of Digital Transformations fail and 50-90% of strategic initiatives also fail.
What?
An organisational transformation is little more than some form of technological adoption, combined with operating model changes and refinements to ways of working, that enable those in the organisation to make better decisions, produce superior results while becoming more efficient at doing so. However, this is rarely what happens. To better understand why leading organisational transformation is so difficult we will cover how individuals and collectives typically respond to change in the moment and over time.
How?
We will map out the seemingly invisible psychology of an organisational transformation with the help of academic behavioural models to answer 4 key questions:
What is the balance of certainty vs uncertainty during change? Revealing the glass-ceiling individual and teams hit.
What is the rate of transformation acceptance in an organisation? Revealing the true scale of those for and against.
What do individuals experience before and during a transformation? Revealing the reason those against outweigh those for.
What is the resulting organisational psychological state? Revealing the scale of actual human fallout and reluctance.
Certainty vs Uncertainty
The model below is based on the work of Dr James Prochaska, Professor of Psychology at the University of Rhode Island and Dr Carlo Di Clementi, Professor of Psychology at the University of Maryland, who developed the Transtheoretical Model of Behavioural Change (TTM). They identified 5 distinct stages of change on a timeline that demonstrates the process of attempting to make a change that specifically involves giving something up without replacing it with something else, i.e. moving from a state of certainty to uncertainty. In the context of this case study we will use it to highlight the effects of organisational transformation driven uncertainty; Digital, HR, Finance, M&A, etc.
Source: Transtheoretical Model of Behavioural Change, Prochaska & Di Clementi 1983
Each stage of change correlates with specific behaviour driven by specific conscious or unconscious beliefs:
Pre-Contemplation: The awareness of the need for change does not exist.
Contemplation: Considering the need to make a change in the near future.
Preparation: Seeking the knowledge to achieve required change.
Action: Implement new-found knowledge required to achieve change.
Maintenance: Sustained successful measurable change.
The transformation evangelist is, at least, in the Contemplation phase, most likely in the preparation phase, possibly the action phase. The audience hearing the idea for the first time are most likely in the pre-contemplation phase, at best the Contemplation phase. This demonstrates that right out the gate a potential psychological gap of anything up to 12 months between the evangelist and various members of the audience in transformation conception, i.e. Building a mental map of the organisational end state, it's pros vs cons and how to get there as it would apply to each member of the audience, a process which only those open to the idea are likely to initiate.
Something I find particularly interesting about this scenario is the scale and impact of an invisible and unspoken request that's taking place without anyone's knowledge. Everyone round the table to is being asked to relinquish the certainty of how things work now and the sense of control and familiarity it provides, and replace it with the uncertainty of how things might work, a potential loss of control and the unfamiliar. Put another way the audience are being asked to sever ties with the positive anchors they rely to live their lives as they know it.
A simple example of an anchor is when a smell, image or sound that triggers a response mentally transporting you to another point in space and time to relive a specific event. PTSD is an extreme negative example of this. We each use anchors all day every day to generate an incalculable number of unconscious rituals that provide comfort and familiarity for every environment we operate in; home, work & play. These form a plethora of invisible handrails that trigger us to run programs that form our daily routines. When any one of these invisible handrails is threatened with removal, especially without consent, we experience involuntary discomfort, the severity of which correlates with the level of reliance we have on it. Let me give you an example:
During a large enterprise organisational transformation I supported, one of the senior leaders was being challenged to eliminate a process from their area of responsibility as it was shown to be of no use in new world. The numbers it produced were neither accurate or relevant, yet this senior leader fought tooth and nail to retain this process. After some discussion I managed to get to the root cause of their behaviour.
The senior leader, who was pro-transformation prior to execution, was overwhelmed during execution with the level of change taking place, which included difficult conversations with direct reports around restructuring, interviewing people they'd worked with for years for roles they would never secure and eventually make them redundant, replacement of obsolete technology and systems with new and the adoption of skills required to operate them, all happening concurrently. Ultimately our senior leader had come face to face with the reality of what it is to be in a leadership role and discovered it was something they were unequipped for.
As a result of our senior leaders escalating anxiety they exhibited a need to cling to a sense of certainty, safety and stability. This manifested as retaining a familiar and ineffective process rather than deal with the discomfort and fear of failure associated with adopting a new process that demonstrably worked. What made this particular instance memorable was the senior leader openly admitting they knew their course of action was futile, but their need to find certainty n any form was so great they became compelled dig their heels in. With this understanding I was able to give them the tools to move forward.
Given instances similar to this are inevitable, how do we determine how many of the population affected will end up in each category?
Acceptance of Transformation
The model below is a bell curve created by Sociologist Everett Rogers, Distinguished Professor of Communication at the University of New Mexico, who discovered the rate of adoption of new technologies in any given population, which he called the Law of Diffusion of Innovation. In the context of this case study it predicts what you will witness in terms of how a transformation will be received and executed by your organisation:
Source: Law of Diffusion of Innovation, Everett Rogers 1962
We can now correlate what we’ve learned about certainty vs uncertainty within any given population, which can also be broken down in to 5 key stages to predict the rate of adoption.
Innovators: Come up with the idea and present a vision based on future possibilities.
Early Adopters: Quick to identify and seize opportunities based on future possibilities.
Early Majority: Get onboard once the benefits have been reliably tested and proven.
Late Majority: Not really interested, get onboard to avoid being left behind.
Laggards: Firmly opposed to change, likely resistant, possible saboteurs.
This model gives us a snapshot to reinforce just how change averse any given population really is. Reality check: Almost everyone is not ok with, or prepared for, the change. In fact, only 16% of your population will have, at best, a conceptual understanding of new world, never mind be prepared for it in practical terms.
In our example an entire £2Billion Large Enterprise was being mapped out for transformation. However, the board had agreed they would take a linear approach by transforming one department at a time. The idea being they could learn from mistakes and refine as they moved from one department to the next. I was supporting the board member who was at the transformation spearhead, the first to act, which directly affected 270 people. With a known headcount we can predict how many people populate each category:
Innovators: 7 were transformation evangelists and encouraged others to get involved.
Early Adopters: 36 agreed with the transformation and actively supported it.
Early Majority: 92 sat on the fence until the idea was proven to be of benefit to them.
Late Majority: 92 eventually jumped on board motivated by fear of missing out.
Laggards: 43 heavily rooted to the present and attempted to undermine change.
227 of our 270 operated on a spectrum from mild resistance to active sabotage as they expect to lose more than they will gain. Remember, from their perspective they are being asked to give up the tangible and reliable for the intangible and unknowable. This explains why so many people queue up to tell you why an idea won’t work and so few will tell you why it will work, hence progress typically seeks forgiveness rather than permission. Otherwise, nothing new would ever occur.
In our example the majority of middle managers and team leaders, all arguably far less experienced than our senior leader, had exposure to similar thoughts and issues, with the added responsibility of keep the department operating during transformation. Many of the team leaders were in their late 20’s or early 30’s with team members considerably older than themselves in their charge.
Some the older team members were openly verbally hostile and aggressive on a daily basis as their younger untrained team leaders watched on unable to intervene. I was able to address this with by rapid deploying a crash-course in deriving underlying incentive from exhibited behaviour, and navigating difficult conversations to resolve the core issues of disruptive individuals.
These team leaders were good empathetic people genuinely doing their best to do the right thing, however, they needed a nudge in the right direction with some rudimentary psychological insight. Thats said, this situation does raise an important point: It’s easy to be professional when everything is running smoothly, the litmus test is can you maintain your professionalism under duress? If you don’t train for worst case scenario in the context of your role, you will be of no use when the moment comes. Ironically, this is what your salary as a leader is actually paying your for. So, what is actually happening inside the untrained mind of someone having this experience?
Fight, Flight or Freeze
Below is my version of a learning curve demonstrated by merging the Dunning Kruger Effect with Noel Burch’s Four Stages of Competence. Professors of Psychology David Dunning & Justin Kruger discovered that people with limited competence tend to over-estimate their abilities due to lack of self-awareness, and Noel Burch’s Competence Model appears to be a simplified and practical adaptation of Flavell’s Metacognition from 1976, that specifically charts learning and competence progression. By combining these two models we can map out the relationship between Confidence, defined as: The belief in one’s abilities, qualities and judgement, and Knowledge, defined as: The application of theory, refined through purposeful practice over time. This creates a learning curve that tracks Competence, defined as: The ability to achieve a desired outcome to an agreed standard in a specific context. In any given moment each of us is somewhere on this curve, and it changes from one moment to the next in any given endeavour.
Source: Dunning Kruger 1999, Noel Burch 1970’s & Kenny Wallace 2019
This model demonstrates Knowledge correlates with competence, and competence comes in four stages:
Unconscious Incompetence: Unaware you lack a required skill, utterly clueless.
Conscious Incompetence: Aware you lack a required skill, in your face reality check.
Conscious Competence: Learning a required skill, having a go with training wheels on
Unconscious Competence: Unaware of a highly developed skill, in the zone.
We can see anxiety exists in the middle two. The closer you are to the bottom of Conscious Incompetence the greater the challenge the greater the anxiety, more commonly known as panic. And the closer you are to the top of Conscious Competence the milder the challenge, the milder the anxiety, more commonly known as excitement.
To continue with our example: The mind of our senior leader was in the process of being hijacked by the Amygdala in their Limbic Brain during their reality check. The Amygdala has no capacity for language and can only induce an emotional response. Our senior leader rapidly tipped from unconscious incompetence to conscious incompetence, manifesting as fear. This was an unconscious response to the sheer number of absent skills required to navigate themselves and those in their charge from old to new world.
This is where, I think, things get very interesting. During high-stress situations, high-stress being unique to each individual and contextually specific, every single person will run an automated program, regardless of whether it’s effective or not, e.g. I’ve witnessed a thirty-something person fall to the floor crying and proceed to throw a tantrum like a two year old during another large organisational transformation announcement, indicating this behaviour had been rewarded throughout their life when faced with what they consider genuine adversity. This isn’t demonstrative of an absent coping strategy, this is their coping strategy. Absence of a coping strategy manifests as fear induced paralysis. A training void. There is no program to run.
As for our senior leader, they had reached their glass-ceiling with the level of adversity they could contend with in this instance, therefore training to cling to certainty and familiarity kicked-in as a way of achieving some form of stability as they had no other program to run.
Learning to become competent in one new thing requires consistent conscious effort over time, so the expectation of being immediately competent in almost every aspect of leading an organisational transformation without any training is absurd. Textbook unconscious incompetence as senior leaders plunge their organisations in to transformations regularly.
At the extreme opposite end of the spectrum are first responders and military personnel, who train with near-relentless repetition to embed automatic responses in preparation for the extreme adversity their profession inevitably brings. They can be deployed into the most hostile conditions imaginable, move toward and overcome the challenge. They remain calm in the chaos.
There is a superb quote that sums up this situation, which I first heard from a Navy Seal: “Under pressure you don’t rise to the occasion, you sink to the level of your training. That’s why we train so hard.” It’s an adaptation of the original quote by Archilochus, a Greek Philosopher and Poet from around 650 BC: “We don't rise to the level of our expectations, we fall to the level of our training.” This is unequivocally the case.
If we combine this model with the previous one we now know with certainty 84% of your organisation will rapidly tip from unconscious incompetence to conscious incompetence. They are being involuntarily tipped in to a situation they have not trained for. So, what happens to an organisation when this phenomenon presents organisation-wide?
Organisational Anxiety
The model below is based on a matrix created by Professor of Psychology, Mihaly Csikszentmihalyi who was first to formally investigate the traits associated with high performance, which he collectively called Flow. I have modified it for a business context to correlate organisational states with culture by looking at the relationship between Challenge, defined as: Degree of difficulty to be overcome, and Support, defined as: Available resources. To determine the culture you predominantly operate in you will witness one or more of the following:
Source: Based on Massimini, Csíkszentmihályi & Carli 1987 and Kenny Wallace 2019
These four cultures correlate with specific behaviours that can be measured in any individual, team or organisation:
Anxiety: Missed deadlines, avoidance of interaction, likely negatively impacting employee sickness absence.
Apathy: Absence of enthusiasm or presence of indifference, likely negatively impacting employee retention.
Comfort: Lack of activity and accountability, likely negatively impacting employee productivity.
Flow: Pro-active, self-directed, emphasis on personal responsibility, likely positively impacting all of the above.
We have already learned from the previous models that anxiety directly correlates with a lack of knowledge and skill, i.e demonstrable incompetence. The example of our senior leader gives us insight on how that might manifest in an individual and we now also know that with mathematical certainty 84% of your organisation will spend the majority of the transformation in the Anxiety quadrant, causing inevitable costly delays, stall your project or cause it to fail all together.
Conclusion
So what?
In terms of openness to change, every participant, will land in one of 5 categories prior to, and during any organisational transformation: Pre-Contemplation, Contemplation, Preparation, Action and Maintenance. These 5 stages provide a timeline demonstrating that human adoption of change will lag considerably behind structural change. Each participant will progress at different rates as they contend with the short and long term professional and personal impact, such as the severance of positive anchors, which we all use daily as hand-rails that enable us to manage a vast array of complex tasks.
In terms of change adoption 84% will not be ok with, or prepared for, the change. Only 16% of your population will have, at best, a conceptual understanding of new world, never mind be prepared for it in practical terms. Those incentivised by the carrot will go on the journey no matter what. Those incentivised by the stick will express reluctance. This is mainly due to the invisible incentive structure humans use to navigate their daily lives. When the incentives offered by the transformation align with personal incentives you will witness acts of assistance, when misaligned you will witness acts of resistance, possibly sabotage.
In terms of competence the vast majority will encounter a glass-ceiling after being involuntarily tipped in to Conscious Incompetence, causing an anxiety spike that lands on a spectrum from fear to excitement. This spectrum has a causal relationship with their capability & skill to navigate and adapt to change and adversity in this context. This social phenomenon will occur at all levels in an organisation. The consequences of this are of greater significance when it happens in those leading the charge as they not only have to manage themselves, but those in their charge, which they are most likely unequipped to do.
In terms of overall organisational state, roughly 16% of your organisation will experience excitement due to their perception of what’s to be gained, the other 84% will sit on a spectrum of anxiety due their perception of what will be lost. The example of our senior reinforces it can negatively affect anyone involved, especially given our senior leader was a transformation proponent. I’m reminded again of one of my favourite Albert Einstein quotes: The world we created is a product of our thinking. It cannot be changed without changing our thinking. No problem can be solved with the same level of consciousness that created it.
Now what?
Given the measurable difference in people’s openness to change, even if they are overtly supportive of the change they still require significant time to process it’s implications, which can be nudged along a timeline with meaningful insight and realistic deadlines. As with managing any stakeholder, they will need to be drip fed the right information at the right time unique to their specific context to generate buy-in and ongoing pro-active support for the transformation. This where patience is a virtue. The mathematical certainty of how the population of your organisation will respond individually and collectively to a transformation can be leveraged as an asset rather than considered a liability.
Identifying your 16% is mission critical as this is the group of people who will carry your transformation to completion, and they won’t all be who you think, coming from the breadth and depth of the organisation. Do not assume all those in leadership roles will be onboard, especially those at senior level. They all have their own agendas and a transformation can easily be considered a hindrance to their own objectives, especially in large enterprise. I’ve witnessed a senior exec/transformation lead be forced to resign after exposing significant financial misconduct at board level due to improved data analysis. I’ve also witnessed a senior exec covertly refuse to cascade transformation information to their department keeping them all in the dark. The parallels with Gordon Ramsay’s Kitchen Nightmares are uncanny, but with hundreds, if not thousands of people involved.
Coming back to competence, the irony is that most who hold leadership appointments are actually inept leaders, something easily demonstrated mathematically using Prices Law, but that’s a topic for another day. Your role as a leader during transformation is not running about about making sure everyone is happy, as I’ve seen many do in attempt to assuage their guilt for imposing potentially negative consequences on so many in their charge. Setting aside organisational structure and technology implementation for a moment, your role is to facilitate those in your charge and prepare them psychologically for the change by providing the tools and information that enables them to make informed decisions about their future, regardless of whether they are happy or not. Remember, Anxiety causally relates with lack of knowledge.
The reasons why organisational transformations have such a high failure rate can be summarised as: Leaders fail to assess Organisational Readiness prior to transformation launch. Typically leaders at the top of organisations are possibility and change orientated, and almost everyone else isn’t, hence those against always outweighs those for. Plenty of psychometric data on this. As a result senior leaders typically fail to balance what’s possible with what’s probable due to a fundamental lack of understanding of how humans actually respond to change, especially at scale. As an ex-Paratrooper I find it continually fascinating that leaders in the business world are wholly unaware of the need to train for worst case scenario to achieve organisational readiness, especially as an organisational transformation is most likely perceived as worst case scenario for 84% of the people in their charge. Organisational readiness is something we accept as the norm, for which we have measurable standards.
What next?
Now that you have a better understanding of why organisational transformations are so difficult fail in terms of leadership blind-spots, population buy-in and organisational readiness:
Are you, your team, department or overall organisation about to embarked on, or already embarked on a transformation project? If so, from a leadership perspective, what considerations are you now aware of that you hadn't previously thought of?
Given that Organisational Transformations are an inevitable part of remaining competitive, and the demonstrable need for assessing Organisational Readiness; What steps will you, your team, middle and senior leaders take to establish this as fact?
With the mathematical certainty your Transformation will have on your people, organisation wide, and it's likelihood of failure; What will you do to mitigate the risks associated with those you are responsible for leading through the transformation?
Take your learning one step further and complete my Case Study Review. Capture your learning from this case study and commit to changes you deem relevant for your situation. A copy of your completed review will be emailed to you instantly.
If you are in the process of preparing to execute an Organisational Transformation and have concerns then do consider working with me to either assess your Organisational Readiness or address issues that have come to your attention as result of this case study.
