DevOps transformations can often become a checklist of tools and processes we need to assemble, like so much Ikea furniture. But DevOps is not a kit of things you can apply to your organisation. It’s a mindset. And you can’t force an organisation of people to think the same way.
I recently listened to an episode of the Idealcast with Jez Humble and Gene Kim. They talk about 15 years of DevOps and look back on five years since the DevOps handbook was published.
I was lucky enough to see Jez give my company a talk back in 2011 perhaps when I was working for a Dutch bank. He told us that regulatory frameworks shouldn’t be a blocker to automation of processes, and we nodded our heads and then we went back to work.
But now, still more often than not, dev, test and release are still gated processes. While there has been some progress, the waterfall (DTAP) software release model is still largely prevalent.
How come?
Is it because managers are often too invested in the details to trust engineers to make the right decisions? Is it because existing command and control structures are too pervasive to enable DevOps nirvana? But does DevOps nirvana exist, and do most of us even need it?
I’ve worked for dozens of global companies over the last 20+ years, and none of the teams I’ve worked with could release to (customer-facing) production on demand without sign-off from a CAB or a group of managers. That doesn’t mean to say that none of the organisations I’ve worked in have released to production daily – it’s just that for most teams, production is usually still a very long way away from development.
So, where are the advantages to DevOps that we’ve been sold for the last 15 years? Could we do away with all the CI pipelines and tests and have the same efficiency levels as before?
Does DevOps still matter? And what does it mean these days?
No thanks, We’re good
Often the existing organizational structures and culture don’t allow change. Or if they do, then the resistance returns it to equilibrium. It will be a brave, perhaps foolhardy manager to go against the established routines. In reality, it will be a hard path to argue for this, and it’s fraught with personal danger. You could lose your job or, perhaps more realistically, as a manager, paint yourself into a corner if things go wrong.
If, for example, a touchless release procedure gets to production, championed by a single or small group of managers, it needs to provide value and be risk-free. This is not just a new release system; it’s a cultural change which should be backed by the whole organisation. Usually though, at the first sign of trouble the organisation reverts to old behaviours and existing processes.
Or worse, it implements new ‘touchless’ systems but they are in fact gated systems.
Why? Because organisational structures and managers are traditionally risk-averse. They don’t want to change. It might be simple enough for those of us who have read the books to say, “we acknowledge Conway’s Law and know that we want to engineer our organisation to reflect the systems that we will build”, but the reality of this means tough decisions and even tougher commitment to follow through on those decisions when things get difficult.
Running to Safety
And why should teams (i.e. the people who keep the lights on) even help with this change? What’s in it for them? Why should they worry about automation, different release patterns, and improving observability to prove success? Existing teams (and certainly ops teams) whole reason for existence is to protect the status quo fiercely and this goes far deeper than just commitment to the company. These are almost personal values.
Therefore, often, the blockers to DevOps adoption come down to simple basic human needs:
- Being too Busy to do Things Differently
- Fear of losing your job.
- Pride.
- Attitude to learning.
- Lack of clarity from decision-makers.
- Dubious benefits.
Being too Busy to do Things Differently
“We are too busy to change because we’re keeping the lights on.” This is the perfect comeback from a team which is running scared of change. Admitting you’re always too busy means that, by definition, you have no time to improve anything. No time to reflect, no time to try something different. But IT systems are always changing. From new technologies to new ways of working, how can we be too busy? Being too busy to improve also means that, by definition, you are falling behind.
Losing Your Job
Just like the Luddites, people fear that a wave of automation tooling will rob them of their livelihoods. This fear is justified, especially in an environment of ongoing tech layoffs and a landscape in which “the future is not evenly distributed”. But it’s not just as simple as saying, “Automation will have my job”, because anyone who has worked in automation or even with the current levels of AI programming assistant knows that it’s not going to take your job. It’s an assistant. It needs guidance from experts. And writing a script to automate a task is something which usually provides benefit for all right?
So, while automation in itself might seem scary, all automation needs guidance and expert engineers to assist it, and, in the case of scripting and pipelines, it needs creating and maintaining.
Pride
The argument that ‘we’ve always done things this way’ or that ‘we take pride as a team in doing things this way’ is a trap for those who don’t want to admit that IT is constantly changing. This argument is not only a logical fallacy, as the proverb has it:
“Pride comes before a fall”.
Taking pride even momentarily can make us take our eye off the ball. Therefore, don’t fixate on a technology, a process, or a single way of doing things – stay open to new ideas and know that things will always change. Pride is in fact perhaps the biggest enemy of operational stability.
Attitude to Learning
Similarly, we allow ourselves to continue learning by keeping an open mind. We become entrenched by not understanding other perspectives or new ways of thinking. Old ways of thinking create pride in existing thinking structures with predictable consequences.
Confused Leadership
Sometimes, it’s the management that can’t make up its mind. Sometimes, the management sends mixed messages because, inevitably, there will be different opinions. Some managers will champion change and want to see progress. Some will want to keep things the way they are. This doesn’t mean that we should abandon our initiatives and revert to doing what we’ve always done. In a confused leadership situation, picking the side that most aligns with our wishes is just easier. Whenever leadership is unclear, there are gaps for those who do the work to justify their ways of working.
Dubious Benefits
So, let’s say, you have all the DevOps tools and processes. You have GitHub and Artifactory and you have SonarQube, you have a mono-repo (or not) and you have test automation. Where does it get you? Can you release daily? Do you have an architecture that lends itself to releasing daily? Do you have confidence that you can do it without breaking production? Can you measure your changes when you make deployments? Can you back them out easily in case of problems? Do you have infrastructure as code and does that make your IT cheaper and more reliable?
Different Strokes for Different Folks
We inhabit a complex landscape and work with humans.
Working in a team on hard, abstract IT and software problems can be fun and challenging and it can be difficult and lonely. People change, teams change, priorities change.
How do you deal with the loose cannon in your team? Do you work with them or wait for them to move on? Do you box them in or put them in charge?
How can you improve your software delivery incrementally without bending or breaking how things work too much?
How can you maintain control and standards while also improving?
Is it enough to have a happy team rather than aim for a perfect team?
No Nirvana
One of my favourite podcasts at the moment is about trail running. It’s called Tea and Trails, and high-performing (usually amateur) trail runners discuss their lives, their training and stories from their events. Their trademark sign-off is to wish you luck and to aim for “progress and not perfection”.
And while we want to attain “DevOps nirvana” we must admit to ourselves that in reality it doesn’t exist. Instead, we should aim to acknowledge our strengths and weaknesses as individuals, managers, teams and as an organisation. And we should constantly aim for progress, not perfection.