Does Continuous Delivery Have an Image Problem?

Webcam Recording

===

[00:00:00] Richard Bown: Before we get into this episode, a quick heads up that the sound is a bit catchy. So apologies in advance. I hope you can stick with it. Hopefully next time. It won't be so bad.

Welcome to the software delivery club. My name is Richard Bown. This is episode 20. Yeah. And this is a ranty one. This is called Does Continuous Delivery, Deliver or Does Continuous Delivery Have an Image Problem or What's So Good About CI/CD? So this is quite a meaty one. So, as you might know, every two weeks or so, I explore different aspects of the business of delivering better software.

From leadership to strategy technology teams, product systems, delivery, customer. I talk with myself mainly and sometimes with industry experts about important subjects about building and delivering amazing software.

The episodes are inspired by some stuff I've been reading or talking about or talking with people about over the last couple of weeks. At the moment, these podcasts are mainly [00:01:00] me talking into a microphone. Because I'm slow to form ideas. However, I'm starting to see some patterns forming.

So I had an online discussion with the CD guru, Bryan Finster. On Twitter. He said It's 2022. And some people think CD is edgy. It's not it's table stakes for competent. If you cannot make and deploy change in the same work day, you're a security risk.

And I read that and thought.

Do people think CD is edgy. I'm not sure. Anyway, I came up with a smart comment And I said, I think CD is edge case rather than edgy. Too many orgs just don't see the point and they probably have a point.

And. To me. That is a little bit smart, a little bit, maybe innocuous, but I didn't know, Brian at this point. And it got him interested. And he wrote a kind of whole rebuttal blog post, which in some ways. I kind of missed the point of what I was trying to say, but maybe it wasn't [00:02:00] clear. What I was trying to say.

What I was trying to say, was that companies are trying to C D and. Then kind of not getting there and wondering if it's still relevant or important for them. So I'm starting to think that it's got an image problem.

So, yes, I'm saying that the CD is edge case. But in my experience, in my experience over 25 years working in software on it, I've rarely seen an organization successfully deploy a consistent CD implementation across the whole org. Which is if you read the books and if you listen to Brian and perhaps Dave Farley as well.

Certain Dave Farley. That's where the organization really wants to get the most value from a CD implementation. So why is that? Why did orgs not get to a point where they feel they've done CD successfully?

Well, maybe it's not the orgs themselves, maybe with the engineers. Maybe it's the value they're not seeing.

So continuous delivery is often seen as a holy grail for a firm's digital transformation. There was pressure to ensure that the code. And [00:03:00] infrastructure are secure audit-able and updatable the promise of being agile in quotes and being able to release on demand in quotes. And all that good stuff around security or flexibility is too much for any CIO CTO. C I C I S

CSO to ignore.

And quite right. You know, this is the 21st century. It's 2022. We're delivering more and more complex software, more and more often. And that's where organizations want to go. You've read Accelerate. I presume if you haven't, you should. Um, hopefully you've read Continuous Delivery as well by Dave Farley, you should as well, those.

Those are two key books at this point in time in software delivery.

And they promise a brighter future where you can deliver software more quickly more repeatedly, and with more surety over the security, particularly of interest right now. And here you see in snapshot. The dangers of attempting to do the right thing. [00:04:00] If you read the foreword in 'Accelerate', there's a nice introduction by Martin Fowler which urges a note of caution. This says that the research that was used to create the data in the book is based on surveys. And that the Westrum generative organization types should prevail in order to reap the full benefits of the CD implementation.

On the first part, I guess we agree that it's anecdotal, but yes, the irrefutable from a data analysis or statistical perspective that organizations that deliver software more frequently do better. That's the underlying thread of 'Accelerate'.

Secondly though, it's predicated on the fact that the organization types should be Westrum generative. That it should be a nurturing, encouraging high cooperation organization, where the leadership is aligned with the wants and the needs of the engineers to ensure that the software that has been delivered is done so in the most open and collaborative way possible.[00:05:00]

So, I'm not saying CD, isn't a worthy goal, but too often, the leaders of these organizations do not feel they have to change themselves or change the organization to enable CD to be implemented completely. So my point is most org don't support the concept of CD because they don't change.

Some teams might get there, but the overall might not.

So enterprises often don't create the environment that is needed for CD to survive and thrive. They ended up with a mishmash confusion of tools and , and it's very hard, unless policies are rewritten, to make sense of anything like CD.

So this is the drift of my slightly offhand statement. I'm still researching it and looking to prove myself wrong. But my idea and my experience is that we've got ourselves here by being too obsessed with taking it on the latest tool technique. Framework and applying them.

And not [00:06:00] consistent enough when it comes to implementation . Because there's every good book on the subject says without fail, there is no one size fits all solution for any of this. Plus it takes a lot of work on the journey is ready, never complete.

So as orgs, we kind of ignore the idea of the hard work. As engineers, we don't. We take it very seriously. But unless the organization supports that then CD may never be achieved properly.

I continued this thread on Twitter and LinkedIn and got some interesting responses. Most of it was around. Oh, well, you've obviously not read these books on CD before. Or you don't know what you're talking about or. Uh, variety or combination of the two. Well, I have, I have done CICD before I've done it for over 25 years.

And I've written. Hundreds of thousands, if not millions of lines of code. And I'll. Walk the walk for a quarter of a century. Having said that. I've also tried to think about things from an organizational perspective.

Why did orgs get to where they are? [00:07:00] They weren't built on software typically. All the time to begin with, it's become something they've had to bring into their way of working. And we're still getting there, you know, So when we rather glibly say, we need to be able to ensure that an organization understands what it means to deliver software.

Most organizations are not software delivery centered organizations. There's just something that they have to do

So apart from that, apart from the leadership style of an organization which may or may not contribute to a failure or success with CD. How do we get to the situation where CD implementations, aren't really getting to a point where they're achieving what they set out to do in the first place.

So I said that, it had some of these different leadership and something to do with culture. But let me ask you this. Have you ever been in this situation?

Two developers are talking to each other. Developer. One says, well, I consider this to be a framing problem. If we consider it from X's point of view and X is a famous author. [00:08:00] Who's written a book, blah, blah, blah, or something or other. If we consider it from X's perspective or ideology, then this becomes clear.

Developer 2 says maybe true, but have you considered it from wise perspective. And Developer 2 has another famous author Y, which he may have heard about, you may have read Y's book, but. Isn't particularly familiar with it. And Developer 1 says, yeah, I know Y's stuff, but it's a bit, yeah. I looked into it and I don't believe it's as good as X's perspective.

And at that point, Developer to just gives up and says, okay, fine. You've convinced me. Let's go and do this thing your way or your guru's way. Decision made code has written. And I see this as see this pattern occurring, quite a lot in development, but also in the deployment of systems, any debate that we have around. Uh, particularly CICD implementations, particularly around tooling implementations to where there's a lot of touch points. A lot of opinions get thrown in maybe not quoting books per se, but a lot of [00:09:00] opinions get thrown in.

And it becomes a situation where it's, there's a lot of pride at stake, so people will throw a lot of effort into supporting their claim and supporting their argument around the way that we do things, because I think that's also what we see as software engineers. We want to be able to affect the way where we like to think of ourselves as clever people who can't understand complex processes and implement them in code.

So we get this pride in everything that we do. I want to roll it out through our tooling and through everything that we do basically.

And. What I'm trying to say here is that. Pride gets in the way when it comes to building software, but also CD pipelines.

So that's my first point. Point one pride driven development.

My second point is that CD pipelines CI/CD pipelines. Are usually created by heroes.

What I mean by that? Well, not everyone in the team likes to get involved with automation. They like to maybe just do that coding bit. They're not so interested [00:10:00] in the packaging deployment bit and that's fair enough. It can be boring. If you read something like the DevOps handbook, then you can see good patterns, which we want to inculcate in our dev and dev ops for software engineers in the 21st century. And we should all be familiar with the tropes, you know, working in small batches, limiting work in progress, respecting Conway's law ecetera

If you look at the examples in the DevOps handbook in particular.

You will see stories of brilliant individuals or small teams saving enterprises, millions by creating an implementation for CI/CD or Kubernetes or whatever. And typically when you look at these changes they're not really from a team perspective or the success stories don't tend to come from a team perspective. They tend to be, I or we did this, we were parachuted in, or we created this.

And. That's great. And the figures are indisputable of course, but we see in 'Accelerate' that the more quickly organization delivers, then that's better for the organization.

And what we've done here about [00:11:00] typically is the longer term impacts of these implementations. So CRCD can be achieved. Yeah, sure. Bill. And what happens later when the team is disbanded or when the organization changes. CD typically is created by a few heroes who understand that complex domain. That it's living in because it's a product in itself in your, in your landscape and your architecture.

There are one or two people who gravitate towards that complexity and see that as a great challenge, but typically supporting that longer term is less appealing. And you end up with a team which is tasked with looking after. Maybe a CI/CD estate which is failing potentially, um, when your morning break and tests.

If that's not aligned with the team. Or if it is aligned with the wrong team, perhaps, or people move on. Then things can change. So CI/CD degrades over time, unless its maintained. So our hero implementation of CD is great. But what is that after that? .

And this goes into my third point. Which is we never finished what we've [00:12:00] started. A few years ago, I was working at an organization where this was a mantra. We finished what we've started and going on between PIs because we were working in SaFE that we were going in between program increments. Every three months, we would look at the epics that were there for the previous increment and take them across into the next one. Filling up maybe half of our board with stuff, which we haven't finished from the previous three months iteration.

So the irony there We ready finish what we started. Because we've kind of got into a cadence where it's not important to finish what we started. It's always going to be that we've got placeholders.

Similarly for CD, perhaps that's too lofty a goal to aim for directly. If we look at this huge program of work that we take on when we start to implement CD. Maybe that's why we only see pockets of success typically. And maybe that's why. In an organization we don't discuss enough that we could reorganize in a way that makes it attainable.[00:13:00]

So what can we do to fix this? Well for Pride driven development to get that pride out of the way. Language is definitely important and explaining ourselves If you read the domain driven design book by Eric Evans, he'll talk about that in the first few chapters around how language is very important between developer and subject, my tax, but all business analyst or business person. So talking the same language as each other is important. But similarly, I would say talking between developers as this is important as well. There is a kind of a unwritten hierarchy in development. Isn't there where we do use this kind of knowledge based approach , or this pride based approach to have you considered it this way.

And typically the one with the loudest voice or the one who knows the most or the one that's got the most experience can win it.

A debate between developers where we can actually be honest and Socratic, perhaps about how we approach the next significant changes in architecture or [00:14:00] codebase, same goes for CD as well. How, maybe this is the points I've missed as well. I need to dive into is that where does. the application stop and where to CD begin. But that's a, that's a topic for another time I believe.

So what can we do to fix the situation with CD?

Here's a few suggestions I've been thinking about. One get pride out of the way. Treat your CD pipeline as you would your application. Debate its structure and architecture openly, but decide together so that you can build it and support it together.

Two: avoid heroes. Language is important. Ensure that you have a common language, which you discuss your CICD pipeline between developers and DevOps with. And actually understand how you can build it as another application in your organization.

Three, keep working at it. Even [00:15:00] drive organizational change through the attitude that you show to building your CD implementation. This is the way we want to go forward. This is the way we work now. Perhaps I'm even suggesting here that you design your CD rather than just letting it grow. That is something for me to think about and research a bit further.

One thing I really like is the approach that Brian Finster also mentions in a podcast. As Brian says, it's really up to you, the kind of organization that you want to be. I will link the No Nonsense podcast episode in the show notes, but I've also created a short summary post, which talks about the bits that Brian mentions around CD.

He is very passionate about his subject and very knowledgeable. And he goes off on the failings of organizations potentially around CD. And he talks about engineering for release, which I really love. This is the concept that you're building your release mechanism even before you've built the first line of code.

Really, this is taking [00:16:00] TDD to the next level. Before you even write a test or a single line of code, you write your pipeline, which enforces that you have everything you need to do to make your software successful. So all in all, we'll make CD a team problem, not an individual thing. This is something I've also heard from Anne Jan Brower when he talked about it at DevOps Days recently in Eindhoven.

Building the pipeline before you've written a line of code.

Perhaps this approach to CD pipeline creation is something which is fundamental to success. Lots to think about. I've learnt and rewritten a whole lot during the recording of this episode. And I feel it still got a long way. I'd love to hear your opinions on this subject. too, and I'm looking forward to talking to you next time on the Software Delivery Club.

Thanks and good luck.