Mark Twain defined a classic as “a book that people praise and don’t read.” I’m concerned that DevOps is becoming a classic, not because it doesn’t add value, but because it is so hard to get right.

I don’t know of any company that has transitioned to DevOps without a struggle. But I also don’t know of any company that doesn’t consider the results of DevOps well worth it. What worries me is that, just as with Agile, too many people are fooling themselves with half-measures and blaming DevOps. They are putting a little bit of pretty red DevOps lipstick on ill-defined processes that oink, squeal, and don’t really deliver value.

Why DevOps is Hard

Nobody finds it easy to get DevOps going because it challenges organizational mores and requires discipline, new skills, and for people to work in new ways.

The first reaction to DevOps is often pushback. Traditionalists argue that specialization is not a barrier to progress but a way of keeping order. If you breakdown silos and allow just anyone to make operational changes, chaos will ensue. If the entire operational environment is “ours” and not owned by an expert, nothing will work.

Making DevOps work means beating back the naysayers. There are a thousand reasons you can’t do DevOps, and many involve “musts.” Storage must be controlled by storage admins, networking must be controlled by network admins, systems must be managed by system admins, and developers must write code but not deploy it to production. If you build a typical cross-functional team to embrace DevOps, you put people with organizational blinders on in the room, people who think they have good reasons not to do DevOps, which can kill the initiative.

Automation and DevOps

Remember a key ingredient of Agile that people often forget: test-driven development (TDD). TDD is the heart and soul of Agile. If you’re going to move fast, the only way to move fast is to build a test for a desired behavior, code against that test and move forward. When you change your code, you need those tests as a safety net to tell you whether you’ve broken something. Without TDD, it’s not Agile.

For DevOps the equivalent of TDD is configuration tools and automation. The reason that a developer can add capacity to a application or create a new test environment on his own is that as much as possible has been automated. In addition, DevOps usually involves an ever-expanding collection of tests that are constantly being run so that if anything does go awry, the cause can be found quickly and any offending changes rolled back.

Growing Your Own

What skills and resources do you need on a DevOps team? DevOps requires what I call “system engineers,” which is a totally different kind of skill set, one that is almost impossible to hire.

My experience at NASA illustrates the difficulty of creating a DevOps team. At NASA when we tried to bring in ops people to run our DevOps process, the first thing they wanted to do was lock all of the developers out of the infrastructure. It was a colossal failure. We tried hiring system administrators and having them work with configuration management tools and automation. That didn’t work either.

To staff a DevOps team, you have to grow and nurture people. You need someone who has an interest and a drive to get to know others and learn what they do. Sure, on rare occasions you’ll find somebody who has grown up as a system administrator, or on the operations side of the house, who actually understands computer science beyond a couple of bash scripts or a little bit of Perl. They can start to understand test-driven development.

How DevOps Makes Things Faster

I’ll close with a fundamental idea from systems thinking: The effectiveness of a system is related to the connections between the components, not the efficiency of individual components. Let’s apply that to DevOps. If you can provision a virtual machine in five minutes but it takes a week to get an IP address, you didn’t get a virtual machine (VM) in five minutes; it took a week. If you can’t get storage configured for a month, it’s really a month before the VM is useful. It’s the relationships between systems that really matter, not the individual components. In essence DevOps means working together toward a single goal. We need to deploy as fast as possible. The only way that we can do that is by working in synergy, not in silos.

When you successfully produce a DevOps team, everybody in the building knows that the most important activity is pushing code to production; there is a common objective. In a DevOps world there is still specialization, but there is a common goal. The sense of “your” apps running on “my” servers disappears. Everything is “ours.”

Once this happens, you’ve got a real shift toward DevOps culture. You can add more tests and automation; there will still be kinks to work out, but you’ll have a real DevOps team, not an entry to the state fair.

Joshua McKenty’s bio:

Prior to co-founding Piston Cloud Computing, Joshua McKenty was the Technical Architect of NASA’s Nebula Cloud Computing Platform and the OpenStack compute components. As a board member of the OpenStack Foundation, Joshua plays an instrumental role in the OpenStack community. Joshua has over two decades of experience in entrepreneurship, management and software engineering and architecture. He was the team lead for the development of the Netscape Browser (vs. 8) as well as AOL’s IE AIM toolbar, and a senior engineer at Flock.com. He also led the successful first release of OpenQuake, an open source software application allowing users to compute seismic hazard, seismic risk and the socio-economic impact of earthquakes.