In the long term of its time as an operating system, service, and software developer, Red Hat has spent a lot of time thinking about development and operations. The result is a thoughtful examination of a path-to-date from Bill Montgomery in an article series starting with “Red Hat IT Begins Its DevOps Journey“.

The strict segregation of duties between operations and development ensures friction and losses. Just when systems head into production, the difficulties suddenly become more obvious–or at least that’s when bugs have the biggest impact. Historically, the two teams have not spoken the same language and have completely different views of the system.

Collaboration between development and operations is particularly relevant in the productive set of software, because there both teams are involved. The collaboration creates synergies, as the development support the operation, for example, directly with specific tools in troubleshooting and operation can optimize the management of an environment for testing or staging.

To optimize the continuous delivery processes at the center of attention, Red Hat started DevOps enablement team called Inception in its IT organization. The idea was to include the principles of agile, lean, and open source to deliver higher quality products and services faster and tools that help enterprises deploy and scale applications.

The starting point

The main goal for the Red Hat team is to show itself as the most open, reliable and least costly of the cloud providers, with the selection of OpenStack as a standard open cloud platform. Red Hat’s ready supply line integrated solutions that include virtualization, cloud management, operating system and functional public cloud.

The company defined the practices of Continuous Integration and Continuous Delivery to address cultural difference across practice. Red Hat implemented a five full-time dedicated and diverse team including system administration, information security, software development, release engineering and agile project management.

The team main focus was to launch the DevOps initiatives and build a DevOps culture. The road map was to improve environment management immediately then “improve other areas essential to CD maturity” and lastly create a “mature CI / CD practice” at the end.

“As an Agile team, we’re refining the road map and decomposing into epics, stories, and tasks at the last responsible moment, with Humble & Farley’s CD maturity model as our compass. But, even standing on the shoulders of these giants, defining a road map for CI/CD that’s engaging and believable has been the most difficult part of getting this new team off the ground,” said Bill Montgomery, who is leading the DevOps initiative.

There are operational considerations regarding compliance, enterprise architecture standards (including ITIL or other methods), IT governance, security, Application Lifecycle Management, application development methodologies, organizational or process-related restrictions and privacy restrictions. For that reason, the team implements open source software so that DevOps models can also be used as a traditional application methods.

One of the first changes made was to introduce two small utilities–jsonstats and talook. These utilities will save a few minutes every week for every developer and release engineer in the organization.

Concrete implementation of DevOps is still a big challenge

While many development teams are now applying Agile software development practices to projects, the development and operations teams on a given project are not always aligned. It’s not all completely smooth sailing though; Bill Montgomery is the first to admit that running a DevOps shop comes with its challenges. The biggest, he says, is finding the right approach to work there.

After about four months running the project, he said that the six individuals from development, ops, security, and program management backgrounds had never worked together and only half had significant experience working on an agile team.

“We had a very broad mission to start with. It took three months to start performing anywhere near the team’s collective potential,” Montgomery said.

Montgomery added that one of the biggest challenge was to change the people’s perspective. Many are not comfortable working on the operations and management side. You need to polite and to begin to trust and then challenge one another.

“We had to develop outside relationships–as a team–with our stakeholders. Plus, what was our mission, and what do we need to do to accomplish it? We all had very different ideas on what “enable DevOps for Red Hat IT” actually meant. Last, we had all the “new team” logistics to sort out (meeting times, seating arrangements, electronic communications, et cetera ad nauseam). All of this took a deceptively high amount of time and effort,” he said.

Development teams are measured on the features they deliver to users, while operations teams are measured by the stability of the system after it has been delivered to users. These are often competing interests in organizations because the development team is not incentivized to ensure the stability of a system once it delivers it to the operations team. Likewise, the operations team is not afforded the appropriate incentives to care about the frequency of releasing new features to users.

At the start, team members had wildly different opinions on what that work should be, ranging from “build a platform-agnostic version of Netflix OSS’ to run on OpenStack, etc.” to “fundamentally change the way departments like Marketing, Engineering, Sales, and IT collaborate with one another. The team did compressive video conference, discussion threads, meetings, surveys, etc. to understand why DevOps is important.

Just as interests between features and stability often compete, tools, processes and languages are often misaligned. Montgomery said for instance, in a meeting sysadmin wouldn’t know the term “full stack”, while a developer can talk about length on this topic.

“If we are having these communication issues with our small, focused, collocated team of native English speakers, then how much is lost in translation when teams send each other tickets at a rate of dozens or hundreds per week in a mid-sized IT shop? How many hours are lost, and customer-impacting errors are made, due to a lack of common language between teams?” he added.

Team members can be more collaborative is by increasing their skill sets in disciplines beyond those traditional specialties that have been established on projects. For example, developers might broaden their skills in infrastructure and databases, while operations engineers could increase their skills in software development and testing.

In summary, Montgomery said that for embarking on a DevOps transformation project, you need to give yourself a little space to form your team, break down your mission, and get everyone speaking the same language. In the process, DevOps enables the whole organization to become more agile and innovative.