The Secrets to DevOps Success - 2

DevOps implementation strategy is key focus of most organizations as they embark on Digital Transformation journey. Though it sounds like a quite a straight forward initiative to automate the software delivery process it has many challenges as I have discussed in the past posts. 

DevOps Market size exceeded $7 billion in 2021 and is expected to grow at a CAGR of over 20% from 2022 to 2028 to a value of over $30 billion. In 2021, 65% of the DevOps market’s value in the USA was made up of DevOps solutions (tools) with 37% accounted for by services. By 2028, around 55% of the market’s value is forecast to be accounted for by DevOps services and the remaining 45% by tools.

In 2001 when I implemented my 1st development process automation it was more about automating redundant manual processes to save time and avoid manual errors in the build and release process. We were a small team delivering a small project for a US client , we used to face failure during every release and it was very embarrassing for the entire team to attend the post release meeting. All we wanted to do was a smooth bug free release without spending all night at the server machines. We automated our build and release process and unknowingly we started working closely together to ensure all the issues we faced in the past did not occur again. We collobarated across teams, we stopped blaming other teams, we learned every step of the code/build/test/release/configure/deploy process ,we automated manual tasks and we monitored every step of the release process. Soon we started doing perfect code drops for every release and we started leaving office together to enjoy post release drinks. We were not doing DevOps but experienced a cultural change and we were working as one team.

Over the last few years we are recommending DevOps to our clients as the right way to do the release thing for their business transformation or the digital transformation journey. What we have observed is that in spite of a large number of new tools, dashboards and on demand infrastructure it is still a big challenge to implement a successful DevOps process in an organization. Lets take a quick look at some of things that can help implement a successful DevOps process.

                                                      To implement technology strategically, businesses need to start by creating a cultural shift toward the principles of DevOps. It starts with people and processes, and then products. You cannot simply throw a tool or even multiple tools at the problem and hope that it will be solved. To transform your business, you need to embrace velocity: making incremental changes, delivering small iterations and going faster. This often means disrupting your own business and cannibalizing your existing offerings before disrupting the market. There are a few key elements of DevOps culture that must be adopted before you begin thinking about your product toolkit.

1) Empower teams by embracing collaboration 

Encouraging collaboration is one crucial way to empower employees. By keeping all stakeholders involved in the process, employees can communicate impact in real time, keeping the execution process moving along. Collaboration enables product manager, development, QA, security and operations teams to work together directly instead of waiting for handoffs. The values of diversity, inclusion and belonging are fundamental to creating a culture of collaboration within your organization. Collaboration across teams, across levels brings in multiple perspectives, and by ensuring that each perspective has a say we invite  innovative ideas, empowered teams and smarter, more informed decision-making. The culture of collaboration has to be driven down the hierarchy by the top leadership leading by example and rewarding collaborators. If collaboration is not one of the KPI for management leaders so far it is time to embrace it now.

2) Iteration Planning

You can go faster by breaking things down into smaller pieces. The smaller we split things up, the smaller the steps we take, the faster we can go.' Smaller iteration are better because they take less time, get delivered faster and there is lower risk and have quick turnaround time encouraging people to be more creative.  I remember my mother telling me to take small bites, chew well and the food will show on you, it worked there and it works everywhere. Encouraging iterations is also a step towards moving away from the stagnant waterfall mentality to developing an agile calculated risk taking attitude.

3) Focus on results

Employees should be acknowledged for what they accomplish and complete, not how long it took them or where they worked. . Create a culture where team members feel trusted to structure their own days and do what it takes to get the results that customers require. Start by finding simple solutions to the problem instead of flashy complicated ones. 

It is impossible to transform a business without setting the mood with collaborative culture.Start by finding ways for collaboration in areas where currently you have silos, iterations where there is stagnancy and efficiency where there are lags

The Secret to DevOps Success

Gartner predicted that through 2022 75% of DevOps initiatives will fail to meet expectations due to issues around organizational learning and change and in 2021 Tech Radar Survey indicated 80% of the DevOps initiatives failed to achieve desired goals - mind you project this is the percent of projects that failed to meet the desired goals and expectations. In other words, people-related factors tend to be the bigger challenge while implementing DevOps as compared to implementation technology/tools challenges.

DevOps delivers Maximum value when aligned to customer value

It has been observed that organizations often launch DevOps efforts with insufficient consideration of business outcomes and without clarity of goals. I&O leaders need to ensure that staff and customers connect with the term "DevOps," and the value it will bring, prior to introducing the initiative.

Organizations should use marketing to identify, anticipate and deliver the value of DevOps in a manner that makes business sense. "Leaders must seek to refine their understanding of customer value on an ongoing business to evolve capabilities and further enable organizational change,"

DevOps fails when right team members & organizational change are not managed

In another Gartner 2017 Enterprise DevOps Survey, 88% of respondents said team culture was among the top three people-related attributes with the greatest impact on their organization's ability to scale DevOps. In 2020 TechRadar did a similar survey and over 90% CIOs responded that their priority was to build DevOps culture However, organizations overlook the importance of getting the right mix of staff on board with the upcoming DevOps change and instead focus efforts on DevOps tools.

Tools are not the solution to a cultural problem

It sounds repetitive but I still need to reiterate that "Tools are not the solution to a cultural problem," Organization have to Identify candidates with the right attitude for adopting DevOps practices. Individuals who demonstrate the core values of teamwork, accountability and lifelong learning will be strong DevOps players.

Lack of collaboration affects success of DevOps

Successful DevOps efforts require collaboration with all stakeholders. More often than not, DevOps efforts are instead limited to I&O. Organizations cannot improve their time to value through uncoordinated groups or those focusing on I&O exclusively.

Break down barriers and forge a team-like atmosphere. Varying teams must work together, rather than in uncoordinated silos, to optimize work. "This might start with seeking out an executive who can carry the teams and champion the effort,"

Trying to do too much too quickly

It is important to realize that a big-bang approach — in other words, launching DevOps in a single step — comes with a huge risk of failure. DevOps involves too many variables for this method to be successful in a large IT organization.

Is is recommended to use an incremental, iterative approach to implement DevOps to enable the organization to focus on continual improvements and ensure that all groups are collaborating. Is is recommended starting with a politically friendly group to socialize the value of DevOps and reinforce the credibility of the initiative.

Unrealistic expectations of DevOps

Similar to struggling with grounding DevOps initiatives in customer value, a disconnect exists in many organizations between expectations for DevOps and what it can actually deliver.

"Expectation management and marketing are continuous and not a one-time affair"

Manage expectations by agreeing on objectives and metrics. Use marketing to identify, anticipate and satisfy customer value in an ongoing manner. Expectation management and marketing are continuous efforts, not a one-time affair.The bottom line is unless entire organization understand develops and appreciates the benefit of DevOps and take efforts to collaborate and work to bring a cultural change across Development, Testing and Operations teams, DevOps will not be successful.

Why todays digital enterprises need DevOps?

There was a time in pre-digital era when a New business idea would disrupt the business landscape. Take example of the restaurant industry that we all are familiar with. Let's assume that couple of decades back  the 1st restaurant, with a seating capacity of 10 people decided that it could grow its business if it started taking home delivery orders on phone. The restaurant increased its business from 10 orders/hour to 30 orders/hour - mind you, there is no software used, just a great business idea implemented using the telephone. Recently another disruptive idea that leverages software changed the food business landscape. I am talking about Cloud Kitchen that has disrupted the food business over the last 4 years. The same restaurants that had seating capacity of 10 people could now outsource its kitchen to 10 cloud kitchens across the city and they started servicing 300 orders every hour. If you do not know what is Cloud Kitchen do read about it it most interesting software driven business disrution that many people don't know about.                                                                                                                                              Today businesses continue to be disrupted not just by a new business idea but also by software agility and innovation. Software defined disruption like Mobile apps or Cloud Kitchens have changed the food business landscape and continues to drive tremendous business value like never before.  Is it important to understand that as software is disrupting the industry, every enterprise is turning into a software company. As a software company all these enterprises need a seamless SDLC (software delivery lifecycle) process integrated with testing tools,deployment tools and application and business monitoring tools along with team that can promptly support any issue in the software delivery with a short turnaround time.

The main driver to adopt devops comes from the need of enterprise to innovate and accelerate the business. Unless the new ideas are implemented in the software and shipped fast by a unified capable team the business does not get the desired business outcome.

Key points to note about business and need for devops are as follows-

Businesses develop software for a reason: to get business value

Today every Businesses is either Digital Businesses or in process of Digital Transformation

Businesses will continue to be disrupted by software innovations & agility for example Mobile Apps

Disruption like Cloud Kitchens are changing landscape & deliver tremendous business value

Software disruption also means that enterprises are becoming software companies.

So the main driver for DevOps is the need for enterprise to innovate and accelerate the business

  

Will Auto Machine Learning replace Data Scientist over next few years ?

What is AutoML?

Auto Machine Learning or AutoML enables developers with limited machine learning expertise to train high-quality models specific to their business needs. Build your own custom machine learning model in minutes. Automated Machine Learning (AutoML) is tied in with producing Machine Learning solutions for the data scientist without doing unlimited inquiries on data preparation, model selection, model hyper-parameters, and model compression parameters.

How does the AutoML process work?

Auto Machine Learning is typically a platform or open source library that simplifies each step in the machine learning process, from handling a raw data-set to deploying a practical machine learning model. In traditional machine learning, models are developed by hand, and each step in the process must be handled separately. 

AutoML automatically maps the optimal type of machine learning algorithm for a given task. It does this with two concepts:

1. Neural architecture search, which automates the design of neural networks. This helps AutoML models discover new architectures for problems that require them.
2. Transfer learning, in which pretrained models apply what they've learned to new data sets. Transfer learning helps AutoML apply existing architectures to new problems that require it.

Users with minimal machine learning and deep learning knowledge can then interface with the models through a relatively simple coding language like Python or R. These are some standard steps of the machine learning process that AutoML can automate, in the order they occur in the process:

• Raw data processing
• Feature engineering and feature selection
• Model selection
• Hyperparameter optimization and parameter optimization
• Deployment with consideration for business and technology constraints
• Evaluation metric selection
• Monitoring and problem checking
• Analysis of results

Why is AutoML a game changer?

I think AutoML is game changer because it represents a milestone in the fields of machine learning and artificial intelligence (AI). AI and machine learning have been subject to the "black box" criticism -- meaning that machine learning algorithms can be difficult to reverse engineer. Although they improve efficiency and processing power to produce results, it can be difficult to track how the algorithm delivered that output. Consequently, this also makes it difficult to choose the correct model for a given problem, because it can be difficult to predict a result if a model is a black box.

AutoML makes machine learning less of a black box by making it more accessible. This process automates parts of the machine learning process that apply the algorithm to real-world scenarios. A human performing this task would need an understanding of the algorithm's internal logic and how it relates to the real-world scenarios. It learns about learning and makes choices that would be too time-consuming or resource-intensive for humans to do with efficiency at scale.

Fine-tuning the end-to-end machine learning process -- or machine learning pipeline -- through meta learning has been made possible by AutoML. We can say AutoML represents a step towards general AI and making AI accessible for non techy domain experts. 

Getting started with Auto ML

You can get started by trying some popular AutoML platforms like :

• Google AutoML - Google's proprietary, cloud-based automated machine learning platform.
• Azure Automated Machine Learning - a proprietary, cloud-based platform.
• Auto Keras - an open-source software library developed by the DATA lab at Texas A&M university.
• Auto-sklearn, - evolves from Scikit learn, which was an open source, commercially usable collection of simple machine learning tools in Python. You can find it on GitHub.

Would AutoML replace Data Scientist?

Like most automation, AutoML is designed to perform rote tasks efficiently with accuracy and precision, freeing up employees to focus on more complex or novel tasks. Things that AutoML automates, like monitoring, analysis and problem detection, are tasks that are faster if automated. A human should still be involved to assess and supervise the model, but no longer needs to participate in the machine learning process step-by-step. AutoML should help improve data scientist and employee efficiency, not replace them. Will AutoML reduce dependency of business on data scientist? Yes to a limit it will reduce the dependency on data scientist to do menial machine learning tasks but it also helps in enabling domain experts to use machine learning and applying across less complex tasks.

As of 2021 Auto Machine Learning is a relatively developing area and even the most popular tools are not yet fully developed. If you look back at history of software it is inevitable for automated tools to evolve and automate the mundane tasks reducing dependency on developers. Those who have entered the field of data science without programming background should be wary that one day Machine Learning will become a packaged software and the demand for ML developers my prediction is the demand for ML developers would reduce in another 3 to 4 years maybe around 2025. For every developer/data scientist entering the data science or machine learning space my advice would be to build a strong programming base along with machine learning and AI which will enable them to adapt to the inevitable change in demand of the IT industry.