Bengaluru, September 11: Today, technological challenges have direct impact on customer satisfaction. This article highlights the importance of monitoring as an intrinsic element of software systems and its impact on user experience and business success. With the evolving industry changes in DevOps, we need to emphasize on the importance of bridging last-mile connectivity challenges and the rapid feedback loop in DevOps.
In this article, we shall learn about ways to automate security testing, vulnerability assessments, and compliance checks. Emerging tools and practices for bridging the gap between public, hybrid, and on-premises environments in the context of DevOps and essential skills for DevOps practitioners.
To give us his valuable and detailed insights on some of the most important aspects within the DevOps industry we welcome Mr. Ashutosh Sharma, Director of Engineering at Myntra in conversation with Mr. Marquis Fernandes, Business Head – Quantic India. Mr. Ashutosh has been a pivotal enabler of innovation and operational excellence and he always believes in having a technology-first approach
1. What sparked your interest in designing and developing cutting-edge solutions within the Platform/Infrastructure Engineering Domain? How has your perception as a professional evolved since the inception?
My journey in the world of Platform/Infrastructure Engineering has been driven by a passionate pursuit of innovation and a relentless commitment to enhancing the scale and reliability of services. When I reflect on what sparked my interest in this domain, it all boils down to two fundamental aspects: direct impact and the thrill of technological challenges.
First and foremost, the opportunity to directly impact and improve the scalability and reliability of services has been a driving force in my career. In the role of Director of Engineering and DevOps, I understand the critical importance of ensuring that our customers experience top notch service quality. By working in Platform/Infrastructure Engineering, I’ve had the privilege of being directly responsible for enhancing the infrastructure that underpins our services, thereby indirectly contributing to customer satisfaction. This tangible connection between my work and customer experience has been an enduring source of motivation.
Secondly, the constant challenge of working with a myriad of complex technologies, including both open-source and proprietary solutions, while simultaneously spanning multiple domains during product development has been both intellectually stimulating and professionally rewarding. This diversity of technologies and domains has enabled me to take a “technology-first” approach. Rather than fitting technology into predefined use cases, I have the freedom to explore and leverage cutting-edge technologies to address unique challenges. This approach has not only broadened my technical expertise but has also enriched my problem-solving skills.
Furthermore, Platform/Infrastructure Engineering offers a distinctive opportunity to bridge the gap in last-mile connectivity challenges that core platforms, such as PaaS from public cloud providers, and other closed or open-source solutions often do not fully address for internal engineering teams. Being at the forefront of solving these challenges empowers us to optimize our internal processes, streamline operations, and unlock efficiencies that might otherwise remain untapped.
One of the most rewarding aspects of this domain is the ability to witness faster results of my deliverables compared to other domains. The dynamic nature of Platform/Infrastructure Engineering means that we can implement improvements and optimizations that manifest quickly in terms of enhanced system performance, resource utilization, and operational efficiency. This rapid feedback loop allows us to iterate and refine our solutions swiftly, delivering tangible value to the organization and ultimately the end-users.
In a nutshell , my journey in Platform/Infrastructure Engineering has been a dynamic and rewarding one. It’s driven by a desire to directly impact customer satisfaction, the exhilaration of tackling complex technological challenges, the freedom to adopt a “technology-first” approach, addressing last-mile connectivity issues, and experiencing the gratification of rapidly visible results. As Director of Engineering and DevOps, my perception of this field has evolved to recognize it as a pivotal enabler of innovation and operational excellence, with an unwavering focus on delivering optimal solutions for our organization and its customers.
2. How can observability and monitoring be integrated into CI/CD pipelines to gain insights into application behaviour and performance?
Integrating observability and monitoring into CI/CD pipelines is not just a technical necessity but a strategic imperative in today’s fast-paced software development landscape. I firmly believe that a robust approach to monitoring and observability in CI/CD pipelines is crucial for delivering high quality software and maintaining a healthy development environment.
First and foremost, it’s essential to recognize that monitoring and observability should be considered intrinsic elements of any system, regardless of the environment it operates in. This extends to CI/CD pipelines, where the stakes are exceptionally high. Metrics such as lead time for change and deployment frequency, directly impact your consumers and users’ experience and, in turn, your business. Understanding these metrics and their correlation is paramount, as they can significantly influence the success of your software delivery process.
One often-underestimated consequence of ignoring the importance of monitoring in CI/CD pipelines is the potential for engineer burnout and dissatisfaction. Broken pipelines not only disrupt the development workflow, but can also lead to frustration among the team. Ensuring the smooth operation of pipelines, the early detection of exceptions, and proactive mitigation of failures are, therefore, equally critical as monitoring any service in your production environment.
To achieve a seamless integration of observability and monitoring in CI/CD pipelines, it is essential to maintain consistency with the monitoring philosophy applied to production environments. This means not reinventing the wheel by creating separate and disjoint systems for observability and monitoring. Instead, your underlying platforms should emit the right telemetry data, which must be captured by your monitoring system and trigger alerts as necessary.
In addition to the specific metrics tied to CI/CD pipeline performance, it’s crucial to focus on actionable golden metrics such as Latency, Traffic, Error, and Saturation. These metrics provide a holistic view of system behaviour and performance, helping you identify bottlenecks and areas for improvement within your pipelines.
While the discussion thus far has primarily centred on monitoring, it’s worth noting that a comprehensive approach should also encompass logging. Log collection, analysis, and alerting should be seamlessly incorporated into CI/CD pipelines, following best practices similar to those used in production environments. Logs can provide valuable insights into the root causes of issues and help facilitate troubleshooting and debugging during the development and deployment phases.
In conclusion, the integration of observability and monitoring into CI/CD pipelines is not merely a technical consideration but a strategic imperative. My perspective is firmly rooted in the belief that a well-executed monitoring and observability strategy within CI/CD pipelines directly contributes to the overall success of software development initiatives, developer satisfaction, and ultimately, the positive impact on your consumers and business.
3. What are some practical ways to automate security testing, vulnerability assessments, and compliance checks within a DevOps pipeline?
In today’s fast-paced development landscape, automating security testing, vulnerability assessments, and compliance checks within a DevOps pipeline is not just a best practice; it’s a necessity. The rapid pace of development increases the risk of introducing security vulnerabilities, and addressing these issues in production can be cost-intensive both in terms of engineer satisfaction and the organization’s market reputation. To proactively prevent vulnerabilities from reaching the production environment, implementing checks and gates in your development process, particularly within CI/CD pipelines, is essential.
Automation is the linchpin of this approach, and it’s worth noting that almost all products, whether closed source, proprietary, or open-source, offer APIs or command line interfaces, making them compatible with a wide range of DevOps tools.
Firstly, define Secure Coding Standards. Start by establishing coding standards that incorporate security best practices. This sets the foundation for secure development.
Secondly, implement SAST (Static Application Security Testing) to analyse source code or binaries for security vulnerabilities. This step identifies issues early in the development process.
Thirdly, integrate DAST (Dynamic Application Security Testing), by following OWASP standards, for scanning your applications. This step assesses applications in a running state and helps identify vulnerabilities that might not be apparent in static analysis.
Fourthly, dependency Scanning, can extend your pipeline’s security by including dependency scanning. This checks for vulnerabilities in third-party libraries and components, reducing the risk associated with external dependencies.
Furthermore, Infrastructure as Code (IaC) and Container Scanning. Apply the same rigorous security checks to your Infrastructure as Code and Docker container images. These elements are critical to modern DevOps practices and should not be exempt from security scanning.
Finally, the challenge that arises with the rapid adoption of open-source software and licensing issues must not be overlooked. Organizations must exercise caution in using open-source software and libraries in their production environments to avoid security and legal complications.
While automating security checks at various stages of the software development life cycle (SDLC) is the ideal scenario, it’s important to address potential pitfalls. False positives are a concern, as they can disrupt pipelines and hinder development activities. To mitigate this, introduce security gates progressively, adjusting their thresholds based on feedback from scanning systems. Regular reporting and addressing critical, recurring development patterns through training and best practices sessions are crucial for a smooth and effective integration of security automation in your DevOps pipeline.
In conclusion, the automation of security testing, vulnerability assessments, and compliance checks within CI/CD pipelines is pivotal to ensuring the integrity and security of your software. By taking a proactive approach and gradually fine-tuning your security gates, organizations can strike a balance between efficiency and safety in their development processes.
4. What are emerging tools, technologies, or best practices are helping Organizations Bridge the gap between public, hybrid, and on-premises environments in the context of DevOps?
Bridging the gap between public, hybrid, and on-premises environments in the context of DevOps can be challenging but is crucial for organizations seeking to optimize their infrastructure, delivery pipelines, and application deployment strategies. Several emerging tools, technologies, and best practices are helping organizations address this challenge:
First is the infrastructure modernisation and unification. This can be achieved during Kubernetes (K8s) and Containers. Kubernetes has become a foundational technology for managing containerized applications across diverse environments. With Kubernetes, organizations can deploy and orchestrate containers consistently in on-premises data centres, public clouds, and hybrid environments.
Secondly, organizations can adopt GitOps. GitOps is an emerging DevOps methodology that emphasizes using Git repositories as the source of truth for infrastructure and application deployment. Tools like ArgoCD and Flux enable GitOps practices, ensuring that configuration changes are applied consistently across environments.
Thirdly, investing in multi-Cloud management platforms. Platforms like Google Anthos, Microsoft Azure Arc, and AWS Outposts are designed to extend public cloud services to on-premises environments and simplify the management of hybrid infrastructure.
Strictly following Infrastructure as Code (IaC). The use of IaC tools such as Terraform, AWS CloudFormation, and Azure Resource Manager allows organizations to define and manage infrastructure configurations consistently across different environments.
Make your CI/CD pipelines hybrid. What I meant is that the build and deploy pipelines should be adaptable to multiple environments. Tools like Jenkins, CircleCI, and GitLab CI/CD enable the creation of flexible CI/CD pipelines that can target various deployment targets, including on-premises servers and multiple cloud providers.
Use APIs and API management tools to create integrations between on-premises and cloud based services. This allows for seamless data and service communication between environments.
Communication among services hosted in hybrid environments is crucial and can pose security challenges as well. Therefore, solve the problem of inter service communication by implementing service discovery across your hybrid cloud and ensure there should be a strong layer of authentication and authorization for services access. This also requires having a strong and reliable identity management system.
Investing in deploying, purchasing an observability and monitoring tool, which is based on open source framework and standards. For example, Prometheus, Grafana, Jaeger are a few examples.
What already talked in detail about security, besides those best practices, it is also important to invest in cloud – native security solutions that can adapt to different environments.
Using tools, like Ansible, Chef, Puppet etc, to automatically provision, manage your infrastructure, application configuration is mandatory to support your hybrid cloud strategy.
Hybrid cloud strategy can create challenges for teams which are operating them. Especially, SRE teams; therefore, it is necessary that those teams should be structured and skilled properly.
We know that none of the clouds follow a single standard, and it complicates life for any engineering building to run applications on a multi cloud environment. There, it is required to have a strategy to invest in platforms which are cloud-agnostic and design services accordingly.
5. According to a senior, experienced leader like you, what should the essential skills of a DevOps practitioner be?
As we know, DevOps is indeed a culture and framework, emphasizing collaboration and continuous improvement in the software development and operations lifecycle. While it’s true that DevOps is not just about individual skills, but rather about fostering a collaborative culture, there are essential skills that empower individuals to contribute effectively to this culture. Here, I’d like to highlight some of these crucial skills:
First is the proficiency with Version Control Systems (e.g., Git): A foundational skill for any DevOps practitioner is the ability to work confidently with version control systems.
Secondly, how to automate provisioning of your infrastructure and do so repeatedly and reliably. Here, understanding and implementing IaC principles is vital. Tools like Terraform and Ansible allow infrastructure to be defined as code.
Thirdly, a DevOps practitioners should be skilled in automating routine tasks, from build and deployment processes to testing and monitoring. Automation streamlines workflows, reduces errors, and accelerates the development pipeline.
Security should be a top priority for DevOps professionals. Knowledge of security best practices, including vulnerability scanning, code analysis, and identity and access management, is essential to protect applications and data throughout the development and deployment process.
Expertise on Cloud Services. In today’s cloud-centric environment, familiarity with cloud platforms such as AWS, Azure, or Google Cloud is invaluable. Understanding cloud services, APIs, and resource provisioning is crucial for deploying and scaling applications in the cloud.
A solid understanding of operating systems is necessary. This knowledge aids in troubleshooting, performance optimization, and efficient resource management.
A DevOps practitioners should be proficient in not only setting up but also efficiently using monitoring systems like Prometheus, Grafana, or ELK Stack. They should understand how to collect and analyse data to ensure the health and performance of applications and infrastructure.
And more important one is collaboration and communication. Soft skills are equally indispensable in DevOps. Effective communication, teamwork, and collaboration with cross-functional teams, including developers, operations, and business stakeholders, are vital for success.
Finally, continuous learning and improving skills in problem-solving and troubleshooting. Though the latter comes with experience, but significant to triage outages and issues quickly.
In essence, DevOps is about fostering a culture of collaboration and automation across the entire development and operations lifecycle. While these skills are essential for DevOps practitioners, it’s equally important to remember that DevOps is a team effort, and the culture of collaboration is the driving force behind its success. Each team member contributes their unique skills to achieve the collective goal of delivering high-quality software efficiently.
6. What should be a leader’s vision in supporting /motivating their team while they face challenges?
A leader’s vision in supporting and motivating their DevOps team during challenging times is critical to the team’s success and adaptability in the ever-evolving tech landscape. Here’s what a leader’s vision should encompass:
Provide a clear vision of the team’s role in enabling continuous delivery and automation. Emphasize how their work directly impacts the organization’s ability to rapidly respond to changes and challenges in the software development lifecycle.
Highlight the importance of building resilient infrastructure and systems. Encourage the team to view challenges as opportunities to enhance system reliability and scalability, making them more resilient in the face of adversity.
Empower team members by emphasizing the value of automation in streamlining processes. Enable them to automate repetitive tasks, allowing more time for creative problem-solving and innovation.
Foster a culture of collaboration between development and operations teams. Stress the importance of effective communication and shared responsibility for delivering reliable and secure applications.
Promote the integration of security practices into the DevOps pipeline. Make it clear that security is a shared responsibility, and team members should be equipped to identify and address security vulnerabilities proactively.
Encourage a culture of continuous learning and experimentation. Emphasize the importance of staying up-to-date with emerging technologies and methodologies in the DevOps space.
Ensure that the team has access to the right tools and technologies to automate, monitor, and manage their infrastructure and deployments effectively. Invest in tools that facilitate collaboration and visibility.
Establish a feedback loop based on metrics and data-driven insights. Help the team understand how their work contributes to key performance indicators (KPIs) related to deployment frequency, lead time, and system stability.
Acknowledge the challenge of balancing speed and stability. Emphasize that the goal is to deliver quickly without compromising quality or security. A leader’s vision should convey the importance of finding the right balance.
Show genuine concern for the well-being of team members. Promote a healthy work-life balance and stress the importance of mental health and self-care, recognizing that DevOps work can be intense.
Paint a picture of the long-term impact of the team’s efforts. A leader’s vision should remind the team that they are not just addressing immediate challenges, but also building a foundation for future success and innovation.
In summary, a leader’s vision for a DevOps team facing challenges should align with the principles of DevOps, emphasizing collaboration, automation, and a commitment to continuous improvement. By providing a clear direction, the right tools, and support for well-being, leaders can help their DevOps teams not only overcome challenges but also thrive in a rapidly changing technology landscape.
To know more about us / publish your article, reach us at
www.quanticindia.com
marquis@quanticindia.com
