**Guest Blog by Dzuy Tran, Klocwork and Helix QAC Senior Sales Engineer, Perforce**

Essentially, Shift-Left is the practice finding defects and preventing them early in software delivery process. This is done by shifting the scanning of defects to improve code quality process to the left of Software Development Life Cycle (SDLC), which is usually composed of four phases of activities: design, develop, test, and release.

Shift-Left also applies to software security testing (SAST). The aspect of shifting left on security is one of a set of capabilities that is the result of higher software delivery and organization performance. According to the 2020 state of DevOps Report published by Puppet Research, high-performing teams spend about 50% less time remediating the security issues than low-performing teams. By better integrating information security (InfoSec) objectives into daily work, teams can achieve higher levels of software delivery performance and build more secure systems with approved compliance standard library and compliance scannable SCA tools.

Production Defects are Costly.

How Does Shift-Left Help Reduce Product Costs?

Production defects are costly, as an identified defect costs about 30 times more than the same defect fixed in development and takes about 5 times longer to resolve. What is more, certain industries — such as embedded software, medical devices, and high-value transactions — experience even higher costs and reputation damage from bugs.

In addition, fixing defects in testing costs about 10 times more than development and takes only about twice the time it would have taken before checking in the code. In waterfall development, it would not present a substantial difference. However, in the continuous integration assembly line methodology, defects found in testing still break the build.

Shifting Left Effectiveness

Shifting left defect detection — from testing and production to developers — is not a new concept. Rather, it becomes more and more critical as software is integrated into more mission-critical systems, IoT devices, and backend enterprise operations. As a result, the cost and impact of production defects increases as well.

When defects are discovered, developers must then find the contributing factors and fix them. In complex production systems, it is not usually caused by a single issue. Instead, it is often a series of factors that interact with one another to cause a defect. This goes for the defects that are found involving security, performance, and availability — all of which are expensive and time-consuming to remedy, and often require some architectural changes. The time required to find the defect, develop a solution, and fully test the fixes are unpredictable — causing delivery dates to be pushed back.

How Effective is Shift Left for Defect Detection and Prevention
The shift left process, optimizes continuous delivery workflows by reducing build pipeline

breakage. It also allows developers to spend less of their time on diagnosing problems and more time on preventing them during development.

In addition, shift left should help enforce more discipline and create awareness of software quality. However, there are also other tools and techniques that can shift-left the defect detection responsibility to the individual developer:

Desktop Static Code Analysis (SCA)

Static code analysis automatically finds potential defects or security vulnerabilities in the source code prior to running the application. SCA tools — such as Klocwork or Helix QAC — can be light and look primarily at code syntax, or more sophisticated and examine the complex application execution paths. Industries such as automotive, healthcare, and aerospace mandate the use of such tools in the testing and validation phase.

Integrating SCA at the build or CI process improves quality and security. True shift-left requires wide adoption of SCA at the developer’s desktop, scanning and cleaning the code prior to checking in rather than waiting for the build to fail.

Use Code Frameworks

By using code frameworks, code components, or libraries, both commercial and open source reduces the volume of custom code developed and therefore eliminate defects making their way into the build. Standard “plumbing” tasks – such as complex UI elements, math and analytics algorithms, data mapping, networking, and so on – can be handled by code libraries while developers focus on true business logic code. Properly tested frameworks that are supported commercially or by a vibrant community, proactively shift-left problem detection by not introducing “plumbing” code defects to begin with.

Developer Side Application Performance Management (APM)

Application performance management solutions provide production performance and failure alerting, which includes analytics designed primarily for production. It’s unusual not to have comprehensive monitoring in place for production applications. Plenty of commercial and open-source solutions are available for different cloud and on-premises environments.

However, they are not built with developers in mind. Lightweight APM tools designed specifically for developers shift-left performance problems and error detection to the developer desktop. Desktop APM — such as XRebel APM for Java and Zend Server developer edition or ZendPHP for PHP – allows developers to proactively optimize code before it enters the integration phase.

Standardize Environments

Adopting a standard, automatically generated application stack and virtualization or containerization that matches the production environments is another shift-left practice. It shifts a class of errors to the developers’ responsibility by not introducing them to begin with. In the cloud or on-premises, standard application stacks reduce chances of configuration and environment mismatch issues from making their way into the build.

Shifting-Left for Security Prevention

Security should be everyone’s responsibility. So, by shifting the security review process to the “left” or earlier in the SDLC phases requires several changes from traditional information security methods with additionally scanning for security vulnerability. However, that is not a significant deviation from traditional software development methods on closer inspection:

Get InfoSec Involved in Software Design

The InfoSec team should get involved in the design phase for all projects. When a project design begins, a security review can be added as a gating factor for releasing the design to the development stage. This review process might represent a fundamental change in the development process. However, this change may require developer training.

Develop Security-Approved Tools

Providing developers with preapproved libraries and toolchains that include input from the InfoSec team can help standardize developer code, and the tools should include a SCA tool — such as Klocwork or Helix QAC — to scan for any security vulnerabilities in the code, such as tainted-data, cross-site scripting, etc.

Using standard code makes it easier for the InfoSec team to help with reviewing it. Standard code also allows automated testing to check that developer is using preapproved libraries. This can help scale the input and influence from InfoSec teams.

Develop Automated Testing

Building security tests into the automated testing process means that code can be continuously tested at scale without requiring a manual review. Automated testing can identify common security vulnerabilities, and it can be applied uniformly as a part of a continuous integration pipeline or build process. Automated testing does require you to design and develop automated security tests (pre- and post-software releases), both initially and as an on-going effort as new security tests are identified.

Blending Quality and Security to Create DevSecOps

The traditionally separate relationship of development and security is long overdue for evolution, which has culminated into a cultural shift known as DevSecOps. The name suggests a blending of development, security, and operations. The DevSecOps methodology is built upon the “shift-left” philosophy of integrating cyber-risk management into the architecture and development process from inception. Built-in, not bolted-on, as they say.

With DevSecOps, security is baked into the code from the start, during the early stages of development. Security is part of the architecture, and the application of automated testing throughout the development process drives a higher level of both product quality and DevOps security. Security issues present earlier, making life easier for developers and less costly for management.

This blog was Created by Dzuy Tran, Senior Sales Engineer, Perforce Software

Dzuy Tran has over 30 years of experience in designing and development of Hardware and Software Embedded Systems, RTOS, Mobile Applications and Enterprise Systems. He helps customers when they have technical questions, assists with Proof of Concepts, and conducts demos of the Static Code Analysis tools and help guided customers on DevOps implementation processes and Continuous Integration deployment. Dzuy holds a master’s degree in Computer Science and Computer Engineering from National Technological University.

Majority of organizations are already deep in their DevOps maturity. Most researches are showing over 40% that adopted the process, and are moving towards automated processes, shift-left, and fast delivery of value to customers.

With that in mind, these organizations that span across different verticals from automotive, financial industries, gaming industries, and many more, need to not only deliver high quality code, but also in many cases meet and comply with specific standards and regulations.

If to only focus on the above compliances, teams must bake into their CI/CD pipelines the scanning against such standards, and ensure that they keep up with modifications that are constantly happening across these.

In addition to the above, there are also very important and critical compliances like Autosar (automotive open system architecture), OWASP (top security vulnerabilities), CERT, CWE, PCI and others.

When dealing with smaller teams, managing the code safety and compliance might be easier, however, when you are 1 squad within a bigger DevOps organization, this requires better governance and control. Keeping up with code changes, merges into different branches, while running continuous testing suites, performing code reviews, and SCA (static code analysis) becomes quite challenging.

Building the Perfect Mix : Safety and Compliance together with Code Standards Adherence

To obtain the right mix, continuously, teams must strategically plan their pipeline in a way that democratizes both continuous functional testing, non-functional testing together with the entire sets of compliances and code standards quality assurance.

If to analyze the above famous DevOps lifecycle diagram, teams can put the proper activities in the right phases of their cycle to cover all of their required goals.

If to look into an example, using an open-source GPS tracing repository , this project has various modules, and quite a lot of Java classes. Ensuring that the code adheres to the proper Java coding standards, as well as does not violate any of the OWASP items, continuously and from CI is not an easy task.

In the below screenshots, you can see that by running a simple Maven SCA job within CI as a batch project upon each code change, can easily generate a comprehensive report (In my case, I am using Klocwork SCA tool):

“kwmaven clean install compile -Dmaven.test.skip=true”

As soon as I run from the project folder the above command, a full build and scan are being performed on the entire code base, using specific compliance modules that I predefined. The developer receives at the end of his Jenkins job a detailed report and can also login to the KW dashboard to review each and every issue or violation (below is a snapshot of the post build report).

In the above case, I was using Java, and was configuring the below sets of compliance coverage to use for the scanning. Obviously, if I was to cover an app from the automotive or other embedded software industry, I could have added an additional taxonomy/ies.

From a process perspective, the developer should follow the process of per each code change, run a build (either using the team CI trigger, or his own local CI. Once he receives a clear report without critical issues surrounding safety, security, and other code quality standards, he can pass his changes toward the next phase of integration testing, functional regression testing and pre-production activities.

It is clear that such code scanning and quality activities, must be filtered properly to avoid redundant noise, false negatives, etc. This is why, relying on SCA tools that can grant the developers the option to filter by severity, modules, configurations and compliances, allows getting the job done while not overwhelming them with irrelevant feedback.

In the above Klocwork zone within Eclipse (or IntelliJ), users could filter through the relevant columns the issues by Severity and more.

To summarize this post: Teams especially within Agile and DevOps practices can enjoy both types of quality gates by employing SCA tools together with coding standards under the same source base, and once these activities generated Green lights, they can allow testing teams to run their jobs with higher confidence.

In most organizations, testing teams are requiring as a pre-requisite to starting their regression an SCA audit report showing that no major issues were detected within the build cycle.

Keep in mind that the above process and tools is 100% automated, and runs within CI which means, ever for large code bases, this is a few minutes of scanning to get to the quality gate with a peace of mind.