Guest Blog Post by: Tzvika Shahaf, Director of Product Management at Perfecto & A Digital Reporting and Analysis Expert
Intro
One of the DevOps challenges in today’s journey for high product release velocity and quality, is to keep track of the CI Jobs/Test stability and health over different environments.
Basically, this is one of the top reasons, bugs slips to production. In other words, lack of visibility into the DevOps delivery pipeline.
Real Life Example
Recently, I’ve met a director of DevOps in a big US-based enterprise who shared with me one of his challenge in this respect.
At the beginning of our meeting he indicated that his organization’s testing activity lacks a view for the feature branches that are under each team’s responsibility. This gap creates blind spots, where even the release manager is struggling to assemble a reliable picture of the quality status.
The release manager and the QA lead are responsible to verify that after each and every build cycle execution, the failures that occurred are not critical and accordingly approve the merge to a master branch (while also issuing a defect in Jira for bugs/issues that weren’t fixed). The most relevant view for these personas is a suite level list report as the QA lead is still busy with drill down activity to the individual test report level as he is interested also in understanding the failure root cause analysis (RCA).
As part of the triage process, the team is looking to understand the trending history of a Job to see under each build what was the overall test result status. The team is mainly interested in understanding if the issue is an existing defect or a new one. In addition, they look for an option to comment during the triage process (think about it as an offline action taken post the execution).
Focusing on the Problem
So far so good, right? But here’s the problem: the work conducted by each team is not siloed in the teams CI view. There’s no data aggregation to display an holistic overview of the CI pipeline health/trend.
Each team is working on different CI branch and the other teams during the SDLC have no visibility to what happened before/happens now.
Even when there’s a bug, the teams will be required to issue the defect to a different Jira project – so the bug fixing process adding more inefficiency, hence time to the release process.
When the process is broken as identified above, each new functionality on the system test level or stage, is being merged while not all failures are being inspected (lack of visibility from within the CI).
Jenkins will ignore the system test results and merge even if there’s a failure.
The Right Approach to Solving The problem
The desired visibility from a DevOps perspective is to cover the Jenkins Job/build directly form the testing Dashboard across all branches in order to understand what was changed in that specific build that failed the tests: which changes in the source code were made? Etc.
What if these teams had a CI overview that capture all testing data that includes:
- Project name & version
- Job name /Build number
- Feature Branch name or Environment description (Dev, Staging, Production etc.)
- Scrum Team name – Optional
- Product type – optional
Obviously, Items 1-3 are a MUST in such a solution, since when displayed in the dashboard UI, teams gain maximum visibility into the relevant module the user is referring to when a bug is found, and this is also part of the standard DevOps process as well. Double clicking on the visibility and efficiency point, such options can significantly narrow the view of the entire dashboard for each team lead/member , and help them focus only on their relevant feature branches.
When the QA lead reviews the CI dashboard he/she can mark a group of tests and name the actual failure reason, that can sometimes be a bug, a device/environment issue or other.
Feel free to reach out to me if you run into such issues, or have any insights or comment on this point of view – Tzvika’s Twitter Handle
Thank You
Tzvika Shahaf
Continuous Testing and DevOps Blog 