In my blog earlier this year, I sat down with Ben Chicoski from CloudBees to discuss the importance of their technology in the federal government, the work they’re doing in the SDLC, and the role they play in DLT’s Secure Software Factory.
Today, I am very excited to share a conversation I recently had with Larry Johnsen, VP of Sales for the Military/Aerospace/Government Solutions verticals at Parasoft.
Parasoft plays a critical role within DLT’s Secure Software Factory and the overall SDLC. They specialize in an area that many organizations/agencies struggle with – automated testing – and have been doing it for over three decades. Interestingly enough, the timing couldn’t be more perfect for this conversation, as we continually hear from various government panelists/speakers how important an automated testing tool is for an agency standing up a DevSecOps practice, or “shift left mentality” when it comes to baking in security (some would argue it is impossible to stand one up without an automated testing tool like Parasoft). However, there seems to be a stigma or an over-reliance on manual testing or free testing tools. We explore this topic and others in our discussion below:
(Jim) - Who is Parasoft? What challenges do you help government agencies solve?
(Larry) - Parasoft focuses on one key area, automated software testing, and is a pioneer in this space. For more than 30 years, we have been building forward-thinking technologies to streamline software development and testing for our government customers. Parasoft’s solutions are all designed for government agencies, both defense and civilian, to build security and quality into their mission-critical systems.
Providing efficient and effective automated software testing solutions allows government agencies to reduce overall software development costs, while accelerating the delivery of high-quality and highly secure applications. What slows government agencies down in their delivery focus is the inability to deliver quality and security at speed. This is caused by many complexities involved with any modern software systems. For instance, a modern IT application may consist of tens of thousands of lines of application code across many architectural components, supported by complex workflows and user-interfaces leveraged to manage a myriad of integrated systems-of-systems many of which are outside the control of the agency responsible for building and delivering this new application. There are just too many critical moving parts!
Parasoft provides automated software testing solutions that drive quality and security at speed of all the parts: the source code itself, individual service-level components, the ever-evolving integrated-system and even the UIs. All to ensure testing is done early and robustly while helping keep these key government programs on schedule.
(Jim) - How can Parasoft assist agencies when development and testing are outsourced?
(Larry) - Government agencies, both civilian and defense, often have quality and security concerns about outsourcing development and testing to system integrators (SIs). Agencies ask: Will the SI deliver a high-quality, secure-minded system? Or will quality and security be an afterthought?
This is an extremely valid concern. Especially, if there are zero or weak contractual requirements targeting policies and processes around how the SI approaches quality and security. This is oftentimes the case.
Historically, system functional requirements are well detailed regarding the required operational capabilities, which are dictated as "shalls.” While some key nonfunctional requirements, like quality and security, from a pure foundational perspective, may be less stringent. They're described as "shoulds" with recommended "guidelines.”
For instance, I've personally seen nonfunctional requirements state that the program “should follow quality guidelines like MISRA and provide a report…” There's nothing stated to ensure that the foundational source code is built with security and quality right from the start. Just like, "You can’t test quality into a system," it’s also true, "You can’t test security into a system." Both quality and security must be built into the system as it’s being developed.
For the sake of discussion, let’s assume that the SI is contractually obligated to leverage key quality and security standards as part of their development processes, using static analysis technologies to help drive these best practices. Even then, the agency may still have concerns because they have no visibility to what, if anything, is really going on. The concerns beg answers to questions:
- Are repeatable processes established and being followed?
- Are developer-focused scans being done prior to check-in to source control?
- Is there analysis on the parts as well as the whole? How often?
- When dangerous issues are found, are they being fixed?
Many of these questions and concerns are left to chance. That leads us back to your question. How can Parasoft assist?
- Would it alleviate concerns if the agency had real time visibility to the SI’s alignment with quality and security throughout the span of the contract? That includes CWE, OWASP, CERT, and other security and quality standards tied to the contract.
- Would it be of value to the agency to know that security and quality are systematically being built into their system?
If "yes" is the answer to both questions, then Parasoft can definitely assist.
Parasoft Development Testing Platform (DTP) is web-enabled and consolidates testing results into intelligent dashboards, detailed reports and actionable analytics. SIs can leverage results of automated security and quality source code scans driven by Parasoft’s Static Application Security Testing (SAST) solutions like C/C++test, Jtest or dotTEST.
These automated scans for C/C++, Java, C#, and other languages are automatically published to DTP. SIs can control the level of reporting visibility to give agencies. While the SIs don't want to distract and confuse agencies with low-level, fine-granular aspects of their processes that are only relevant to them in the throes of development and testing, they could provide the agency with a high-level, real-time view into development progress toward building in quality and security as the system evolves over time.
Any DTP reporting provided by an SI to an agency could simply be an online walk-through of key quality and security information via online meetings. Alternatively, SIs can share live feeds directly to agency-specific interactive dashboards with pie charts, histograms, heatmaps and more. These dashboards reveal high-level quality and security information like ongoing trends analysis, current alignment to chosen standards such as CWE, OWASP and more.
Regardless of the level of access, or whether visibility is required into the quality and security of a single system or rollup of a portfolio of systems, Parasoft DTP enables both the agency and SI to strike the necessary balance to keep both parties on the same path toward success.
(Jim) - What are the cost benefits of using software like Parasoft provides?
(Larry) - Costs to an organization come in many flavors. Let’s take a look at a tangible, expensive resource: the full-time engineers (FTEs).
Imagine agency X with 100 developers at a full-laden rate of $100K. The cost to Agency X from a pure manpower perspective is $10M per year. There are industry estimates that developers spend 40-50% of their time doing testing and debugging, which amounts to over $4M per year.
What if you could streamline this testing with efficient and effective software test automation? This is where Parasoft shines and where Parasoft separates itself from the pack.
Each Parasoft solution is designed to slice costly testing cycles by enabling automations tied to three key areas: test creation, test execution, and issue remediation. Parasoft’s set of language solutions support static analysis, unit testing and code coverage across several development languages including Java, C/C++ and C#. Let’s speak to the three key areas as they apply to Parasoft Jtest, for example, which focuses on test automation of Java applications.
From a test creation perspective, Jtest’s Unit Test Assistant (UTA) enables the automatic generation of Junit tests and stubs and supports Java framework configurations like Spring. This eliminates, by upwards of 50%, much of the traditional and costly manual aspects of unit testing. Many development organizations spend 20% of their overall time supporting unit testing efforts. This could amount to a yearly savings of more than $1M.
Let’s talk about the test execution. Imagine that these 100 developers are together creating tens of thousands of tests and much more over an application's development lifespan. Parasoft has visibility and traceability to pinpoint exactly which test case, when executed, drives which lines of code in the underlying Java code base. With this knowledge, Parasoft’s Test Impact Analysis (TIA) capabilities allow for regression testing focused on only those tests that are impacted by code changes. There’s no reason to run thousands and thousands of tests again and again if they never exercise the updated code. Not only can developers reduce their personal testing cycles leveraging TIA, but for those organizations driving CI/CD frameworks, this time saver gives back many minutes, if not hours, to each automated CI/CD regression run.
Lastly, issue remediation is a key area that most organizations don’t consider when looking for cost/benefits tied to FTEs. The reality of software testing and especially regression testing is that tests will fail. When they do, how quickly you determine where the issue occurred, who might be responsible for remediating such an issue, and providing an environment to reproduce this issue all significantly impact Mean Time to Remediation (MTTR). Parasoft extends its visibility and traceability capabilities to also understand:
- Who touched what code last?
- What code comprises which components?
- Which components are integrated into which subsystems?
Parasoft understands the entire popcorn trail of development and expedites the remediation process by pushing the right information to the right person immediately. MTTR is greatly reduced and saves an organization an enormous amount of wasted FTE cycles tied to head-scratching, sifting through cryptic logs and more. At the end of the day Parasoft’s automated software testing solutions are all keenly aware that time is money.
(Jim) - What role does Parasoft play in DLT’s Secure Software Factory?
(Larry) - Parasoft plays an essential role in DLT’s Secure Software Factory (SSF). With SSF’s focus on providing a secure software pipeline to government agencies, Parasoft provides continuous quality mechanisms ensuring that SSF deliverables are built and tested with security and quality at the forefront. Whether it’s a SAST approach for security and quality compliance at the code level or streamlining component- level integration testing and application-level testing, all security and quality watermarks must be met to ensure our government’s systems are robust and safe.
DLT’s SSF, because it is truly DevSecOps-enabled, it has a primary focus to ensure security is built into the DevOps focus on software delivery. Security doesn’t just happen. There must be a concerted effort to build in security as systems evolve. And as the complexities of these systems grow, implement continuous quality measures to ensure security and quality do not fall through the cracks. This is where Parasoft shines.
Parasoft takes a security-first approach to establish a baseline on security analysis and compliance to support the ever-evolving system. Some security-focus compliance areas might include CWE, OWASP, CERT, DISA-STIG, etc. As these systems evolve, Parasoft is continually scanning for adherence to policies and reporting issues and key supporting test artifacts directly to personnel. They're empowered to quickly and effectively remediate any findings, continually raising the watermark of quality and security as applications evolve. In the end, the SSF leverages Parasoft’s continuous quality solutions to ensure that applications are safe and robust when delivered to our government’s end users.
(Jim) - How has artificial intelligence advanced automated testing capabilities for the public sector?
(Larry) - This is a great question and targets the heart of Parasoft’s automated software testing solutions. Parasoft employs artificial intelligence (AI) and machine learning (ML) in all our solutions for all layers of the software testing pyramid. As most folks know, the software testing pyramid is a foundational approach on building high-quality and secure solutions from the ground up:
- At the source code layer through best practices like static analysis and unit testing.
- At the component layer, focusing on API testing and component-based load and performance testing.
- At the UI level where ensuring that the functional user experience meets user story expectations and other requirements.
Parasoft is the only company leveraging AI/ML at all layers of the pyramid, focused on positively affecting an agency’s end game to deliver mission-critical applications with purposes toward speeding up delivery, increasing quality and security while reducing costs, and reducing risks of the system in production.
Let’s take the source code layer first. When applying Parasoft’s static analysis solutions to look for high priority, quality, or security issues tied to compliance standards such as CWE, OWASP, CERT, DISA-STIG, MISRA, AUTORSAR, etc., there are times when detected issues may be categorized by the user as being false-positives. Meaning that while the automated scan believes it found an issue, it may not necessarily be one due to a number of factors including design of the software. Maybe high-level checks were already put in place to mitigate the perceived vulnerability or issue. Another factor could be that scan depth or breadth were not calibrated appropriately. These are just a couple of the several possible reasons.
An organization’s review of false-positive issues can take an inordinate amount of inspection man hours to determine its false nature, which costs money and time — not to mention frustration for those doing the reviews. Parasoft substantially reduces this type of inspection and remediation noise by employing AI/ML. Parasoft leverages AI/ML by keeping track of how a development group reacts to patterns of issues, by either acknowledging the issues and fixing them or suppressing them due to realization they are false positives. Parasoft uses AI-based clustering analysis to make predictions on whether a found issue will fit ultimately into a fixable category or one that will eventually be suppressed as a false positive or irrelevant issue. Parasoft can provide category accuracy up to 90% with as little as twenty-five data points of prior or modeled team behavior in how they fix/suppress issues. The team can now focus their core attention on those highly predicted fixable issues. This capability provides schedule savings and reduces costs all while keeping developers sane and their code base true-to-form for quality and security.
Let’s jump up to the UI level where many organizations are still doing a lot of manual testing or making the move into key UI test automation technologies like Selenium. Don’t worry, we aren’t skipping the component-based layer of the pyramid where API-driven testing comes into play. So bear with me. You’ll quickly understand that by applying AI/ML at the UI testing layer, you can kill two birds with one stone by leveraging AI/ML once again, behind-the-scenes, to automatically create API-driven tests as an outcome of your UI testing!
Without going into too much of the gory details about Parasoft Selenic and Parasoft SmartAPI, let me tell you how the combination of these two solutions, support by their AI/ML capabilities, allow agencies to exponentially evolve their software test automation maturity.
Although Selenium is a UI test automation technology that has estimated utilization in 65-75% of testing organizations and provides a lot of great UI test automation capabilities, it has its downsides. The first being the maintenance and upkeep of the Selenium tests as the UI constantly changes. As locators, wait times, and other key areas of an underlying UI change, the associated Selenium test suites fail to run successfully. This can wreak havoc on the successful forward momentum of application releases.
Add to this, the maintenance nightmare trying to determine where the failure occurred, what exactly the issue was, and how to fix the issue. Organizations can have thousands of automated Selenium tests and if something goes wrong — and it will — the production line slows or stops, which negatively affects the velocity of the delivery and increases costs to the program.
Here's where Parasoft steps in to help.
Parasoft Selenic monitors Selenium test executions. If a runtime issue is encountered, let's say tied to a bad locator or wait time, Selenic’s AI-enabled self-healing capabilities analyze for potential fixes and steps in mid-stream to heal the test, allowing the test to run to successful completion.
Selenic doesn’t physically modify the Selenium test suites based on its self-healing. Instead, Selenic provides the tester a report revealing its AI-decided fixes, calculated probability of success and a full list of other potential fixes with their respective ranked probabilities.
With this report the tester can quickly go back and decide if Selenic’s runtime determined modifications were correct. If they are, then Selenic can make the fixes and apply them to the entire test suite so builds are not broken. And there's enormous value in streamlining the remediation process for all of Selenium testing going forward, significantly reducing the maintenance nightmare traditionally tied to Selenium testing.
That’s just one aspect of what Selenic does. In addition, by recording the manual interaction of a given web-based application, Selenic can automatically create Selenium tests based on the page object model (POM). This is a great way for teams to bootstrap themselves into modernizing their UI test automation practices, if they're not already using Selenium. While this is a powerful capability, these same browser recording capabilities also provide the added benefit of giving testing organizations insight into the API messaging layer to the backend application servers while exercising an application. This is where Parasoft SmartAPI’s AI/ML capabilities step in.
Parasoft SmartAPI monitors the message traffic to and from the application backend as a tester manually exercises the Web application. From this interaction SmartAPI automatically creates API-level tests specific to the operations exercised on backend services. Also, through its AI/ML, SmartAPI is able to learn and understand underlying data payloads and their relationship across multiple service operations.
With this knowledge, SmartAPI can stitch together the sequences of these API operations and automatically generate API scenario tests replicating the flow across the various services. Now, the testing organization has a set of API level tests they can easily automate as part of their CI/CD framework where their automated execution help to better isolate potential issues in complex systems and these same scenario tests can be easily automated to support load/performance testing on the application services as well.
That’s a lot to take in. To sum it up, if an agency is looking to make the move from manual testing to a modernized push for CI/CD test automation, then the coupling of both Parasoft Selenic and Parasoft SmartAPI’s AI/ML capabilities provides enormous value. Imagine that with a single manual testing session you can go from having zero test automation to having an ecosystem of CI/CD-ready self-healable Selenium tests along with their associated API scenarios tests supporting both functional and load/performance testing. This is a real game-changer for testing organizations in the public sector.
(Jim) – If an agency is interested in procuring Parasoft, what is the easiest path for them to do it?
(Larry) - Parasoft is available on major GWAC contract vehicles (CIO-CS, SEWPV), and we’re working with Jim and his team at DLT to get Parasoft on their GSA IT Schedule 70 contract vehicle.
Related Blog Posts
