Why You Can’t Trust A Single Security Scanner

Why Using a Single Scanning Tool Can Leave You Vulnerable

Written by Ed Engelking, Principal Engineer, AlphaBravo.

AlphaBravo is in the process of building a new security aggregation tool called ABScan. The concept for this tool was born during our time assisting a branch of the U.S. Armed Forces with implementing security in software systems.

Specifically, we observed many teams struggling with the use of multiple vulnerability scanning applications. After some time working with these groups and understanding the specific pain points encountered, the team at AlphaBravo believed it was possible to solve this complex problem.

Using Multiple Security Tools

The Department of Defense typically uses multiple security tools to scan endpoints to determine what vulnerabilities might exist within a given platform. When asked why this was the case, it was explained that the Department of Defense doesn’t necessarily trust a single source to return all possible vulnerabilities. As a result, they deploy at least two different scanning tools and then work to take the output of these tools, manually deduplicate the results, and hope that it provides some meaningful insight into what should be resolved.

AlphaBravo found this to be an interesting use case and set off to determine if we could determine this assumption to be true: do scanning tools agree with one another or would using different tools return different results?

Testing The Theory

To determine if different security scanning tools would return the same results or unique findings, we used two of the most popular open-source products on the market and tested them against one another. We selected a random sample of popular software containers and Git repositories and began the work to scan them all using these tools.

As the tools both support JSON output, this allowed for collecting the scan results, storing them in a database, and generating queries against the metrics we captured. It didn’t take very long to discover several examples from either scanner that was unique or false positive.

This means that if a security professional was relying on a single tool, critical items may go unreported. This provides a false sense of security, as additional vulnerabilities may exist which may be left unresolved and thereby increase the attack surface of a given container or software stack.

Why Do Discrepancies Occur?

Each security tool relies on an underlying database to query and report findings. Different tools make use of different vulnerability databases, which may lead to different results despite scanning the same software packages. Additionally, the rate at which these tools update their databases varies, so one tool may have a more recent dataset than the others.

The two most common vulnerability databases are the MITRE Common Vulnerability and Exposures (CVE) and NIST National Vulnerability Database (NVD). The majority of vulnerabilities discovered are reported to the CVE, which creates a unique record of the vulnerability and provides details such as descriptions, problem types, affected vendors/products, and references to help understand the scope of the vulnerability. The CVE captures vulnerabilities in applications, operating systems, and hardware.

The NVD is a downstream database populated with data from the CVE, with a sync occurring hourly. While the CVE data is useful for understanding a vulnerability that exists, the NVD takes this information and expands upon it. It does so by adding a base vulnerability score, data analysis, reference tags, additional problem types, and known affected software configurations.

However, while incredibly popular and often considered the cornerstone of vulnerability data, the CVE and NVD are not the only databases available. Over time, additional vulnerability databases have become available, some of which specialize in specific software stacks such as Python or Golang, while others are more generic and cover only applications and operating systems while leaving out hardware. Here are some example vulnerability databases:

• Github’s Security Advisory Database
• Gitlab Advisory Database
• Red Hat Vulnerability Database
• Debian Security Advisory Database
• Python Vulnerability Database
• Golang Vulnerability Database
• Ruby Gems Advisory Database

The list of available vulnerability databases grows regularly, and the different vulnerability scanning tools often use one or more of these databases to provide results. However, the frequency of the updates against the upstream databases as well as the total number of databases being imported directly impacts a scanner’s ability to detect a vulnerability. As scanning tools may use different resources, differing results may occur.

Missing Data Is Problematic

One of the bigger issues we discovered during our proof of concept phase was a problem where vulnerability data was not being shared between vulnerability databases. We discovered that some scanners would report CVEs, but in turn, we could not find the results in the NIST NVD or the MITRE CVE databases.

Over time, the AlphaBravo team discovered that this was not a one-off occurrence. Over the past year, we have found many examples where a vendor would register a CVE and publish the details on their vulnerability database, but this information would not be shared back to the CVE database and, in turn, would not be available for analysis within the NVD.

In the wild, this can be difficult to visualize, but because ABScan collects information from multiple scanning tools, performs ETL on the results, and deduplicates the data, the team can easily find examples where a vulnerability has been published in one database but not shared with another. Here are a few examples where our team found some discrepancies:

CVE-2021–38561

This vulnerability is marked as reserved in the MITRE CVE database, and as a result, cannot be found in the NIST National Vulnerability Database. However, when searching for this CVE, it can be found in Red Hat’s Vulnerability Database and the Github Security Advisory Database. According to the notes in GHSA, the vulnerability was closed as it was marked as ignored. As a result, this can be considered a false positive.

CVE-2022–3064

This vulnerability is also marked as reserved in the MITRE CVE database and consequently is not available in the NVD. When searching for this vulnerability, links to the vulnerability can be found that reference an issue with the Go gopkg.in/yaml.v2, however, the links are dead. Attempting to search the Go Vulnerability Database also does not return a result. As a result, this can be considered a false positive.

CVE-2022–3209

As with the examples above, this vulnerability is marked as reserved in the MITRE CVE database and is not available in the NVD. When searching for this vulnerability, it was discovered that an issue existed with the libxml2 package in the Alpine Linux distribution. However, it was not published upstream to other databases such as CVE, NVD, or even the Github Security Advisory Database. This is a great example of why including as many vulnerability databases as possible is important, as even the most mainstream vulnerability databases contained no updates on this specific vulnerability.

Attempting To Solve The Problem

ABScan was created to solve the problem of trust with vulnerability data. It uses two or more security scanners against one or more endpoints, aggregates the results, and provides

meaningful insight into the content which was scanned. Let’s look at a few examples where we have performed security scans and returned useful results.

In the screenshot labeled Example 1, we can see where ABScan performed a vulnerability scan against the PHP container 8.1-rc-zts-bullseye from Docker Hub. A total of 326 results were found and agreed upon by both the Trivy and Grype scanners. So, while quite a few issues exist in the container, both scanners are in 100% agreement with the given results. This is the best-case scenario, and what we hope to achieve with most scans.

Example 1 — Scanning the php:8.1-rc-zts-bullseye container.

However, in the screenshot labeled Example 2, we can see where ABScan ran a vulnerability scan against the python:3-buster container from Docker Hub. A total of 2248 results were returned, but when we look at the Trivy and Grype tabs, we can see that Grype found two (2) individual results that Trivy did not.

Example 2 — Grype found two unique results which Trivy did not.

We can dig in further by clicking on the Grype tab to determine what specifically the scanner discovered. In this case, a CVE against the Mercurial software package made it possible to use symlinks and sub repositories to defeat Mercurial’s path-checking logic and write files outside a repository. Additionally, another vulnerability was shown with the zlib package that could allow a heap-based buffer over-read or buffer overflow via a large gzip header extra field. These results can be seen below in the Example 3 screenshot.

Example 3 — Showing the vulnerability results only discovered by the Grype scanner.

These findings go both ways. While the examples shown above show Grype finding unique results which Trivy did not have, there are just as many examples where Trivy finds results that

Grype did not. In the Example 4 screenshot below, we can see an example scan against the Bitnami Wordpress 6.0.2-debian-11-r9 container, where Trivy found 10 unique results while Grype found one. Had we not used both scanners, we would have missed out on multiple results.

Example 4 — The Trivy scanner finds 10 unique results while the Grype scanner finds one.

In Conclusion

Vulnerability discovery is not a simple process. Many of the security software vendors in the market today are trying to place themselves as the singular solution provider for all security needs. However, the AlphaBravo team using ABScan has provided evidence that this is not the case. Trusting a single vulnerability scanner may leave you open to an attack.

By using a tool like ABScan, an organization can easily scan multiple endpoints using multiple scanners via a simple-to-use web interface. The results allow for more informed vulnerability results to help ensure all possible vulnerabilities are discovered and mitigated across all of your environments.