How We Test OSSScan: 40+ Real Projects, Every Layer, Every Run

If OSSScan tells you a package is copyleft, or that a license is unknown and needs a human, or that your coverage is only partial, that answer has to hold up. It also has to be stable. The same project scanned today and next week should produce the same verdict, or we have a bug hiding behind noise.

So we built a test system whose whole job is to keep OSSScan consistent and accurate as the code underneath it keeps changing. This post walks through how it works. We are not going to dwell on any particular bug we caught along the way. The interesting part is the breadth: how many real conditions we try to put in front of the scanner before a release ever ships.

We test against real software, not toy fixtures

At the center of the system is a corpus of more than 40 well-known open-source projects. These are real repositories, cloned as they are, chosen deliberately to stretch the scanner across the messy variety of how software actually declares its dependencies.

The spread is the point. The corpus covers, among others:

• JavaScript / TypeScript: React, Next.js, Deno
• Go: Hugo, Kubernetes, Grafana, Syft, Grype
• Rust: the Rust toolchain samples, Serde
• Python: Django, Home Assistant, conda-lock
• Java / JVM: Apache Commons Lang, Jackson Databind, Gradle projects
• PHP: Laravel
• Ruby: Rails, Mastodon
• .NET: the .NET runtime, eShopOnWeb
• C / C++: FFmpeg, Mosquitto, CMake with Conan, vcpkg async samples
• And more: Swift and CocoaPods, Dart and Flutter, Elixir and Phoenix, R (renv), Terraform, Helm, Nix, and others

Each benchmark ships with a human-readable expectation note, one file per project, recording why we picked it, which findings actually matter, what noise is acceptable, and which gaps are already understood. That note is the reasoning behind the automated checks, written down so it does not live only in someone's head.

Why so many ecosystems? Because every packaging system states licenses and pins versions differently, and each has its own way of being ambiguous.

A scanner that is correct on npm can be quietly wrong on Conan or renv. The only way to know is to point it at all of them, over and over.

A layered pipeline, fastest checks first

We do not run one giant test. We run a staged pipeline, ordered from cheapest and fastest to slowest and most thorough, so a quick pass can catch obvious problems in seconds while the exhaustive pass grinds through every benchmark before a release. Every stage runs even if an earlier one fails, so a full pass gives you the complete picture in one shot rather than stopping at the first problem.

The stages, in order:

1. Unit tests. Fast, deterministic, no network and no benchmarks. They pin down focused logic like license-name normalization, attribution categories, and how a prior review carries forward.

2. CLI smoke test. Scans OSSScan's own repository through the real headless app and confirms it still produces an SBOM, a CVE report, a license report, and the bulk-export folders that AI agents consume.

3. License classification. Described in its own section below.

4. Coverage assessment. Validates the “Check Project Coverage” grade (Strong, Partial, or Limited) against every benchmark, so the scanner is honest about how much of a project it could actually see.

5. CLI benchmark suite. Scans every benchmark headlessly and checks each result against its recorded expectations.

6. Evaluations. A human-in-the-loop path: auto-classify a project, mark a license finding and a CVE finding as problems with reviewer comments, export, and confirm the review carries forward into the next delta export. This exercises the workflow a real reviewer uses, end to end.

7. CLI vs UI comparison. Described below.

8. Cross-tool SBOM comparison. Described below.

The whole thing writes a single timestamped pass or fail report, plus a running scoreboard that lets you re-run just the benchmarks that need another look without re-running everything. Discipline like this matters more than it sounds: benchmark folders are shared across stages, so a stray build artifact left behind by one scan could quietly skew every later scan of that project. A cleanup stage sweeps them all before anything else starts, so every pass begins from a known-clean state.

The CLI and the UI must agree

OSSScan can be driven two ways: through the graphical app and through a command line meant for automation and AI agents. Those are two code paths to the same answer, and a scanner is only trustworthy if they never disagree.

So we scan the same project both ways and diff the results: packages found, license assignments, CVE and license exports, auto-classified items, and evaluations. We do this in two flavors on purpose:

• A comparison run reports every difference, so we can see and reason about it.
• A parity run treats differences as failures. It also builds the command line from the app's own “create command line” builder, so if that builder ever emits an incomplete or wrong command, the parity run catches it rather than letting it ship silently.

Together they mean a change that helps the UI but quietly drifts the CLI, or the reverse, cannot slip through unnoticed.

License classification: never guess, never under-report

Getting a license right is the whole game, so it gets a dedicated harness driven from a single source of truth: one entry per current SPDX license identifier, each paired with the risk tier OSSScan should assign it. From that we generate two synthetic bills of materials.

That last part is the important discipline. When the data is ambiguous or missing, OSSScan fails closed to “unknown, needs review” rather than guessing a convenient answer. There is a specific reason this matters so much in OSSScan: an unknown is not a dead end, it is a handoff. It is what gets routed to an AI agent to investigate, read the actual component, and reach a real conclusion. A scanner that guesses a wrong-but-plausible license does not just record a bad row, it feeds that false confidence into the agent and quietly steers the investigation off course. An honest “I do not know” sends the agent in to look; a confident wrong answer tells it there is nothing to look at. So a wrong “safe” guess is worse than an honest gap, and this test exists to keep the scanner honest about the difference.

Cross-checking against other tools, and the network itself

We do not assume OSSScan's view of a project is the only possible one. The cross-tool stage generates a bill of materials for the same project from independent tools (GitHub's own SBOM, cdxgen, and Syft) and compares license coverage and package counts across all of them. Where they diverge, we want to understand why, not paper over it.

We also treat the network as something to verify, not trust. During scans, tool traffic runs through an isolation proxy whose log is a security surface: it is how a user can tell whether a scanner reached somewhere it should not have. Tests hold that logging to account, so what the proxy reports is accurate and something a user can rely on.

Auditing the bundled tools for source-code upload

OSSScan orchestrates three third-party tools as child processes: Syft, which builds the bill of materials; Grype, which checks it for CVEs; and ScanCode, which reads license text. All three run with full read access to the source tree you point them at, which may be proprietary or sensitive. So we guard against a compromised or trojaned tool binary quietly shipping your source off-machine, and we do it two ways that back each other up.

The first is runtime enforcement. OSSScan's network isolation mode blocks internet access for those three tool processes during a scan while still allowing OSSScan's own enrichment lookups (ClearlyDefined, deps.dev, package registries). That is a live control that stops egress as it happens, and its logging is what the isolation proxy tests above lock down.

The second is a periodic point-in-time source review. On a recurring basis we pull each of Syft, Grype, and ScanCode to its latest commit and audit the actual source for any path that could take raw file contents, not just package names, versions, and hashes, and put them into an outbound request by any mechanism: HTTP bodies, headers, and query strings, but also VCS clones, shelled-out git / scp / rsync / ssh / curl, telemetry and crash-reporting SDKs, and custom transports. Each tool is reviewed on its own as a full-attention pass, and destination-selection logic (proxy chains, mirror lists, fallback branches) is traced through every branch, since a second egress mechanism can hide behind a case next to the one you already found.

The tools are held to different bars, because their expected behavior differs. Syft and Grype legitimately make some network calls to resolve package metadata, so for them the question is precisely what data flows into each call, and every call site is cataloged and tagged either package-metadata-only or needs-attention. ScanCode is expected to make no outbound calls at all, so the bar there is stricter: any network-capable code path is noteworthy, and a clean result still has to show its search methodology so the “zero network calls” claim is reproducible rather than merely asserted.

We hold the same standard to the thing we hand you. Before a Windows OSSScan build is uploaded for download, it is submitted to VirusTotal, which runs it past a large panel of independent antivirus engines. It is one more point-in-time check, on our own artifact this time rather than a bundled tool, so a build has to clear an outside verdict and not just our own word before it reaches you.

Same result on macOS and Windows

OSSScan runs on macOS and Windows, so “consistent” has to mean consistent across both. We run the full pass on each platform and then diff a compact results digest between them, surfacing any place where the same project grades differently on one operating system than the other. Cross-platform drift is exactly the kind of subtle inconsistency that is easy to miss and embarrassing to ship, so we look for it directly.

Practices that keep the suite usable

A test system only helps if people actually run it, so we put real effort into making it easy to run and easy to act on.

One command runs everything. A single script runs the whole pipeline, fastest stage to slowest, and writes one pass or fail report. It has lighter lanes too: a quick sanity subset for day-to-day work, and the exhaustive every-benchmark sweep reserved for a release gate. Before it runs, it checks its own prerequisites, so a missing dependency fails loudly up front instead of halfway through an hours-long pass.

Re-run only what failed. A full pass writes a running scoreboard: one entry per test, each flipping to pass or fail in place as it runs. When something needs another look, one command reads that scoreboard, finds every entry currently failing, and re-runs exactly those. You can chase a problem to ground and watch the one scoreboard converge to all-green, without ever re-running the whole set to confirm it. Single-benchmark re-runs update that same scoreboard in place, so you never lose the picture of a completed pass while you work through the stragglers.

We deliberately simulate the conditions that make scanners lie. A lot of the suite is not about the happy path at all. It is about what happens when something upstream degrades, and whether OSSScan stays honest when it does. A few examples, all deterministic and network-free because they stub the failure in on purpose:

• A metadata lookup that flakes. When a version-pinned registry lookup misses and the only thing left to match is a bare package name, OSSScan must not resolve it to some unrelated stranger's license that happens to share the name. The test asserts the answer fails closed to Unknown, because a confidently wrong license is worse than an honest gap.
• Copyleft hidden inside a permissive package. We build a package that declares MIT and vendor real GPL license text inside its source tree, in the hardest case with no manifest of its own, so nothing that only reads metadata can catch it and only a text scan can. Audit mode must raise the disagreement, that the tree contains copyleft the declaration never mentioned, rather than trusting the MIT label at the top.
• A dependency with no source on disk. Some dependencies live in a shared cache off to the side rather than inside your tree, and a naive scan can point the license reader at whatever code sits nearest instead, then stamp that neighbor's license, copyleft included, onto dozens of unrelated packages. The test forces this layout and asserts OSSScan fails closed to Unknown for those packages rather than borrowing a license that was never theirs.
• A speed optimization that could hide a finding. OSSScan can prune parts of a scan it can prove are noise, to cut how much you have to investigate. The test scans the same project both with and without pruning and asserts that no package carrying a real license, a copyleft one above all, ever disappears or downgrades to Unknown as a side effect. An optimization is not allowed to lose a finding.

The thread through all of these is the same principle the license work follows: when conditions are bad, OSSScan should fail closed to a cautious, honest answer, and we write tests that force those bad conditions to prove it does.

The point

None of this guarantees perfection. What it does is make it hard for OSSScan to become quietly less accurate without something turning red first. More than 40 real projects across a dozen ecosystems, checked at every layer from a single function up to the full app, run the same way through both the CLI and the UI, cross-referenced against independent tools, and compared across two operating systems.

When the scanner tells you a package is copyleft, that a license needs a human, or that your coverage is only partial, this is the machinery standing behind that answer.

← Back to the OSSScan Blog