Most tool trials end with a vague verdict: "it looked good" or "the team did not love it." A 30-day pilot should end with a number-backed decision to buy, extend or stop. This guide shows how to scope the trial, agree the criteria in advance, track a few measures every week, and score the result.
The short answer
Pick two or three repositories, write down what success means before the tool is connected, measure the same few things each week, and decide on day 30 against the criteria you set. Treat the pilot as an experiment on your own code, not a demo on someone else's.
Before you start
Spend the first days on setup decisions. They matter more than the tool.
- Choose the repositories. Take two or three that your team actively works on. Include one with a known history of findings and one with a healthy test suite. Avoid toy projects and avoid your most fragile system.
- Name an owner. One engineer is responsible for the pilot and for the weekly numbers.
- Name a decision maker. Someone with budget authority agrees to read the scorecard on day 30.
- Record a baseline. Count open findings by severity, and note how long the last ten fixes took from first look to merge. Without a baseline you cannot show change.
- Agree the data terms. Know what code the tool reads, where it is processed, and what is retained. Settle this with security and legal before connecting anything.
Write the success criteria first
Agree the thresholds before the trial, so the result cannot be argued afterwards. Examples to adapt, not targets to copy:
| Question | Example criterion |
|---|---|
| Does it find real problems? | Most reviewed findings are confirmed as real by your engineers |
| Is the noise manageable? | Engineers spend a small, agreed amount of time per week triaging |
| Are the fixes usable? | A meaningful share of proposed fixes are merged, with edits allowed |
| Is it safe? | No fix is merged without review, and no test regression reaches the main branch |
| Does it fit the workflow? | Findings and fixes arrive as pull requests in the tools the team already uses |
| Does it save time? | Hours per fix fall against your baseline |
Choose the thresholds yourself from the baseline. A number copied from a vendor deck does not describe your code.
The four weeks
Week 1: connect and baseline. Connect the repositories, run the first scan and review a sample of findings by hand. Mark each one real, not real, or unclear. This gives you a precision estimate from your own code.
Week 2: fixes in review. Let the tool propose fixes. Have engineers review them as they would any pull request. Record which are merged as is, merged after edits, or rejected, and why.
Week 3: normal work. Remove special attention. Developers handle findings in their usual flow. Watch whether the tool is used when nobody is watching, and how much time it takes.
Week 4: measure and decide. Collect the numbers, fill in the scorecard, and hold a short decision meeting.
What to measure each week
Keep it to a short list that one person can update in ten minutes.
- Findings reviewed, and how many were real.
- Fix pull requests opened, merged, edited and rejected.
- Median time from finding to merged fix, against the baseline.
- Engineer time spent on triage and review.
- Test failures or regressions caused by a proposed fix.
- Developer comments, in their own words.
Count merged fixes rather than findings. A tool that reports a thousand issues has made work. A tool that gets fixes merged has removed it.
A scorecard you can copy
Score each row from 0 to 2: 0 means it missed the criterion, 1 means partly met, 2 means met. Write the evidence next to each score.
| Area | Score (0 to 2) | Evidence |
|---|---|---|
| Finding quality | Share of reviewed findings confirmed real | |
| Noise and triage time | Weekly hours against the agreed limit | |
| Fix quality | Share merged, edited, rejected | |
| Safety | Review gate held; regressions found | |
| Workflow fit | Where findings and fixes appeared | |
| Time saved | Hours per fix against the baseline | |
| Data handling | Terms agreed with security and legal | |
| Support and communication | Response to questions during the trial |
Do not average blindly. Decide in advance which areas are must-pass. A tool that fails on safety or data handling should not proceed, whatever its total.
Deciding on day 30
Three outcomes are reasonable:
- Buy. Must-pass areas met, and the time saved or risk removed justifies the cost. Move to a contract with the pilot repositories as the first rollout.
- Extend. The results are promising but one area is unclear. Extend for a fixed period on a new repository, with the open question written down.
- Stop. Must-pass areas failed or the benefit is too small. Write down why, so the next evaluation starts further ahead.
Whichever you choose, share the scorecard with the people who took part.
Mistakes to avoid
- No baseline. Without it, any result sounds good.
- Changing criteria midway. If the goal moves, the pilot proves nothing.
- Judging on the demo repository. The vendor's sample project is tuned to look good. Use your code.
- Counting findings as value. More findings are not better. Fixed findings are.
- Skipping engineer review. Merging fixes without review turns a trial into a risk.
- Letting it drift. Set a hard end date and a decision meeting on the calendar from day one.
Frequently asked questions
How many repositories should we use? Two or three is enough to learn, and few enough to review properly.
How many engineers should be involved? The owner plus the engineers who normally work in those repositories. More people add opinions, not information.
Can we run two tools at once? Yes, on different repositories or in sequence, with the same scorecard. Do not run them on the same pull requests, or reviewers will not know which tool to credit.
What if the tool finds very little? That can be a good result for a mature codebase. Check against the baseline, and consider seeding a known issue in a test branch to see whether it is caught.
What should we ask the vendor to provide? A clear description of what code is read and retained, support during the trial, and help exporting the results.
For how fix quality can be judged, see the security remediation gap, and for presenting the cost of an unfixed backlog, see the business case for fixing your vulnerability backlog.
This guide is general guidance. The criteria in the examples are illustrations, not benchmarks.