Vulnerability management can be purchased as software that the organisation runs itself, as a managed service delivered by a provider, or as a combination of the two. Each option promises visibility and control, and most demonstrations look convincing. The evaluation work lies in testing whether a particular option will fit your estate, your people and your processes once the demonstration is over. This article sets out the criteria that matter and a practical way to test them before committing budget.
Start with your requirements, not the product
Before speaking to any provider, write down what the programme must do. List the asset types in scope, including cloud accounts, containers, endpoints, network devices and applications. Record the systems that cannot tolerate intrusive scanning. Identify the ticketing, reporting and identity tools the programme must integrate with. Note the frameworks or contractual obligations that shape the evidence you need.
A requirements document of this kind lets you judge options on their merits. Without it, evaluation tends to drift towards the features shown in the most polished demonstration.
Coverage and method
Ask how each option discovers assets, how it scans each type of asset and whether it uses authenticated checks where those are needed. Ask what it cannot see. A provider that is candid about limitations is usually easier to work with than one that claims universal coverage. Request a description of the scanning methods in writing, and check that each method is appropriate for the systems on your list.
Validation and accuracy
Validation determines how much manual effort the programme will consume. Ask how the option distinguishes real findings from false positives, how it handles duplicates and whether it can explain why a finding applies to a particular system. Find out whether validation is automated, performed by people, or a combination, and what the process is for disputing a result.
Prioritisation logic
Ask how priority is calculated and whether the logic can be adjusted to reflect your risk appetite and business context. A prioritisation model that cannot be changed will eventually conflict with the way your organisation works. Check whether asset attributes such as owner, data sensitivity and internet exposure can be used as inputs, and whether the outputs can be exported for review.
Workflow fit
Many programmes fail at the handoff between detection and remediation. Evaluate how findings reach owners, whether they can be assigned automatically, how tickets are created and synchronised, how exceptions are recorded and how verification is performed. Ask the provider to demonstrate the workflow using a realistic sample, not only the reporting dashboard.
Data handling and security of the service
Vulnerability data describes weaknesses in your systems, so it is sensitive. Ask where scan data is stored, which jurisdictions it may be processed in, who can access it, how credentials are protected and how long records are retained. Ask about the provider's own security controls, the independence of any assurance it holds and how it notifies customers of incidents. Where the provider relies on sub-processors or third-party infrastructure, ask for a current list and for confirmation of their roles.
Verify any certification or assurance statement directly rather than accepting it from a sales presentation. Certifications apply to specific scopes, and the scope may not include the service you are buying.
Contract and exit terms
Read the terms with the same care as the technical proposal. Clarify the service levels, the reporting obligations, the rights to the data at the end of the contract, the process for data return or deletion, and the responsibilities for scanning credentials and changes to scope. Pricing models vary, and it is worth checking how scope changes, new asset types and additional users are charged.
Proof of value
A short, bounded proof of value is more informative than a demonstration. Agree success criteria before it starts, such as the accuracy of validated findings on a sample of assets, the time taken to route findings to owners and the ability to integrate with existing tools. Use a sample that includes at least one difficult system, such as an older platform or a sensitive production service, so that the trial reveals real constraints.
Record results against the criteria and review them with the owners who would use the service. Their assessment of usability and workability is as important as the technical results.
Testing vendor claims against your own evidence
Demonstrations are designed to show a product at its best, and written materials are designed to persuade. Neither reflects the messy reality of an estate with legacy systems, unusual configurations and teams short of time. Before accepting a claim, ask what evidence supports it, how that evidence was produced and whether it applies to an environment like yours. A claim about coverage, for example, needs a defined list of asset types, the methods used to reach them and examples of systems the provider could not assess.
References from other customers can help, but they should be treated with care. Ask whether the reference reflects a similar size, sector and technical mix, and what the reference would do differently if starting again. Avoid relying on a single positive account. A short conversation with two or three people who run the service day to day is usually more revealing than a polished case description.
Check whether the claims can be verified after purchase. Ask how coverage will be reported, how accuracy will be sampled and how the organisation will know if the service stops performing as described. A well-designed agreement includes these checks from the outset, so that the evaluation does not end when the contract is signed.
The evaluation team should record each claim, the evidence offered and the outcome of the test. That record is useful when the decision is reviewed later, and it lets the next evaluation start from a more informed position.
Common mistakes
- Choosing the tool before defining requirements.
- Accepting a demonstration in place of a bounded trial using your own assets.
- Overlooking data handling and exit terms in the contract.
- Treating vendor certification as proof of the specific service being bought.
- Ignoring the people who will run the process day to day.
Next steps
Prepare a one-page requirements summary and a short list of criteria with weightings. Send the same questions to each shortlisted option, run a proof of value against an agreed sample and compare results against the criteria rather than against each other's presentations.
Frequently asked questions
Should we buy software or a managed service?
The answer depends on your internal capacity, the complexity of the estate and how much control you want to retain. Many organisations choose a blend, keeping ownership and prioritisation in-house while outsourcing scanning or triage.
How long should a proof of value run?
Long enough to test the difficult cases and the workflow with real owners. The duration should be set against the agreed success criteria rather than a fixed period.
What should we ask a provider about data location?
Ask where data is stored and processed, which sub-processors are involved, who can access it and what the arrangements are for deletion at the end of the contract.
Related reading
If you are working through this topic, these related articles cover neighbouring questions:
