How to Evaluate a QA Provider

Who this guide is for

This guide is for leaders about to brief, shortlist or sign with a QA company. It follows the order of a real evaluation, from writing your own requirements to a scoring sheet you can copy.

Before you talk to anyone

Write down four things before the first call: the scope (which applications, platforms and journeys), the release cadence, the environments the provider will test against, and what "done" means for a release. A staging environment, a copy of the application used for testing before release, is one of the things providers ask for; one company lists access to code, environments and CI/CD plus one technical owner on the client side[1].

With this page in hand, every provider answers the same brief and their proposals can be compared line by line.

People

Ask for the names and seniority of the people who will work on your product, the hours they overlap with your team, and the terms for replacing someone. Providers describe their staffing in different units: a dedicated lead with daily stand-ups and shared dashboards[2], or a dedicated QA lead and a small team joining your sprints[3]. Verify that the people named in the proposal are the people who start.

Process

Ask how tests are designed, who reviews them, how often you receive reports and what a defect report contains. Ask for a sample report before signing. One company states it typically provides a daily progress report[4]. As an example of the format questions to ask, our own sample report shows run metadata, expected and actual outcomes and the items needing a decision; it is sample data from Testivra, labelled as such.

Change handling

Your application will change during the engagement. Ask what happens to the suite when a page changes, how quickly broken tests are fixed, and whether maintenance is included in the price. In one managed model the provider updates test cases, frameworks and environments as the product changes[5]. Ask how a regression suite, the set of checks re-run after each change, stays current, and who pays for the work.

Agreement

Read the agreement for minimum commitments, the notice period, ownership of tests and frameworks, and data access. Some providers publish their terms: one states monthly billing[6] and no annual commitment with a notice period to leave[7]. Another states that the scripts and artefacts it creates belong to the client[8]. If ownership is not written down, ask for it in writing.

Verification

Check the provider's own site for the services and engagement models it states, then confirm company facts such as founding year and location in a second source, such as press coverage or a public registry. Review platforms are useful for finding reference clients; read the reviews themselves rather than the scores. Case studies count only when they are on the provider's own domain and name a product you can look up.

Red flags

• No named people before signing. The team you meet in sales may not be the team that starts.
• No sample report. Without one you cannot judge what you will receive each week.
• Ownership of tests left unstated. You may lose the suite if you leave.
• Maintenance priced as an extra with no estimate. Upkeep is a recurring cost of any suite.
• Company facts that no second source confirms. Basic facts should be checkable.
• Pass rates offered as proof of quality. A high pass rate can mean the tests check too little.

A scoring sheet you can copy

Copy these rows into a spreadsheet, set your own weights and score each provider from its written proposal.

Questions to ask

1. Who exactly will work on our product, and what is their seniority?
2. Which hours will your team overlap with ours?
3. How do you replace someone who leaves, and how fast?
4. Can we see a sample report and a sample defect report?
5. How often will we receive reports, and in which tools?
6. Who writes test cases, and who reviews them before they run?
7. What happens to the suite when a page in our application changes?
8. Is maintenance included in the price, and how is it estimated?
9. Who owns the tests and frameworks you create, and in what format do we receive them on exit?
10. What are the minimum commitment and the notice period?
11. Which environments and data will you need access to, and how is that access controlled?
12. Which public sources confirm your founding year and location?