Validate imaging AI against radiologists, on real cases
Radiology AI validation on an independent platform: see how an AI tool performs alone, how radiologists perform alone, and how they perform together.
Trusted by leading healthcare institutions
What radiology AI validation means (standalone and reader study)
Radiology AI validation measures how an imaging AI tool performs on real cases. A standalone validation scores the AI on its own against a reference standard. A reader study compares radiologists reading with and without the AI. You can run either one locally, or as an external validation to support a regulatory submission.
A vendor report shows how an AI tool performed on the vendor's data. A reader study shows how it performs on your data and your population: the same cases, read with and without the AI, scored on one form. Collective Minds runs that study in one workflow, so your time goes into the design and not into moving files.
How it works
- Define the question and the reference standard. Decide what the AI is tested on and how ground truth is set.
- Upload the cases. Send the exams through Connect or upload them on the web.
- Configure the read. Build the form and, if the design needs it, the reading phases.
- Run the AI and the readers. Attach results from one or more AI engines. Each reader works from a personal task list, with and without the AI.
- Analyze. See where the AI helped and where it did not.
- Export the results. Per examination, per image and per finding.
What changes
- Independent ground truth. Expert readers set the reference standard before the AI is scored, so results hold up.
- AI and reader arms on one platform. One form, every reader. See where the AI helped and where it did not.
- A record of the study. Who read which case, and when, with results you can export for review.
Collective Minds is the independent platform the AI is tested on. We do not sell the AI tools being validated.
Projects like this
Regulatory reader study for an AI company
An AI company needed a compliant, scalable platform for a reader and annotation study to support a regulatory submission.
What we did: a cloud workflow for central review, annotation and consensus of more than 300 exams, with more than 20 project members.
Regulated reader study for a MedTech company
A MedTech company needed a cloud solution for a regulated multicenter study with central review.
What we did: secure data transfer and de-identification, quality control, reading events for the classifications. Blinded review by two readers and one adjudicator.
Fracture detection in children with a rare bone disease
A children's hospital and a university tested a commercial AI tool.
What we did: 48 patients, 336 images, 7 readers, read with and without the AI.
Result: In the Press Release button.
Two AI engines, one dataset
A university hospital wanted to compare two commercial fracture-detection engines.
What we did: 1,006 radiographs, read by radiologists and radiologist trainees, so both engines were judged on identical cases.
A reader study for an AI vendor
A vendor needed reader evidence in Europe, and wanted to see if inexperienced readers with AI assistance performed at the same level as the experts.
What we did: 340 X-ray subjects, 15 readers in three geographies, three experts creating the ground truth. All image evaluation was done on the Collective Minds platform using a clinical-grade DICOM viewer and custom evaluation forms.
Reference reads for a stroke AI company
A stroke AI company needed expert reference reads on CT angiography and non-contrast CT.
What we did: neuroradiologists read each case twice, and a third reader settled disagreements.




Frequently asked questions
Everything you need to know about validating radiology AI.
What is radiology AI validation?
It measures how an imaging AI performs on real cases, standalone against a reference standard and in a reader study with radiologists reading with and without the AI.
What is the difference between standalone and reader-study validation?
Standalone validation measures the AI on its own against ground truth. A reader study measures whether radiologists read better, faster or more consistently with the AI.
Can you run validation for a regulatory submission?
Yes. Studies can be designed and run on the platform to support regulatory submissions, with a locked design, defined ground truth and an audit trail.
How is ground truth established?
By a defined reference standard, for example expert consensus, follow-up or pathology, set before reading so results are defensible.
Do our own radiologists or external readers take part?
Either. You can validate locally with your own readers or bring in external readers from the platform's network for independent validation.
What is a reader study for AI validation?
Radiologists read the same cases with and without an AI tool, and the results are compared with a reference standard. It shows the accuracy of the AI alone, the readers alone and both together.
Can we compare more than one AI engine?
Yes. Attach results from several engines to the same cases and compare them on identical data. One university hospital did this with two commercial engines.
How are patient details protected?
Exams are de-identified before they are used in the study, and each project is open only to invited people. Your data protection team should confirm the setup for your study.
Who does what?
Collective Minds provides the platform and the workflow. Who recruits the readers, sets the ground truth and runs the analysis depends on the study. We agree it with you at the start.
Planning an AI validation?