How Intelligent Is Artificial Intelligence?
AI For Human And Veterinary Radiology
Radiology Artificial Intelligence, commonly referred to as AI, is in full development, and the FDA is actively testing AI in human radiology. In veterinary radiology we’re not far behind and are quickly catching up. Veterinary medicine AI product development’s greatest strength is also its greatest weakness. That is, oversight, or more specifically, the lack of oversight by a governing body.
In human radiology, AI products require FDA oversight and approval prior to coming to market (see the American Association of Veterinary Radiologists GMLP write-up here to learn more. Some veterinary products do fall under FDA oversight, BUT veterinary radiology AI isn’t one of them. This means a veterinary radiology AI company can develop quickly, but also has no formal obligation to demonstrate that its product actually works.
If a company doesn’t provide clinical test results, it is up to you, the veterinarian, to determine the worthiness of the product. A true ‘caveat emptor.’
Assessment Is Critical
So how should a veterinarian assess a veterinary radiology AI product? The same way the entire medical community evaluates any diagnostic test for a condition, by measuring clinical performance.
The two standard measures of clinical performance are SENSITIVITY and SPECIFICITY.
To better understand how these measures can assist us to asses clinical performance, we will briefly revisit a couple of formulas from that old favorite – statistics class.
Sensitivity And Specificity
SENSITIVITY is the probability that a test will identify a patient who HAS a condition (true positive). It is calculated by the following formula:
SPECIFICITY is the probability a test will correctly identify a patient who DOES NOT have a condition (true negative). It is calculated by the following formula:
The Confusion Matrix
These two standard measures of clinical performance lead us to the four outcomes possible for each patient:
True Positive | False Positive | True Negative | False Negative
The four outcomes are typically reported in a 2 x 2 table called a confusion matrix, showing the total numbers of true and false positives and negatives. A generic example is shown here next.
A Typical Confusion Matrix
Name of Condition or Disease
Total Number of Cases Measured
% Sensitivity
% Specificity
| Radiologist Positive | Radiologist Negative | |
| AI Positive | # of cases | # of cases |
| AI Negative | # of cases | # of cases |
Vetology’s AI Testing
Vetology’s AI testing evaluates AI results against veterinary radiologist reports as a reference standard. The results and confusion matrix tables are displayed just below on this page.
The Truth Is In The Confusion Matrix
To help you evaluate a product’s performance, always ask an AI vendor for their confusion matrix tables.
Also, be aware that an AI product MUST be 100% autonomous to have a valid result. If a human intervenes during any part of the result creation, it’s not artificial intelligence, it’s human intelligence.
Next we show the confusion matrix for several diseases among 75 random cases:
Cardiomegaly
47 Cases
90% Sensitivity
76% Specificity
| Radiologist Positive | Radiologist Negative | |
| AI Positive | 17 | 19 |
| AI Negative | 3 | 8 |
Heart Failure
39 Cases
100% Sensitivity
89% Specificity
| Radiologist Positive | Radiologist Negative | |
| AI Positive | 3 | 4 |
| AI Negative | 0 | 32 |
Dynamic Airway Pattern
30 Cases
100% Sensitivity
85% Specificity
| Radiologist Positive | Radiologist Negative | |
| AI Positive | 3 | 4 |
| AI Negative | 0 | 23 |
Dynamic Airway Collapse
30 Cases
67% Sensitivity
93% Specificity
| Radiologist Positive | Radiologist Negative | |
| AI Positive | 2 | 2 |
| AI Negative | 1 | 25 |
Buyers Beware
As we said earlier in this article, AI brings to the forefront of your purchase decision-making, the phrase ‘caveat emptor,’ or buyer beware. Make sure you review the provider’s confusion matrix tables (if they have them at all), and make sure their AI is fully autonomous, else you’ll just be buying expensive human intelligence.
At Vetology, we assertively and proactively test ourselves and continually train and improve the AI for everyone’s benefit. We are as transparent as possible. We have the data and are willing to publish it.
If you have any questions about our veterinary radiology software or services, we encourage you to contact us..
