A. Explanation of AI Model Development

Our AI model is developed using a rigorous and systematic process to ensure reliability and clinical relevance.

• Data Collection and Preprocessing: We aggregate a diverse dataset of anonymized canine and feline X-rays from various sources to capture a wide range of anatomical variations and pathological conditions. Data is de-identified in compliance with privacy regulations.
• Annotation and Labeling: Experienced veterinary radiologists annotate the images, providing high-quality labels that serve as ground truth for training. This ensures that the model learns from expert interpretations.
• Model Architecture: We employ state-of-the-art deep learning architectures optimized for image analysis, such as convolutional neural networks (CNNs), to effectively process and interpret radiographic images.
• Training Process: The model is trained using supervised learning techniques with hyperparameter tuning to optimize performance metrics like accuracy, sensitivity, and specificity.
• Bias Mitigation: We actively identify and mitigate potential biases by ensuring balanced representation across species, breeds, ages, and pathological conditions in the training dataset.

B. Explanation of AI Model Pre-release Assurance of Safety and Effectiveness

Before releasing the AI tool, we conduct comprehensive evaluations to ensure it meets safety and efficacy standards.

• Validation Studies: The model undergoes rigorous validation using separate datasets not seen during training to assess its generalizability and robustness.
• Performance Metrics: We evaluate key performance indicators, including accuracy, sensitivity, specificity, and area under the ROC curve (AUC), to quantify the model’s diagnostic capabilities.
• Clinical Testing: Pilot studies are conducted in clinical settings where veterinary professionals use the tool in real-world scenarios to provide feedback on usability and effectiveness.
• Regulatory Compliance: We ensure that our AI tool complies with FDA guidelines for Good Machine Learning Practices.
• Risk Assessment: A thorough risk analysis is performed to identify potential failure modes, and appropriate safeguards are implemented to mitigate any identified risks.

C. Explanation of AI Production Model Deployment and Ongoing Monitoring

Upon deployment, we maintain vigilant oversight of the AI tool’s performance and impact.

• Integration with Existing Systems: The AI tool is seamlessly integrated into veterinary workflows, ensuring minimal disruption and maximal utility.
• User Training and Support: We provide comprehensive training materials and support services to help veterinary professionals effectively utilize the tool.
• Performance Monitoring: Continuous monitoring systems are in place to track the model’s performance in real-time, detecting any deviations from expected outcomes.
• Feedback Mechanisms: We encourage and facilitate user feedback to identify any issues or areas for improvement, fostering a collaborative environment for refinement.
• Data Security: Robust security protocols protect sensitive data during transmission and storage, complying with industry standards for data protection.
  • Our Privacy Policy

D. Explanation of AI Device Ongoing Re-training, Modifications, and Versioning

We recognize the importance of keeping the AI model current with evolving medical knowledge and practices.

• Periodic Re-training: The model is periodically retrained with new data to incorporate the latest findings and address any drift in performance.
• Change Management: Any modifications to the model undergo a structured change management process, including re-validation and documentation.
• Version Control: We maintain strict version control, documenting changes between versions and ensuring traceability of updates.
• Regulatory Updates: Updates to the model are evaluated for regulatory impact, and we ensure continued compliance with applicable regulations.
• Transparency: Users are informed of significant updates or changes to the AI tool, including improvements in performance or functionality.

Conclusion

Vetology is dedicated to providing safe, effective, and reliable AI solutions for veterinary radiology. Through adherence to Good Machine Learning Practices, we strive to enhance diagnostic capabilities while maintaining the highest standards of quality and ethics.

AI For Human And Veterinary Radiology

Radiology Artificial Intelligence, commonly referred to as AI, is in full development, and the FDA is actively testing AI in human radiology. In veterinary radiology we’re not far behind and are quickly catching up. Veterinary medicine AI product development’s greatest strength is also its greatest weakness. That is, oversight, or more specifically, the lack of oversight by a governing body.

In human radiology, AI products require FDA oversight and approval prior to coming to market (see the American Association of Veterinary Radiologists GMLP write-up here to learn more. Some veterinary products do fall under FDA oversight, BUT veterinary radiology AI isn’t one of them. This means a veterinary radiology AI company can develop quickly but also have no formal obligation to demonstrate their product actually works.

If a company doesn’t provide clinical test results, it is up to you, the veterinarian, to determine the worthiness of the product. A true ‘caveat emptor.’

Assessment Is Critical

So how should a veterinarian assess a veterinary radiology AI product? The same way the entire medical community evaluates any diagnostic test for a condition, by measuring clinical performance.

The two standard measures of clinical performance are SENSITIVITY and SPECIFICITY.

To better understand how these measures can assist us to asses clinical performance, we will briefly revisit a couple of formulas from that old favorite – statistics class.

SPECIFICITY is the probability a test will correctly identify a patient who DOES NOT have a condition (true negative). It is calculated by the following formula:

The Confusion Matrix

These two standard measures of clinical performance lead us to the four outcomes possible for each patient:

True Positive  |  False Positive  |  True Negative  |  False Negative

The four outcomes are typically reported in a 2 x 2 table called a confusion matrix, showing the total numbers of true and false positives and negatives. A generic example is shown here next.

Vetology’s AI Testing

Vetology’s AI testing evaluates AI results against veterinary radiologist reports as a reference standard. The results and confusion matrix tables are displayed just below on this page.

The Truth Is In The Confusion Matrix

To help you evaluate a product’s performance, always ask an AI vendor for their confusion matrix tables.

Also, be aware that an AI product MUST be 100% autonomous to have a valid result. If a human intervenes during any part of the result creation, it’s not artificial intelligence, it’s human intelligence.

Next we show the confusion matrix for several diseases among 75 random cases:

Buyers Beware

As we said earlier in this article, AI brings to the forefront of your purchase decision-making, the phrase ‘caveat emptor,’ or buyer beware. Make sure you review the provider’s confusion matrix tables (if they have them at all), and make sure their AI is fully autonomous, else you’ll just be buying expensive human intelligence.

At Vetology, we assertively and proactively test ourselves and continually train and improve the AI for everyone’s benefit. We are as transparent as possible. We have the data and are willing to publish it.

If you have any questions about our veterinary radiology software or services, we encourage you to reach out to us via email.