|2018 ASM / R-0041||
|A six-year Australian and New Zealand experience of artificial intelligence techniques applied to MRI liver data – tricks and traps|
Other things to consider when building a neural network:
This paper reports a six-year experience applying machine learning models to small and large image datasets. The key elements that have resulted in an acceptable model are the growth in computer processing power, access to advanced machine learning algorithm libraries, and access to large volumes of consistently acquired and well labelled data.
Although the majority of the analysis in this paper has focussed on the application of computer algorithms to automate the analysis of MR images of the liver for the quantification of iron and fibrosis and the insights and learnings achieved over this period, there must also be consideration given to the successful deployment of the CNN as a regulated medical device. When seeking regulatory clearance of the CNN (Software as a Medical Device, SaMD), the requirements for security, traceability, reproducibility, and performance all need to be addressed, and it is more efficient to consider these from the outset.
In the development stage, the emphasis is on performance (training the network to improve accuracy and reproducibility). Once a suitable performance is achieved, in order to meet regulatory requirements (FDA, CE Mark, TGA) for reproducibility, the algorithm must no longer be learning. In our example, which has now achieved regulatory clearance in several jurisdictions, the final model had been trained on a sufficient number of datasets and had reached a saturation point, where diagnostic performance did not significantly improve with additional training data. The software was then able to be pushed to production, with a defined diagnostic performance.
Whilst there may be examples where CNNs have not been trained on sufficient datasets to reach maximum accuracy and continued learning may result in increased/changed performance, from a regulatory perspective these models will not meet the current fundamental requirement for reproducibility. In effect, it is a particular output that receives regulatory clearance. If a device attains regulatory authority clearance and then the performance improves due to additional training of a test system, this improvement is not cleared without additional validation work, and potentially a new submission to regulatory authorities.
Regulatory clearance for SaMD also requires absolute security and traceability of data. All data need to be secure at all times (in transit and at rest) for HIPAA compliance, and the ability to trace data access and usage is essential. Regulators assess these factors very closely to ensure the entire device meets these requirements. With our CNN, we allowed for several ways for the user to access the algorithm (web-portal and third-party platforms). Each of these access methods has had best endeavours consideration given to meeting the full requirements of regulators.
The last six years have seen an enormous growth in the use of computer programs, collated under the broad term “Artificial Intelligence” across all industries. This paper tracks those changes in a local scenario culminating in the production of a regulatory cleared Medical Device that uses artificial intelligence to quantify liver iron concentration from MRI.
There are valid fears that have been expressed about the impact Artificial Intelligence will have on jobs as part of the fourth industrial revolution. However, there are many solutions to many diagnostic problems that will need to come together before radiologists are replaced by machines. Current commercial offerings, like the solution described above, solve one problem only.
We prefer to refer to AI as Augmented Intelligence, creating tools that will assist and enhance radiologists in the performance of their duties. Augmented Intelligence will reduce the mundane and the routine, improving product and service by increasing diagnostic accuracy, reducing error, and increasing throughput.
When used as decision making support tools for increased efficiency and accuracy, AI will allow capacity for reproducible and rapid decision making to radiologists globally. The liver iron measurement tool described above is now available in nine developing countries that struggle with access to affordable healthcare, either due to cost or lack of availability of appropriately trained medical staff. AI may be able to bring solutions at low cost to many countries that currently have insufficient expertise and will provide increased efficiencies and support for radiologists in general.
The development of these tools requires large bodies of accurately labelled data on which to learn. Australian and New Zealand radiology is well placed to participate in building these solutions. We encourage you to find partners and get to work!
Thematically related posters
2018 ASM / R-0114
A novel risk-based system for the safe and rapid implementation of new technology in radiation oncology: Risk and Benefit Balance Impact Templates (RABBITs)