Reviews on Deep Learning in Healthcare and Medical Imaging Analysis

This is a blog post review on publications about Deep Learning in healthcare and medical imaging.

Credits to Weina Jin

General deep learning

1. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539

   As a review paper, this is a high-level overview of the recent advances in deep learning. It introduced the idea of multilayer neural networks, backpropagation, convolutional neural networks, recurrent neural networks, and the advances they brought to computer vision and language processing. The key advantage of deep learning is that it avoids the number of engineering skills and domain expertise when extracting good features. The features are “learned automatically using a general-purpose learning procedure”. It also summarized 4 key ideas behind ConvNets: local connections, shared weights, pooling, and the use of many layers. I gained a new understanding of the pooling part: its role is to “merge semantically similar features into one, because the relative positions of the features forming a motif can vary somewhat, reliably detecting the motif can be done by coarse-graining the position of each feature”. Pooling also creates “an invariance to small shifts and distortions”.

   Given its publication year of 2015, for someone who already familiar with deep learning, this paper did not add on too much additional information. However, since his paper is written by LeCun, Y., Bengio, Y., & Hinton, G, and has received 8k+ citations (published in Nature), it is worth taking a look, especially it introduced the changes of research from a historical perspective. For those who have little machine learning background, this paper is quite easy to read and do not hinder the reader with profound mathematical formulas.

Machine learning in healthcare

1. Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J. T. (2017). Deep learning for healthcare: review, opportunities and challenges. Briefings in Bioinformatics. https://doi.org/10.1093/bib/bbx044

   This paper highlights the recent progress of deep learning in the application filed of clinical imaging, electronic health records, genomics, and sensory data. Noticeably, it listed challenges and opportunities within the domain: data integration(integrating heterogeneous data source, and also incorporating expert knowledge), interpretability, security, and temporal modeling. It also pointed out that deep learning can serve as a “guiding principle to organize both hypothesis-driven research and exploratory investigation(eg: clustering, visualization of patient cohorts, stratification of disease populations).

2. Ravì, D., Wong, C., Deligianni, F., Berthelot, M., Andreu-Perez, J., Lo, B., & Yang, G. Z. (2017). Deep Learning for Health Informatics. IEEE Journal of Biomedical and Health Informatics, 21(1), 4–21. https://doi.org/10.1109/JBHI.2016.2636665

   This is a more technique-focused review paper published in IEEE. The graphs well-demonstrated the distribution of papers in the subfields of medical imaging, medical informatics, bioinformatics, sensing, and public health. It also gave percentage most used deep learning methods in health informatics: with recurrent neural network, ConvNets, and deep neural network leading the majority, followed by deep belief network, deep boltzmann machine, and deep autoencoder. It addressed similar challenges include:

   1. model not interpretable
   2. disease-specific data is often limited
   3. needs preprocessing; hyperparameter searching is a blind exploration
   4. machine learning models are susceptible to be manipulated

3. Obermeyer, Z., & Emanuel, E. J. (2016). Predicting the Future — Big Data, Machine Learning, and Clinical Medicine. New England Journal of Medicine, 375(13), 1216–1219. https://doi.org/10.1056/NEJMp1606181

   Written by two MDs, the NEJM perspective article introduced the machine learning statistical method to doctors at a high level. What is machine learning? What can machine learning do? What kind of data does machine learning algorithm need? And what is the limitation of machine learning? With the ability to transform data into knowledge, what area in medicine will be disrupted by machine learning? The authors pointed out three directions: 1) prognosis estimation, 2) radiology and anatomical pathology; and 3) the improvement in diagnostic accuracy.

   I personally really like this article. Let me quote the last paragraph:

   Clinical medicine has always required doctors to handle enormous amounts of data, from macro-level physiology and behavior to laboratory and imaging studies and, increasingly, “omic” data. The ability to manage this complexity has always set good doctors apart from the rest. Machine learning will become an indispensable tool for clinicians seeking to truly understand their patients. As patients’ conditions and medical technologies become more complex, the role of machine learning will grow, and clinical medicine will be challenged to grow with it. As in other industries, this challenge will create winners and losers in medicine. But we are optimistic that patients, whose lives and medical histories shape the algorithms, will emerge as the biggest winners as machine learning transforms clinical medicine.

4. Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., … Wang, Y. (2017). Artificial intelligence in healthcare: past, present and future. Stroke and Vascular Neurology, svn-2017-000101. https://doi.org/10.1136/svn-2017-000101

   It is another medical-focused review written by neurologists. The highlight in it is that it listed researches using machine learning for the detection and treatment of neurological diseases, mainly for stroke. However, some of the statements in this paper were a bit arbitrary, for example, the achievement of some ML system it referred was from news report rather than research papers which underwent certain experiment and evaluation. This is a way reflects that clinicians tended to hold a believe-or-not attitude. The lack of critical thinking for some media news may root from that the doctors know little about the underlying technologies of deep learning.

Machine learning in medical imaging analysis

1. Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., … Sánchez, C. I. (2017). A Survey on Deep Learning in Medical Image Analysis. Medical Image Analysis, 42, 60–88. https://doi.org/10.1016/j.media.2017.07.005

   The medical imaging analysis community has transited from systems that use handcrafted features to systems that learn features automatically from the data, the so-called “end-to-end” learning. This is a very comprehensive paper that can be regarded as a roadmap to deep learning in medical image analysis. It summarized the recent research progress of using deep learning for medical image analysis, both from an application and a methodology-driven perspective.

   It listed a variety of model architectures for canonical tasks in medical image analysis: classification, detection, segmentation, registration, retrieval, image generation and enhancement. Here I summarized the main model architectures as follows:

   1. Classification and detection

   Transfer learning with CNN is the most popular architecture for image classification. For object or lesion classification/detection that focuses on the local lesion, multi-stream CNN, fCNNs, REMs, SAEs, convolutional sparse auto-encoders, and multiple instance learning (MIL) were used in previous research.

   1. Segmentation

   Segmentation is the most common subject of papers applying deep learning to medical imaging. Therefore it has the widest variety in methodology. U-net is the most well-known novel CNN architecture. It combines skip connections between opposing convolution and deconvolution layers. It can generate a segmentation map with one forward pass of the entire images. And this allows U-net to take into account the full context of the images, as opposite to patch-based CNNs. V-net is a 3D-variant of U-net. It performs 3D image segmentation using 3D convolutional layers with an objective function directly based on the Dice coefficient. 3D fCNNs is another direction to segment volumetric images.

   In the “Key aspects of successful deep learning methods” part, it gives some helpful tips for training deep neural nets: 1) data pre-process & argumentation. 2) fit the network input size and receptive field to the specific problem. 3) hyperparameter optimization, but it was regarded not as important as the previous two.

2. Greenspan, H., Ginneken, B. van, & Summers, R. M. (2016). Guest Editorial Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique. IEEE Transactions on Medical Imaging, 35(5), 1153–1159. https://doi.org/10.1109/TMI.2016.2553401

   This paper introduced problems in medical image analysis, and how researchers used different deep learning tools as approaches. For example, for the problem of unbalanced dataset, Van Grinsven et al. dynamically selected misclassified negative samples during training. It brought about several important issues in training the deep networks: 1) 3D volumatric data; 2) unsupervised vs. supervised learning; 3) unbalanced data; 4) transfer learning and fine tuning; and 5) ground truth, from the experts to the non-experts.

3. Shen, D., Wu, G., & Suk, H.-I. (2017). Deep Learning in Medical Image Analysis. Annual Review of Biomedical Engineering, 19(1), 221–248. https://doi.org/10.1146/annurev-bioeng-071516-044442

   This review listed some previous work of applying deep learning to medical image registration, anatomical/cell structures detection, tissue segmentation, computer-aided disease diagnosis and prognosis. Unlike the above reviews that gives many research progress on different applications, this review only listed one or two works and went into great details, but it lacked the comparison and correlation analysis between works.

4. Jha, S., & Topol, E. J. (2016). Adapting to Artificial Intelligence: Radiologists and Pathologists as Information Specialists. JAMA, 316(22), 2353–2354. https://doi.org/10.1001/jama.2016.17438

   Will Radiologists and Pathologists lose their jobs as the AI gradually reached expert-level performance? This paper brings about an interesting concept of “Information Specialists”. Its role is to “manage the information extracted by artificial intelligence in the clinical context of the patient”. AI will gradually replace some high-volume and tedious screening work, and will request radiologists and pathologist to face more challenging work. Some of its viewpoints sounded like science fiction: they are aggressive, but still quite reasonable. Dr Topol was one of the authors, and he is a famous digital health researcher.

5. D. L. Rubin, H. Greenspan, and J. F. Brinkley, “Biomedical Imaging Informatics,” in Biomedical Informatics, E. H. Shortliffe and J. J. Cimino, Eds. London: Springer London, 2014, pp. 285–327.

   medical imaging informatics overview

   This book chapter is a nice introduction and overview of the biomedical imaging informatics field. It focuses on methods for computational representation and processing images in biomedicine. Medical images are challenging data to be processed by computers since they are unstructured data, very little use of geometric shape models can be defined fro a priori knowledge. Besides, there are large inter-person variations, and the large spectrum of abnormal states can greatly modify tissue properties or deform structure.

   Because no single imaging technique satisfies all the desiderata fro depicting the broad variety of types of pathology, there are many difference imaging modalities that specialized in different pathologies. The imaging modalities include: Light, Ultrasound, X-ray, CT, MRI, Nuclear Medicine Imaging. They varies by two characteristics: spatial and functional information. These modalities varies according to spatial resolution and functional information. For example, MRI provides high spatial resolution, whereas fMRI and PET provides functional information. When it introduces CT, one impressive part is that CT did not became practical for generating high quality images until the advent of powerful computers and development of computer-based reconstruction techniques, which represent one of the most spectacular applications of computers in medicine.

   Image processing pipelines are generally built using one or more of the following fundamental image processing methods: global processing, image enhancement, image rendering/visualization, image quantitation, image segmentation, image registration, and image reasoning. Popular segmentation techniques include reconstruction from serial sections, region-based methods, edge-based methods, model or knowledge-based methods, and combined methods.

6. B. Erickson and R. A. Greenes, “Imaging Systems in Radiology,” in Biomedical Informatics, E. H. Shortliffe and J. J. Cimino, Eds. London: Springer London, 2014, pp. 593–611.

   Imaging system

   This book chapter gives a overview of methods for generating and manipulating images.

   The roles for imaging in biomedicine are:

   1. detection and diagnosis;
   2. assessment and planning
   3. image-guided procedures
   4. communication
   5. education and training
   6. research
   7. the radiologic process and its interactions

   The key components of medical image interpretation are detection, description, and diagnosis.

7. P. Lakhani et al., “Machine Learning in Radiology: Applications Beyond Image Interpretation,” Journal of the American College of Radiology, vol. 15, no. 2, pp. 350–359, Feb. 2018.

   radiology applications

   Machine learning is revolutionizing radiology. Written by radiologists, this paper provides some application area and use cases of machine learning in radiology beyond imaging interpretation. It can be served as a need analysis with radiologists when developing novel machine learning systems for radiology.

   The authors list and explain some potential applications of machine learning, including: creating study protocols or decision support, hanging protocols, improving image quality (image reconstruction or super-resolution), decreasing MR scanner time, optimizing staffing and scanner utilization, billing and collections, reporting (image-text generation), text understanding, image quantification, radiomics, and population health.

   It would be better if the authors provide some research projects or public dataset as examples for each area of applications. It also didn’t mention what kind of data will be needed, or how to curate such dataset for these application areas. It can also be improved by pairing each application with one or more machine learning research areas.

8. W. Luo et al., “Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research: A Multidisciplinary View,” Journal of Medical Internet Research, vol. 18, no. 12, p. e323, 2016.

   reporting machine learning models

   With the popularity of machine learning method in biomedical research, some guidelines are needed to regulate the proper use of machine learning. This study conducted interviews with machine learning researchers to generate the expert consensus as the guideline.

   The guideline includes items to report in the publication. It is very helpful as a checklist for researchers to check at different research stages, from study proposal, experiment design, to metrics evaluation and discussion.

   When reporting the final model and performance, it states that “comparison with other models in the literature should be based on confidence intervals”. For classification problems, the common reporting metrics include sensitivity, specificity, positive predictive value, negative predictive value, area under the ROC curve, and calibration plot. It also suggests reporting which subpopulation has the best prediction and which subpopulation is most difficult to predict.

   I found the data leakage part is very helpful. It divided the problem into two kinds: 1) outcome leakage: the predicted labels leak to the independent variable. 2) validation leakage: ground truth in the training set may propagate to the validation set. A common mistake is that the same patients appear in both the training and validation set.

   Although most of the recommendations are common sense in the field, it would be better if it could list references for these recommendations to make the expert opinions more solid.