Received: 28 January 2016
接收：2016 年 1 月 28 日
Accepted: 27 April 2016
接受：2016 年 4 月 27 日
Published: 23 May 2016
发布日期：2016 年 5 月 23 日

Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis
深度学习作为提高组织病理学诊断准确性和效率的工具

Abstract 摘要

Geert Litjens $^{1}$ , Clara I. Sánchez $^{2}$ , Nadya Timofeeva $^{1}$ , Meyke Hermsen $^{1}$ , Iris Nagtegaal $^{1}$ , Iringo Kovacs $^{3}$ , Christina Hulsbergen - van de Kaa $^{1}$ , Peter Bult $^{1}$ , Bram van Ginneken² & Jeroen van der Laak $^{1}$
Geert Litjens $^{1}$ ，Clara I. Sánchez $^{2}$ ，Nadya Timofeeva $^{1}$ ，Meyke Hermsen $^{1}$ ，Iris Nagtegaal $^{1}$ ，Iringo Kovacs $^{3}$ ，Christina Hulsbergen - van de Kaa $^{1}$ ，Peter Bult $^{1}$ ，Bram van Ginneken² & Jeroen van der Laak $^{1}$

Pathologists face a substantial increase in workload and complexity of histopathologic cancer diagnosis due to the advent of personalized medicine. Therefore, diagnostic protocols have to focus equally on efficiency and accuracy. In this paper we introduce ‘deep learning’ as a technique to improve the objectivity and efficiency of histopathologic slide analysis. Through two examples, prostate cancer identification in biopsy specimens and breast cancer metastasis detection in sentinel lymph nodes, we show the potential of this new methodology to reduce the workload for pathologists, while at the same time increasing objectivity of diagnoses. We found that all slides containing prostate cancer and micro- and macro-metastases of breast cancer could be identified automatically while 30-40% of the slides containing benign and normal tissue could be excluded without the use of any additional immunohistochemical markers or human intervention. We conclude that ‘deep learning’ holds great promise to improve the efficacy of prostate cancer diagnosis and breast cancer staging.
病理学家因个性化医疗的出现而面临工作量和工作复杂性的显著增加。因此，诊断方案必须同样关注效率和准确性。在本文中，我们引入“深度学习”作为一种提高组织病理学切片分析客观性和效率的技术。通过两个例子，即活检标本中的前列腺癌识别和哨兵淋巴结中的乳腺癌转移检测，我们展示了这种新方法减少病理学家工作量的潜力，同时增加诊断的客观性。我们发现，所有含有前列腺癌和乳腺癌微转移和宏观转移的切片都可以自动识别，而 30-40%含有良性组织和正常组织的切片可以在不使用任何额外的免疫组化标记或人工干预的情况下排除。我们得出结论，深度学习对提高前列腺癌诊断和乳腺癌分期效率具有巨大潜力。

Microscopic analysis of hematoxylin and eosin (H&E) stained sections has been the basis for cancer diagnosis and grading for the past century

^{1}

. Protocols for the complete workup of biopsies or resected tissue specimens, including microscopic analysis, exist for many of the most common cancer types (e.g. lung, breast, prostate). Use of these protocols has led to strong prognostic and widely used grading strategies (e.g. the Gleason grading system

)^{2}

.
显微镜分析苏木精和伊红（H&E）染色切片在过去一个世纪一直是癌症诊断和分级的基础。对于许多最常见的癌症类型（例如肺癌、乳腺癌、前列腺癌），存在包括显微镜分析在内的活检或切除组织标本的完整检查方案。使用这些方案已导致强大的预后和广泛使用的分级策略（例如，Gleason 分级系统）。

Due to the rise in cancer incidence and patient-specific treatment options, diagnosis and grading of cancer has become increasingly complex. Pathologists nowadays have to go over a large number of slides, often including additional immunohistochemical stains, to come to a complete diagnosis. Moreover, there is an increase in the amount of quantitative parameters pathologists have to extract for commonly used grading systems (e.g. lengths, surface areas, mitotic counts)

^{3}

. Due to these difficulties, analysis protocols have been adapted and fine-tuned to offer the best balance between prognostic power and feasibility in daily clinical routine

^{4}

.
由于癌症发病率的上升和针对患者的特定治疗方案，癌症的诊断和分级变得越来越复杂。病理学家现在必须检查大量切片，通常包括额外的免疫组化染色，才能得出完整的诊断。此外，病理学家需要提取的定量参数数量增加，用于常用的分级系统（例如长度、表面积、有丝分裂计数）

^{3}

。由于这些困难，分析方案已经适应并微调，以在日常临床实践中提供最佳的前瞻性力量和可行性之间的平衡

^{4}

。

The recent introduction of whole-slide scanning systems offers an opportunity to quantify and improve histopathologic procedures. These systems digitize glass slides with stained tissue sections at high resolution. Digital whole slide images (WSI) allow the application of image analysis techniques to aid pathologists in the examination and quantification of slides

^{5}

. One such technique which has gained prominence in the last five years in other fields is ‘deep learning’. While ‘deep learning’ cannot be considered a single technique, it can roughly be described as the application of multi-layered artificial neural networks to a wide range of problems, from speech recognition to image analysis. In recent years, ‘deep learning’ techniques have quickly become the state of the art in computer vision. A specific neural network subtype (convolutional neural networks;

{CNN}^{7 / 8}

has become the de facto standard in image recognition and is approaching human performance in a number of tasks

^{6}

. These systems function by learning relevant features directly from huge image databases (typically millions of images). This is in contrast to more traditional pattern recognition techniques, which strongly rely on manually crafted quantitative feature extractors.
近期引入的全切片扫描系统为量化并改进组织病理学程序提供了机会。这些系统能以高分辨率数字化带有染色组织切片的玻片。数字全切片图像（WSI）允许应用图像分析技术，帮助病理学家检查和量化切片

^{5}

。在过去五年中，在其他领域获得显著关注的某项技术是“深度学习”。虽然“深度学习”不能被视为单一技术，但它大致可以描述为将多层人工神经网络应用于广泛的问题，从语音识别到图像分析。近年来，“深度学习”技术在计算机视觉领域迅速成为最先进的技术。一种特定的神经网络亚型（卷积神经网络；

{CNN}^{7 / 8}

已成为图像识别的事实标准，并在多项任务中接近人类性能

^{6}

。这些系统通过直接从巨大的图像数据库（通常是数百万张图像）中学习相关特征来运行。这与更传统的模式识别技术形成对比，后者强烈依赖于手动制作的定量特征提取器。

In spite of these huge successes, ‘deep learning’ techniques have not yet made a big impact on the field of medical imaging. One of the main reasons is that for the traditional imaging based specialties (e.g. radiology) the large
尽管取得了巨大成功，但“深度学习”技术尚未对医学影像领域产生重大影响。其中一个主要原因是，对于基于传统影像的专业（例如放射学）来说，大量的

Figure 1. Processing pipeline of a convolutional neural network for the detection of prostate cancer in H&E-stained whole slide biopsy specimens. The four layers indicated with C, meaning a convolutional layer, can be considered a ‘feature extraction’-stage were consecutively higher level features are extracted from the image patch. The layers indicated by the letter M are max pooling layers which reduce image size and provide improved translational invariance to the network. The last three layers are the ‘classification’ layers (indicated with F ) which, based on the given features, indicates whether the image patch contains cancer or not. Such a network can subsequently be applied to every pixel in a whole slide image in a ‘sliding window’-fashion

^{27}

.
图 1. 用于检测 H&E 染色全切片活检标本前列腺癌的卷积神经网络处理流程。用 C 表示的四个层，代表卷积层，可以视为“特征提取”阶段，依次从图像块中提取更高层次的特征。用字母 M 表示的层是最大池化层，它们减小图像尺寸并提高了网络的对齐不变性。最后三个层是“分类”层（用 F 表示），基于给定特征，指示图像块是否包含癌症。这样的网络可以随后以“滑动窗口”的方式应用于全切片图像的每个像素。
numbers of images that are needed to train complex ‘deep learning’ systems are not readily available. In digital histopathology this is easier: one WSI typically contains trillions of pixels from which hundreds of examples of cancerous glands (in the case of prostate or breast cancer) can be extracted.
所需训练复杂“深度学习”系统的图像数量并不容易获得。在数字病理学中，这要容易一些：一张全切片显微镜图像通常包含万亿像素，从中可以提取出数百个癌性腺体（例如前列腺癌或乳腺癌）的示例。

Some initial work has been published over the last five years discussing the application of ‘deep learning’ techniques to microscopic and histopathologic images. Ciresan et al. were the first to apply convolutional neural networks to the task of mitosis counting for primary breast cancer grading

^{9}

. Furthermore, in a different publication, they showed the applicability of patch-driven convolutional neural networks to segmentation tasks

^{10}

. Wang et al. later expanded the work on mitosis detection by combining hand-crafted features and convolutional neural networks

^{11}

. Other applications of convolutional networks include primary breast cancer detection

^{12}

, glioma grading

^{13}

and epithelium and stroma segmentation

^{14}

. Last, Su et al. used another ‘deep learning’ technique, called stacked denoising auto-encoders to perform cell detection and segmentation in lung cancer and brain tumors

^{15}

.
过去五年中，一些关于将“深度学习”技术应用于显微镜和病理图像应用的初步工作已发表。Ciresan 等人是第一个将卷积神经网络应用于原发乳腺癌分级中细胞分裂计数任务的人

^{9}

。在另一篇论文中，他们展示了基于补丁驱动的卷积神经网络在分割任务中的应用

^{10}

。Wang 等人后来通过结合手工特征和卷积神经网络扩展了细胞分裂检测的工作

^{11}

。卷积网络的其他应用包括原发乳腺癌检测

^{12}

、胶质瘤分级

^{13}

和上皮和间质分割

^{14}

。最后，Su 等人使用另一种名为堆叠降噪自编码器的“深度学习”技术，在肺癌和脑肿瘤中进行细胞检测和分割

^{15}

。

This study investigates the general applicability of CNNs to improve the efficiency of cancer diagnosis in H&E images by applying it to two novel tasks: the detection of prostate cancer in biopsy specimens and the detection of breast cancer metastases in resected sentinel lymph nodes.
这项研究探讨了卷积神经网络（CNNs）在提高 H&E 图像癌症诊断效率方面的通用适用性，通过将其应用于两个新颖的任务：检测活检标本中的前列腺癌和检测切除的哨兵淋巴结中的乳腺癌转移。

The number of prostate biopsy sections has strongly increased in the past decades due to the advent of prostate specific antigen (PSA) testing

^{16}

. Because of the nature of the standard biopsy procedure (eight to twelve random biopsies under ultrasound-guidance), each procedure results in several slides. The majority of these slides typically do not contain cancer. The histopathological analysis could be streamlined significantly if these normal slides could automatically be excluded without expelling any slides containing cancer. We collected consecutive single-center biopsy specimens of 254 patients who underwent MR-guided biopsy procedures for prostate cancer at our institution. These specimens were prepared according to standard histopathologic protocol and subsequently digitized using an Olympus VS120-S5 system (Olympus, Tokyo, Japan).
前列腺活检切片数量在过去几十年中显著增加，归因于前列腺特异性抗原（PSA）检测的出现

^{16}

。由于标准活检程序（在超声引导下进行八到十二个随机活检），每次程序都会产生几个切片。这些切片中的大多数通常不含有癌细胞。如果能够自动排除这些正常的切片而不排除含有癌细胞的切片，病理学分析可以显著简化。我们收集了 254 名在我机构进行前列腺癌 MR 引导活检的患者连续单中心活检标本。这些标本按照标准病理学协议制备，并随后使用 Olympus VS120-S5 系统（Olympus，东京，日本）进行数字化。

The sentinel lymph node procedure is well known for its tedious inspection protocol

^{4}

. Several sections of the lymph node have to be investigated for micro-metastases (

0.2 - 2 mm

) and macro-metastases (

> 2 mm

). Furthermore, around

60 - 70 %

of the sentinel lymph nodes do not contain any metastases

^{17}

. In this paper we focus on the sentinel lymph node procedure for breast cancer with the aim of identifying slides which do not contain micro- or macro-metastases. Also, we tried to identify the correct location of the metastases within a specific slide. In total 271 patients were included from our institution. Specimens were prepared according to the standard histopathologic protocol and subsequently digitized using a 3DHistech Pannoramic 250 Flash II scanner (3DHistech, Budapest, Hungary).
哨兵淋巴结手术因其繁琐的检查程序而闻名

^{4}

。需要调查淋巴结的几个部分以寻找微转移

0.2 - 2 mm

和宏转移

> 2 mm

。此外，大约

60 - 70 %

的哨兵淋巴结不含有任何转移

^{17}

。在本文中，我们专注于乳腺癌的哨兵淋巴结手术，目的是识别不含有微转移或宏转移的切片。同时，我们试图确定特定切片中转移的正确位置。总共纳入了我们机构 271 名患者。标本按照标准组织病理学程序制备，随后使用 3DHistech Pannoramic 250 Flash II 扫描仪（3DHistech，布达佩斯，匈牙利）进行数字化。

After digitization of the H&E-stained slides cancer and metastases were manually delineated using a computer mouse by a resident of pathology (I.K., prostate cancer experiment) and a lab technician (M.H., sentinel lymph node experiment), under the supervision of experienced pathologists (C. A. H. K., P. B.). From these annotated areas small prototype image regions (‘patches’) were extracted to train CNNs to detect cancer areas in validation data sets (schematic overview in Fig. 1). These validation data sets were used to optimize the network parameters. After training, the CNN was converted to a fully convolutional network which gave per-pixel predictions on the presence of cancer and metastases in separate, not previously used, test data sets. For prostate cancer detection the CNNs were evaluated on a per-slide level using receiver-operator curve (ROC)-analysis. We also investigated how well the system could exclude slides without cancer from further diagnostic processing. For the sentinel lymph node procedure, we assessed how well the system was capable of identifying individual micro- and macro-metastases using free-response ROC (FROC) analysis and if it is capable of excluding slides which do not contain any metastases using ROC analysis.
在 H&E 染色切片的数字化后，癌症和转移灶由病理科住院医师（I.K.，前列腺癌实验）和实验室技术人员（M.H.，哨兵淋巴结实验）使用计算机鼠标手动描绘，在经验丰富的病理学家（C. A. H. K.，P. B.）的监督下。从这些注释区域提取了小型原型图像区域（“补丁”），以训练 CNN 在验证数据集中检测癌症区域（图 1 中的示意图）。这些验证数据集用于优化网络参数。训练后，CNN 被转换为全卷积网络，在单独的、以前未使用的测试数据集中对癌症和转移灶的存在进行每像素预测。对于前列腺癌检测，CNN 在每张切片级别上使用接收者操作特征曲线（ROC）分析进行评估。我们还研究了该系统排除无癌切片进行进一步诊断处理的能力。对于哨兵淋巴结手术，我们评估了系统使用自由响应 ROC（FROC）分析识别单个微转移和宏转移的能力，以及它是否能够通过 ROC 分析排除不含有任何转移的切片。

Results 结果

Subjects. Prostate cancer. From the initial set of 254 patients, eleven were excluded because the glass slides were not available. Four were excluded because no biopsy was taken during the procedure and one was excluded as the tissue sample was too small for pathologic analysis. Out of the remaining 238, we randomly selected 225 glass slides for digitization, of which 100 were assigned to the training set, 50 to the validation set and 75 to the test set. The training set sampling was stratified such that a near-50/50-distribution between slides containing
主题。前列腺癌。从最初的 254 名患者中，有 11 名被排除，因为玻璃切片不可用。有 4 名被排除，因为在手术过程中没有进行活检，有 1 名被排除，因为组织样本太小，无法进行病理分析。在剩下的 238 名中，我们随机选择了 225 张玻璃切片进行数字化，其中 100 张分配到训练集，50 张分配到验证集，75 张分配到测试集。训练集的抽样是分层进行的，以便在包含和不含感兴趣特征的切片之间实现近 50/50 的分布。

Nr. of slides per category 每类别幻灯片数量	Training 训练	Validation 验证	Test 测试	Total 总计
Cancer 癌症	48 (62.94 $\pm 29.23)$ 48 (62.94 $\pm 29.23)$ )	$31 (62.32 \pm 27.88)$	$45 (64.90 \pm 25.22)$	$124 (64.02 \pm 26.78)$
$2 + 3$	0	1	0	1
$3 + 2$	0	2	0	2
$3 + 3$	11	9	14	34
$3 + 4$	23	9	12	44
$3 + 5$	0	0	1	1
$4 + 3$	7	6	10	23
$4 + 4$	5	1	3	9
$4 + 5$	2	2	3	7
$5 + 3$	0	1	0	1
$5 + 4$	0	0	2	2
Normal 正常	52	19	30	101
Total 总计	100	50	75	225

Table 1. Data details for the whole slide biopsy specimens used for the prostate cancer experiments. The first column indicates the categories and the first row indicates the different data sets. For the cancer category, slide distribution is also indicated according to Gleason Score. The numbers between brackets for the 'Cancer’row indicate the average volume percentage of cancer within the slides and the corresponding standard deviation.
表 1. 用于前列腺癌实验的整个切片活检标本的数据细节。第一列表示类别，第一行表示不同的数据集。对于癌症类别，根据 Gleason 评分也指出了切片分布。'癌症'行中的括号内的数字表示切片中癌症的平均体积百分比和相应的标准差。

Nr. of slides per category
每类别幻灯片数量

Training 训练

Validation 验证

Test 测试

Consecutive 连续

Total 总计

At least one macro-metastasis
至少一个宏观转移

46

无宏观转移，至少一个微观转移

No macro-metastasis, at least

one micro-metastasis

49

无宏观或微观转移，至少一例 ITC

No macro- or micro-metastases,

at least one instance of ITC

24

无宏观或微观转移，无 ITC 病例

No macro- or micro- metastases

and no instances of ITC

152

Total 总计

271

Table 2. Data details for the whole slide sentinel lymph node specimens used for the breast cancer metastasis experiments. The first column indicates the categories and the first row indicates the different data sets. (ITC = isolated tumor cells).
表 2. 用于乳腺癌转移实验的整个切片哨兵淋巴结标本的数据详情。第一列表示类别，第一行表示不同的数据集。（ITC = 纯肿瘤细胞）。
cancer and slides not containing cancer was obtained. All slides were successfully digitized and annotated. Further details on the selected slides can be found in Table 1.
癌症和不含癌症的切片已获得。所有切片均成功数字化并标注。有关所选切片的更多详细信息，请参阅表 1。

Breast cancer sentinel lymph nodes. Data collection for the sentinel lymph node experiments was performed in two batches. The first batch was obtained by including 173 slides from the case files of an experienced breast pathologist (P.B). These initial slides were split into a training (98), validation (33) and test (42) set. These slides were subsequently digitized and every metastasis was annotated. To make sure our results were not biased to a single pathologist’s case selection, we acquired a second set of data by including all the consecutive sentinel lymph node cases for breast cancer from October 2014 to April 2015, resulting in an additional 98 whole-slide images. For the second batch no on-slide annotations were available, only the per-case outcome (presence of macroand/or micro-metastases and isolated tumor cells (ITC)). Further details on the included cases can be found in Table 2. Of cases with only ITC, 22 out of 24 had additional immunohistochemistry ordered by the pathologist.
乳腺癌哨兵淋巴结。哨兵淋巴结实验的数据收集分为两批进行。第一批是通过包括一位经验丰富的乳腺病理学家（P.B）的病例档案中的 173 张切片获得的。这些初始切片被分为训练集（98 张）、验证集（33 张）和测试集（42 张）。这些切片随后被数字化，并对每个转移进行了标注。为了确保我们的结果不会受到单个病理学家病例选择的影响，我们通过包括 2014 年 10 月至 2015 年 4 月期间所有连续的乳腺癌哨兵淋巴结病例，获得第二组数据，从而增加了 98 张全切片图像。对于第二批，没有切片上的标注，只有每个病例的结果（存在宏观和/或微观转移和孤立肿瘤细胞（ITC））。关于所包括病例的更多详细信息，请参阅表 2。在只有 ITC 的病例中，24 例中有 22 例由病理学家额外进行了免疫组化。

Prostate cancer detection. A cancer likelihood map (CLM), the output of the CNN indicating cancer likelihood per pixel, for a representative WSI from the test set with cancer covering

30 %

of the tissue area is shown in Fig. 2. The cancerous glands indicated by the pathologist’s outline (in magenta) are correctly identified with high likelihood. The stroma within the annotation areas is correctly identified as a low cancer likelihood region (in green, most easily identifiable in the high-resolution sub-images).
前列腺癌检测。显示在图 2 中，对于测试集中具有癌症覆盖

30 %

组织面积的代表性全切片图像（WSI），CNN 输出的癌症可能性图（CLM），表示每个像素的癌症可能性。病理学家用洋红色勾勒出的癌性腺体被以高可能性正确识别。注释区域内的基质被正确识别为低癌症可能性区域（绿色，在高分辨率子图像中最容易识别）。

Several other examples are presented in Fig. 3. In Fig. 3b, we show a high-resolution sub-image of a false positive region. Due to cutting and histopathologic processing, tissue at the edges of the biopsy specimens often deforms and tears, resulting in abnormal appearance. If we examine this area closely, we can see that the false positive glands indeed show some features which are comparable to those of cancer (e.g. fusing glands, irregular shape). In general, we can clearly see a distinct separation between malignant (Figs 2 and 3a) and benign biopsy specimens (Fig. 3b,c) based on the CLMs.
图 3 展示了其他几个示例。在图 3b 中，我们展示了一个假阳性区域的超高分辨率子图像。由于切割和病理处理，活检样本边缘的组织常常变形和撕裂，导致外观异常。如果我们仔细检查这个区域，我们可以看到假阳性腺体确实显示出一些与癌症（例如融合腺体、不规则形状）相似的特征。总的来说，我们可以根据 CLMs 清楚地看到恶性（图 2 和图 3a）和良性活检样本（图 3b、c）之间的明显区别。

Quantitatively, the result of performing histogram analysis on the CLMs can be best represented using ROC analysis. In Fig. 4 we present the ROC curves for both median and

90^{th}

-percentile analysis of the cumulative histogram of the CLM over the independent test set. Indicated with the dashed lines are the raw ROC curves, the
定量地，对 CLMs 进行直方图分析的结果最好用 ROC 分析来表示。在图 4 中，我们展示了独立测试集上 CLM 累积直方图的均值和

90^{th}

百分位数的 ROC 曲线。虚线表示的是原始 ROC 曲线，

$^{1}$ Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands. $^{2}$ Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands. $^{3}$ Department of Pathology, Amphia Breda Medical Center, The Netherlands. Correspondence and requests for materials should be addressed to G.L. (email: gjslitjens@gmail.com)
; 病理学系，拉博德大学医学中心，尼美根，荷兰。 $^{2}$ 放射科与核医学系，拉博德大学医学中心，尼美根，荷兰。 $^{3}$ 病理学系，阿姆菲亚布里达医疗中心，荷兰。信件和材料请求应寄至 G.L.（邮箱：gjslitjens@gmail.com）

Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis 深度学习作为提高组织病理学诊断准确性和效率的工具

Abstract 摘要

Results 结果

Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis
深度学习作为提高组织病理学诊断准确性和效率的工具