Focal Loss for Dense Object Detection
致密物体检测的焦距损失

RetinaNet forms a single FCN comprised of a ResNet-FPN backbone, a classification subnet, and a box regression subnet, see Figure 3. As such, inference involves simply forwarding an image through the network. To improve speed, we only decode box predictions from at most 1k top-scoring predictions per FPN level, after thresholding detector confidence at 0.05. The top predictions from all levels are merged and non-maximum suppression with a threshold of 0.5 is applied to yield the final detections.
RetinaNet 形成一个单一的 FCN，由 ResNet-FPN 主干网、分类子网和盒回归子网组成，参见图 3。因此，推理涉及简单地通过网络转发图像。为了提高速度，在将检测器置信度阈值为 0.05 后，我们仅从每个 FPN 级别最多 1k 个得分最高的预测中解码框预测。合并所有级别的最高预测，并应用阈值为 0.5 的非最大抑制以生成最终检测结果。

We use the focal loss introduced in this work as the loss on the output of the classification subnet. As we will show in §5, we find that $\gamma=2$ works well in practice and the RetinaNet is relatively robust to $\gamma\in[0.5,5]$. We emphasize that when training RetinaNet, the focal loss is applied to all $\scriptstyle\sim$100k anchors in each sampled image. This stands in contrast to common practice of using heuristic sampling (RPN) or hard example mining (OHEM, SSD) to select a small set of anchors (e.g., 256) for each minibatch. The total focal loss of an image is computed as the sum of the focal loss over all $\scriptstyle\sim$100k anchors, normalized by the number of anchors assigned to a ground-truth box. We perform the normalization by the number of assigned anchors, not total anchors, since the vast majority of anchors are easy negatives and receive negligible loss values under the focal loss. Finally we note that $\alpha$, the weight assigned to the rare class, also has a stable range, but it interacts with $\gamma$ making it necessary to select the two together (see Tables 1a and 1b). In general $\alpha$ should be decreased slightly as $\gamma$ is increased (for $\gamma=2$, $\alpha=0.25$ works best).
我们使用本文中介绍的焦点损失作为分类子网输出上的损失。正如我们将在 §5 中展示的那样，我们发现它 $\gamma=2$ 在实践中运行良好，并且 RetinaNet 对 $\gamma\in[0.5,5]$ .我们强调，在训练 RetinaNet 时，焦点损失应用于每个采样图像中的所有 $\scriptstyle\sim$ 100k 锚点。这与使用启发式采样 （RPN） 或硬示例挖掘（OHEM、SSD）为每个小批量选择一小组锚点（例如 256）的常见做法形成鲜明对比。图像的总焦距损失计算为所有 $\scriptstyle\sim$ 100k 个锚点的焦距损失之和，由分配给地面实况框的锚点数归一化。我们通过分配的锚点数量而不是总锚点来执行归一化，因为绝大多数锚点都是简单的负数，并且在焦点损失下获得的损失值可以忽略不计。最后， $\alpha$ 我们注意到，分配给稀有类的权重也具有稳定的范围，但它与 $\gamma$ 需要同时选择两者相互作用（见表 1a 和 1b）。一般来说 $\alpha$ ，随着增加，应略有 $\gamma$ 减少（对于 $\gamma=2$ ， $\alpha=0.25$ 效果最好）。

We experiment with ResNet-50-FPN and ResNet-101-FPN backbones [20]. The base ResNet-50 and ResNet-101 models are pre-trained on ImageNet1k; we use the models released by [16]. New layers added for FPN are initialized as in [20]. All new conv layers except the final one in the RetinaNet subnets are initialized with bias $b=0$ and a Gaussian weight fill with $\sigma=0.01$. For the final conv layer of the classification subnet, we set the bias initialization to $b=-\log((1-\pi)/\pi)$, where $\pi$ specifies that at the start of training every anchor should be labeled as foreground with confidence of $\scriptstyle\sim$$\pi$. We use $\pi=.01$ in all experiments, although results are robust to the exact value. As explained in §3.3, this initialization prevents the large number of background anchors from generating a large, destabilizing loss value in the first iteration of training.
我们试验了 ResNet-50-FPN 和 ResNet-101-FPN 骨干网 [ 20]。基本 ResNet-50 和 ResNet-101 模型在 ImageNet1k 上进行了预训练;我们使用 [ 16] 发布的模型。为 FPN 添加的新层将初始化，如 [ 20 所示。除 RetinaNet 子网中的最后一个层外，所有新的卷积层都使用偏置 $b=0$ 和高斯权重填充进行 $\sigma=0.01$ 初始化。对于分类子网的最终卷积层，我们将偏置初始化设置为 $b=-\log((1-\pi)/\pi)$ ，其中 $\pi$ 指定在训练开始时，每个锚点都应标记为前景，置信度为 $\scriptstyle\sim$ $\pi$ 。我们在所有实验中都使用 $\pi=.01$ ，尽管结果对确切值是稳健的。如 §3.3 中所述，此初始化可防止大量后台锚点在训练的第一次迭代中产生较大的、不稳定的损失值。

RetinaNet is trained with stochastic gradient descent (SGD). We use synchronized SGD over 8 GPUs with a total of 16 images per minibatch (2 images per GPU). Unless otherwise specified, all models are trained for 90k iterations with an initial learning rate of 0.01, which is then divided by 10 at 60k and again at 80k iterations. We use horizontal image flipping as the only form of data augmentation unless otherwise noted. Weight decay of 0.0001 and momentum of 0.9 are used. The training loss is the sum the focal loss and the standard smooth $L_{1}$ loss used for box regression [10]. Training time ranges between 10 and 35 hours for the models in Table 1e.
RetinaNet 使用随机梯度下降 （SGD） 进行训练。我们在 8 个 GPU 上使用同步的 SGD，每个小批量总共 16 个图像（每个 GPU 2 个图像）。除非另有说明，否则所有模型都针对 90k 迭代进行训练，初始学习率为 0.01，然后在 60k 时除以 10，在 80k 迭代时再次除以。除非另有说明，否则我们使用水平图像翻转作为数据增强的唯一形式。使用0.0001的重量衰减和0.9的动量。训练损失是用于箱式回归的焦点损失和标准平滑 $L_{1}$ 损失的总和 [ 10]。表1e中模型的训练时间范围为10至35小时。

Our first attempt to train RetinaNet uses standard cross entropy (CE) loss without any modifications to the initialization or learning strategy. This fails quickly, with the network diverging during training. However, simply initializing the last layer of our model such that the prior probability of detecting an object is $\pi=.01$ (see §4.1) enables effective learning. Training RetinaNet with ResNet-50 and this initialization already yields a respectable AP of 30.2 on COCO. Results are insensitive to the exact value of $\pi$ so we use $\pi=.01$ for all experiments.
我们第一次尝试使用标准的交叉熵（CE）损失来训练RetinaNet，而没有对初始化或学习策略进行任何修改。这很快就会失败，因为网络在训练期间会发散。然而，简单地初始化我们模型的最后一层，使得检测对象的先验概率为 $\pi=.01$ （参见§4.1），就可以实现有效的学习。使用 ResNet-50 训练 RetinaNet，并且此初始化已经在 COCO 上产生了 30.2 的可观 AP。结果对 的 $\pi$ 确切值不敏感，因此我们 $\pi=.01$ 用于所有实验。

Our next attempt to improve learning involved using the $\alpha$-balanced CE loss described in §3.1. Results for various $\alpha$ are shown in Table 1a. Setting $\alpha=.75$ gives a gain of 0.9 points AP.
我们改进学习的下一个尝试涉及使用§3.1中描述的 $\alpha$ 平衡CE损失。各种 $\alpha$ 结果如表1a所示。设置 $\alpha=.75$ 时，AP 增益为 0.9 点。

Results using our proposed focal loss are shown in Table 1b. The focal loss introduces one new hyperparameter, the focusing parameter $\gamma$, that controls the strength of the modulating term. When $\gamma=0$, our loss is equivalent to the CE loss. As $\gamma$ increases, the shape of the loss changes so that “easy” examples with low loss get further discounted, see Figure 1. FL shows large gains over CE as $\gamma$ is increased. With $\gamma=2$, FL yields a 2.9 AP improvement over the $\alpha$-balanced CE loss.
使用我们建议的焦点损失的结果如表1b所示。焦点损失引入了一个新的超参数，即聚焦参数 $\gamma$ ，用于控制调制项的强度。当 $\gamma=0$ 时，我们的损失相当于 CE 损失。随着 $\gamma$ 损失的增加，损失的形状会发生变化，因此具有低损失的“简单”示例会进一步打折，参见图 1。FL 显示出比 CE 更大的收益，因为 $\gamma$ 增加了。使用 $\gamma=2$ ，FL 比 $\alpha$ 平衡 CE 损耗提高了 2.9 AP。

For the experiments in Table 1b, for a fair comparison we find the best $\alpha$ for each $\gamma$. We observe that lower $\alpha$’s are selected for higher $\gamma$’s (as easy negatives are down-weighted, less emphasis needs to be placed on the positives). Overall, however, the benefit of changing $\gamma$ is much larger, and indeed the best $\alpha$’s ranged in just [.25,.75] (we tested $\alpha\in[.01,.999]$). We use $\gamma=2.0$ with $\alpha=.25$ for all experiments but $\alpha=.5$ works nearly as well (.4 AP lower).
对于表1b中的实验，为了进行公平的比较，我们发现每个 $\gamma$ 实验都是最好的 $\alpha$ 。我们观察到，较低的 $\alpha$ 's 被选为较高 $\gamma$ 的 's（由于简单的否定因素被降低了权重，因此需要较少强调积极因素）。然而，总的来说，改变 $\gamma$ 的好处要大得多，事实上，最好的 $\alpha$ 范围只有 [.25，.75]（我们测试 $\alpha\in[.01,.999]$ 了）。 $\gamma=2.0$ 我们 $\alpha=.25$ 用于所有实验，但 $\alpha=.5$ 效果几乎一样好（AP 低 0.4）。

To understand the focal loss better, we analyze the empirical distribution of the loss of a converged model. For this, we take take our default ResNet-101 600-pixel model trained with $\gamma=2$ (which has 36.0 AP). We apply this model to a large number of random images and sample the predicted probability for $\scriptstyle\sim$$10^{7}$ negative windows and $\scriptstyle\sim$$10^{5}$ positive windows. Next, separately for positives and negatives, we compute FL for these samples, and normalize the loss such that it sums to one. Given the normalized loss, we can sort the loss from lowest to highest and plot its cumulative distribution function (CDF) for both positive and negative samples and for different settings for $\gamma$ (even though model was trained with $\gamma=2$).
为了更好地理解焦点损失，我们分析了收敛模型损失的经验分布。为此，我们采用默认的 ResNet-101 600 像素模型 $\gamma=2$ （具有 36.0 AP）进行训练。我们将该模型应用于大量随机图像，并对 $10^{7}$ 负窗口和 $\scriptstyle\sim$ $10^{5}$ 正窗口的预测概率 $\scriptstyle\sim$ 进行采样。接下来，分别针对正值和负值，我们计算这些样本的 FL，并将损失归一化，使其总和为 1。给定归一化损失，我们可以将损失从低到高排序，并绘制其正负样本和不同设置的累积分布函数 （CDF $\gamma$ ）（即使模型是用 $\gamma=2$ 训练的）。

Cumulative distribution functions for positive and negative samples are shown in Figure 4. If we observe the positive samples, we see that the CDF looks fairly similar for different values of $\gamma$. For example, approximately 20% of the hardest positive samples account for roughly half of the positive loss, as $\gamma$ increases more of the loss gets concentrated in the top 20% of examples, but the effect is minor.
正性样品和负性样品的累积分布函数如图4所示。如果我们观察正样本，我们会发现 CDF 对于不同的 值 $\gamma$ 非常相似。例如，大约 20% 的最难阳性样本约占正损失的一半，因为 $\gamma$ 增加的损失集中在前 20% 的样本中，但影响很小。

The effect of $\gamma$ on negative samples is dramatically different. For $\gamma=0$, the positive and negative CDFs are quite similar. However, as $\gamma$ increases, substantially more weight becomes concentrated on the hard negative examples. In fact, with $\gamma=2$ (our default setting), the vast majority of the loss comes from a small fraction of samples. As can be seen, FL can effectively discount the effect of easy negatives, focusing all attention on the hard negative examples.
$\gamma$ 对阴性样品的影响截然不同。对于 $\gamma=0$ ，正 CDF 和负 CDF 非常相似。然而，随着 $\gamma$ 增加，更多的权重集中在硬性反面例子上。事实上，在（我们的默认设置）下 $\gamma=2$ ，绝大多数损失来自一小部分样本。可以看出，FL可以有效地打折简单否定的影响，将所有注意力集中在硬否定的例子上。

[31] proposed to improve training of two-stage detectors by constructing minibatches using high-loss examples. Specifically, in OHEM each example is scored by its loss, non-maximum suppression (nms) is then applied, and a minibatch is constructed with the highest-loss examples. The nms threshold and batch size are tunable parameters. Like the focal loss, OHEM puts more emphasis on misclassified examples, but unlike FL, OHEM completely discards easy examples. We also implement a variant of OHEM used in SSD [22]: after applying nms to all examples, the minibatch is constructed to enforce a 1:3 ratio between positives and negatives to help ensure each minibatch has enough positives.
[ 31] 提出通过使用高损耗示例构建小批量来改进两级探测器的训练。具体来说，在OHEM中，每个示例都根据其损耗进行评分，然后应用非最大抑制（nms），并使用最高损耗示例构建小批量。nms 阈值和批量大小是可调参数。与焦点损失一样，OHEM 更强调错误分类的例子，但与 FL 不同的是，OHEM 完全放弃了简单的例子。我们还实现了 SSD 中使用的 OHEM 变体 [ 22]：将 nms 应用于所有示例后，小批量被构建为强制执行正负之间的 1：3 比例，以帮助确保每个小批量都有足够的正极。

We test both OHEM variants in our setting of one-stage detection which has large class imbalance. Results for the original OHEM strategy and the ‘OHEM 1:3’ strategy for selected batch sizes and nms thresholds are shown in Table 1d. These results use ResNet-101, our baseline trained with FL achieves 36.0 AP for this setting. In contrast, the best setting for OHEM (no 1:3 ratio, batch size 128, nms of .5) achieves 32.8 AP. This is a gap of 3.2 AP, showing FL is more effective than OHEM for training dense detectors. We note that we tried other parameter setting and variants for OHEM but did not achieve better results.
我们在具有较大类别不平衡的单阶段检测设置中测试了两种 OHEM 变体。表 1d 显示了原始 OHEM 策略和选定批量大小和 nms 阈值的“OHEM 1：3”策略的结果。这些结果使用 ResNet-101，我们使用 FL 训练的基线在此设置下实现了 36.0 AP。相比之下，OHEM 的最佳设置（无 1：3 比率，批次大小 128，nms 为 .5）达到 32.8 AP。这是 3.2 AP 的差距，表明 FL 在训练密集探测器方面比 OHEM 更有效。我们注意到，我们尝试了OHEM的其他参数设置和变体，但没有获得更好的结果。

Finally, in early experiments, we attempted to train with the hinge loss [13] on $p_{\textrm{t}}$, which sets loss to 0 above a certain value of $p_{\textrm{t}}$. However, this was unstable and we did not manage to obtain meaningful results. Results exploring alternate loss functions are in the appendix.
最后，在早期的实验中，我们尝试用铰链损耗 [ 13] 进行 $p_{\textrm{t}}$ 训练，将 的损耗设置为 0 高于某个值 $p_{\textrm{t}}$ 。然而，这是不稳定的，我们没有设法获得有意义的结果。探索替代损失函数的结果见附录。

One of the most important design factors in a one-stage detection system is how densely it covers the space of possible image boxes. Two-stage detectors can classify boxes at any position, scale, and aspect ratio using a region pooling operation [10]. In contrast, as one-stage detectors use a fixed sampling grid, a popular approach for achieving high coverage of boxes in these approaches is to use multiple ‘anchors’ [28] at each spatial position to cover boxes of various scales and aspect ratios.
在单级检测系统中，最重要的设计因素之一是它对可能的图像框空间的覆盖密度。两级检测器可以使用区域池化操作对任意位置、比例和纵横比的盒子进行分类[10]。相比之下，由于单级探测器使用固定的采样网格，在这些方法中实现高覆盖率的一种流行方法是在每个空间位置使用多个“锚”[28]来覆盖各种比例和纵横比的盒子。

We sweep over the number of scale and aspect ratio anchors used at each spatial position and each pyramid level in FPN. We consider cases from a single square anchor at each location to 12 anchors per location spanning 4 sub-octave scales ($2^{k/4}$, for $k\leq 3$) and 3 aspect ratios [0.5, 1, 2]. Results using ResNet-50 are shown in Table 1c. A surprisingly good AP (30.3) is achieved using just one square anchor. However, the AP can be improved by nearly 4 points (to 34.0) when using 3 scales and 3 aspect ratios per location. We used this setting for all other experiments in this work.
我们扫描了 FPN 中每个空间位置和每个金字塔级别使用的比例和纵横比锚点的数量。我们考虑的情况从每个位置的单个方形锚点到每个位置的 12 个锚点，跨越 4 个亚八度音阶 （ $2^{k/4}$ ， for $k\leq 3$ ） 和 3 个纵横比 [0.5， 1， 2]。使用ResNet-50的结果如表1c所示。仅使用一个方形锚点即可实现令人惊讶的良好 AP （30.3）。但是，当每个位置使用 3 个比例和 3 个纵横比时，AP 可以提高近 4 分（达到 34.0）。我们在这项工作的所有其他实验中都使用了这个设置。

Finally, we note that increasing beyond 6-9 anchors did not shown further gains. Thus while two-stage systems can classify arbitrary boxes in an image, the saturation of performance w.r.t. density implies the higher potential density of two-stage systems may not offer an advantage.
最后，我们注意到，超过 6-9 个锚点并没有显示出进一步的收益。因此，虽然两级系统可以对图像中的任意框进行分类，但性能的饱和度意味着两级系统的较高电位密度可能无法提供优势。

Larger backbone networks yield higher accuracy, but also slower inference speeds. Likewise for input image scale (defined by the shorter image side). We show the impact of these two factors in Table 1e. In Figure 2 we plot the speed/accuracy trade-off curve for RetinaNet and compare it to recent methods using public numbers on COCO test-dev. The plot reveals that RetinaNet, enabled by our focal loss, forms an upper envelope over all existing methods, discounting the low-accuracy regime. RetinaNet with ResNet-101-FPN and a 600 pixel image scale (which we denote by RetinaNet-101-600 for simplicity) matches the accuracy of the recently published ResNet-101-FPN Faster R-CNN [20], while running in 122 ms per image compared to 172 ms (both measured on an Nvidia M40 GPU). Using larger scales allows RetinaNet to surpass the accuracy of all two-stage approaches, while still being faster. For faster runtimes, there is only one operating point (500 pixel input) at which using ResNet-50-FPN improves over ResNet-101-FPN. Addressing the high frame rate regime will likely require special network design, as in [27], and is beyond the scope of this work. We note that after publication, faster and more accurate results can now be obtained by a variant of Faster R-CNN from [12].
骨干网络越大，精度越高，推理速度也越慢。输入图像比例（由较短的图像侧定义）也是如此。我们在表1e中显示了这两个因素的影响。在图 2 中，我们绘制了 RetinaNet 的速度/准确性权衡曲线，并将其与最近使用 COCO test-dev 上的公共数据的方法进行了比较。该图显示，RetinaNet在焦点损失的支持下，在所有现有方法上形成了一个上包络线，从而降低了低精度状态。带有 ResNet-101-FPN 和 600 像素图像比例的 RetinaNet（为简单起见，我们用 RetinaNet-101-600 表示）与最近发布的 ResNet-101-FPN Faster R-CNN [ 20] 的准确性相匹配，每张图像的运行时间为 122 毫秒，而每张图像的运行时间为 172 毫秒（均在 Nvidia M40 GPU 上测量）。使用更大的尺度使 RetinaNet 能够超越所有两阶段方法的精度，同时仍然更快。为了获得更快的运行时间，只有一个操作点（500 像素输入），在该操作点上使用 ResNet-50-FPN 比 ResNet-101-FPN 有所改进。解决高帧率问题可能需要特殊的网络设计，如[27]所示，这超出了这项工作的范围。我们注意到，在发表后，现在可以通过 [ 12 的 Faster R-CNN 变体获得更快、更准确的结果。