Since the first proposal of Mixture-of-Experts (MoE) Jacobs et al. (1991); Jordan & Jacobs (1994), different MoE models have been proposed based on various experts models, for example, hidden Markov models (Jordan et al., 1996), Gaussian Process (Tresp, 2000), and support vector machine (Collobert et al., 2001). With the rise of deep learning, Eigen et al. propose the use of several sets of routers and experts to build a stacked model, namely Deep MoE Eigen et al. (2013).
自从杰克布斯等人（1991年）以及乔丹与杰克布斯（1994年）首次提出专家混合模型（MoE）以来，基于各种专家模型，已经提出了不同的MoE模型，例如，隐马尔可夫模型（乔丹等人，1996年），高斯过程（特雷斯普，2000年）和支持向量机（科洛贝特等人，2001年）。随着深度学习的兴起，艾根等人提出使用多组路由器和专家来构建一个堆叠模型，即深度MoE（艾根等人，2013年）。

A MoE layer consists of a router function, denoted as $h(\cdot;{\bm{W}}_{r})$, followed by $K$ experts in parallel, denoted as $\{f_{i}(\cdot;{\bm{\theta}}_{i})\}_{i=1}^{K}$. Usually, the router function is set as a linear function, i.e., $h({\mathbf{x}};{\bm{W}}_{r})={{\bm{W}}_{r}}^{\top}{\mathbf{x}}$ where ${\bm{W}}_{r}\in{\mathbb{R}}^{d\times K}$ for input ${\mathbf{x}}\in{\mathbb{R}}^{d}$, and experts are multi-layer perceptrons (MLPs) with a non-linear activation function (Chen et al., 2022; Fedus et al., 2022; Shazeer et al., 2017). The output of a MoE layer takes the form:
MoE层由一个路由函数 $h(\cdot;{\bm{W}}_{r})$ 组成，后面跟着并行的 $K$ 个专家 $\{f_{i}(\cdot;{\bm{\theta}}_{i})\}_{i=1}^{K}$ 。通常，路由函数被设置为线性函数，即 $h({\mathbf{x}};{\bm{W}}_{r})={{\bm{W}}_{r}}^{\top}{\mathbf{x}}$ ，其中 ${\bm{W}}_{r}\in{\mathbb{R}}^{d\times K}$ 为输入 ${\mathbf{x}}\in{\mathbb{R}}^{d}$ ，而专家是带有非线性激活函数的多层感知机（MLPs）（Chen等，2022；Fedus等，2022；Shazeer等，2017）。MoE层的输出形式为：

where ${\mathbb{I}}$ contains the selected indices of experts and the scaling factor $\alpha_{i}$ is defined as
其中 ${\mathbb{I}}$ 包含了专家的选定索引，而缩放因子 $\alpha_{i}$ 被定义为

Different selection mechanism of ${\mathbb{I}}$ leads to different models. The soft-routing model (Jordan & Jacobs, 1994) selects all experts, i.e., ${\mathbb{I}}=[K]$, which leads to high computational overheads. The switch-routing model (Fedus et al., 2022) selects the top-$1$ expert, i.e., ${\mathbb{I}}=\operatorname*{arg\,max}_{i\in[K]}\alpha_{i}(\cdot)$, introducing little extra computational overhead.
不同的选择机制会导致不同的模型。软路由模型（Jordan & Jacobs, 1994）选择所有专家，即 ${\mathbb{I}}=[K]$ ，这导致了高计算开销。开关路由模型（Fedus等，2022）选择排名前 $1$ 的专家，即 ${\mathbb{I}}=\operatorname*{arg\,max}_{i\in[K]}\alpha_{i}(\cdot)$ ，引入了很少的额外计算开销。

Encouraged by the advantage of MoE-based large models that drastically increasing the number of parameters leads to little computational overhead, many large-scale architectures have been proposed such as the Sparsely-Gated MoE (Shazeer et al., 2017), Gshard (Lepikhin et al., 2020), and Switch Transformers (Fedus et al., 2022). Specifically, the Sparsely-Gated MoE proposes a trainable router function to determine the expert to be activated for each sample, which makes it possible to build very large MoE-based models as it improves the computational efficiency by a large margin compared to the soft-routing selecting all experts. The Sparsely-Gated MoE scales LSTM models to 137 billion parameters achieving outstanding performance. Switch Transformers, the most widely used transformer-based large MoE, converts T5 models (Raffel et al., 2020) to their MoE versions. All Switch Transformers outperform their foundation dense model with the same FLOPs.
受到基于MoE的大型模型的优势鼓舞，即大幅增加参数数量导致的计算开销很小，许多大规模架构被提出，如稀疏门控MoE（Shazeer等，2017年）、Gshard（Lepikhin等，2020年）和Switch Transformers（Fedus等，2022年）。具体来说，稀疏门控MoE提出了一种可训练的路由函数，用于确定每个样本要激活的专家，这使得构建非常大的基于MoE的模型成为可能，因为与选择所有专家的软路由相比，它大幅提高了计算效率。稀疏门控MoE将LSTM模型扩展到1370亿参数，取得了卓越的性能。Switch Transformers是最广泛使用的基于变换器的大型MoE，将T5模型（Raffel等，2020年）转换为其MoE版本。所有Switch Transformers在相同的FLOPs下都超过了它们的基础密集模型。

In our study, we found that large MoE models do not efficiently utilize GPUs. As shown in Eq. 1, we denote an expert as activated if $i\in{\mathbb{I}}$. Inactivated experts remain idle in the forward pass, leading to low effective GPU memory utilization. Effective GPU memory refers to the memory storing parameters that are effective for the forwarding of the model. The inactivated experts occupy a large amount of GPU memory while remaining idle, leading to low effective GPU memory utilization. To quantitatively analyze the GPU memory utilization, we provide a summary of Switch Transformers on model size and MoE layer size in Table 2. It is shown that for all Switch Transformers, especially the large ones, MoE layers occupy a large portion of GPU memory. Meanwhile, most of the parameters of the MoE layers are idle during one forward pass. To ascertain the amount of ineffective GPU memory, we feed samples from the SST2 dataset to Switch Transformers and record the corresponding effective memory utilization rates. The results are depicted in Fig. 2. For large Switch Transformers such as Switch-base-128 and Switch-base-256, the ineffective GPU memory for short sentences is around 24GB and 50GB, respectively. Even for the longest sentences with 80 tokens, the ineffective GPU memory is around 20GB and 46GB, respectively. Our method, SiDA, can save all ineffective GPU memory, outperforming baselines by a large margin. Further results on GPU memory reduction across datasets can be found in Section 4.
在我们的研究中，我们发现大型MoE模型并不能有效利用GPU。如方程1所示，我们将一个专家定义为激活状态，如果 $i\in{\mathbb{I}}$ 。未激活的专家在前向传播中保持空闲，导致有效GPU内存利用率低。有效GPU内存指的是存储对模型前向传播有效的参数的内存。未激活的专家占用了大量GPU内存同时保持空闲，导致有效GPU内存利用率低。为了定量分析GPU内存的利用率，我们在表2中提供了Switch Transformers模型大小和MoE层大小的总结。结果显示，对于所有Switch Transformers，特别是大型的，MoE层占用了大量GPU内存。同时，MoE层的大多数参数在一次前向传播中处于空闲状态。为了确定无效GPU内存的数量，我们向Switch Transformers输入SST2数据集的样本，并记录相应的有效内存利用率。结果如图2所示。对于大型Switch Transformers，如Switch-base-128和Switch-base-256，短句子的无效GPU内存分别约为24GB和50GB。即使对于最长的含有80个词汇的句子，无效GPU内存也分别约为20GB和46GB。我们的方法SiDA可以节省所有无效GPU内存，大幅度超过基准线。关于跨数据集GPU内存减少的进一步结果可以在第4节找到。

Apart from the low effective GPU memory utilization, we also observed a high overhead on expert selection in the feedforward pass of MoE. Specifically, in all baseline implementations of MoE models, a non-negligible amount of time is consumed in the process of selecting the most suitable experts. We conduct experiments on SST2 with multiple MoE models and provide the profiling results of averaged inference time and expert selection overhead in Fig. 3. It is shown that the expert selection process consumes nearly $75\%$ of the total inference time for Switch-base-256, which is a bottleneck of the inference latency. Notably, the overhead associated with expert selection escalates with the scale of the model, further emphasizing the imperative of addressing the bottleneck in inference efficiency.
除了低有效GPU内存利用率外，我们还观察到MoE前馈过程中专家选择的高开销。具体来说，在所有MoE模型的基线实现中，选择最合适的专家的过程消耗了不可忽视的时间。我们在SST2上使用多个MoE模型进行实验，并在图3中提供了平均推理时间和专家选择开销的分析结果。结果显示，对于Switch-base-256，专家选择过程消耗了总推理时间的近一半，这是推理延迟的瓶颈。值得注意的是，随着模型规模的扩大，与专家选择相关的开销增加，进一步强调了解决推理效率瓶颈的重要性。

The sparse selection of experts is one of the critical observations that motivate SiDA. Our observation verifies that only a small portion of experts will be activated during inference.
稀疏专家选择是激励SiDA的关键观察之一。我们的观察验证了在推理过程中只有一小部分专家会被激活。

For each token, the router function will select either top-$K$ (Shazeer et al., 2017) or top-$1$ (Fedus et al., 2022) experts inducing a token level expert activation sparsity. However, the sparsity on sentences, typically with 512 or 768 tokens, remains elusive. Not to mention in the training stage, an expert loading balance loss must be applied, which forces the router to assign an almost equal number of tokens to each expert. Otherwise, router’s outputs will collapse to few experts leading to capacity degradation (Chen et al., 2022).
对于每个令牌，路由函数将选择顶尖的 $K$ （Shazeer等，2017年）或顶尖的 $1$ （Fedus等，2022年）专家，引发令牌级别的专家激活稀疏性。然而，对于通常包含512或768个令牌的句子，其稀疏性仍然难以捉摸。更不用说在训练阶段，必须应用一个专家负载平衡损失，这迫使路由器将几乎相同数量的令牌分配给每个专家。否则，路由器的输出将崩溃为少数几个专家，导致容量降级（Chen等，2022年）。

We test Switch Transformers with different number of experts on the SST2 dataset and report the sentence level sparsity in Fig. 4. Our observation verifies that the sparse activation pattern still exists at the sentence level for large MoE models such as Switch-base-128 and Switch-base-256. As shown in the figure, down to less than $40\%$ experts and $20\%$ experts are activated for Switch-base-128 and Switch-base-256, respectively. Even for the longest sentences with around 80 tokens, the ratio of idle experts is still higher than $70\%$ for Switch-base-128 and $80\%$ for Switch-base-256.
我们在SST2数据集上测试了具有不同专家数量的Switch Transformers，并在图4中报告了句子级别的稀疏性。我们的观察验证了，对于像Switch-base-128和Switch-base-256这样的大型MoE模型，稀疏激活模式在句子级别仍然存在。如图所示，对于Switch-base-128和Switch-base-256，激活的专家数量分别减少到少于 $40\%$ 个和 $20\%$ 个。即使对于大约有80个词符的最长句子，Switch-base-128和Switch-base-256的空闲专家比率仍然高于 $70\%$ 和 $80\%$ 。

We introduce a novel framework, Sparsity-inspired Data-Aware (SiDA), for efficient inference of large MoE models, whose overview is shown in Fig. 5. SiDA contains two parallel threads that run simultaneously, namely the Inference thread and the Hash-building thread. Consider a sequence of incoming batches, batch ${\mathbb{X}}_{j}$ is fed to the hash-building thread to build the hash table ${\mathbb{H}}_{j}$ storing expert activation patterns for batch ${\mathbb{X}}_{j}$, which will be pushed to the hash table queue. At the same time, the inference thread is handling the precedent batch ${\mathbb{X}}_{i}$ and operating dynamical offloading on MoE layers based on the hash table ${\mathbb{H}}_{i}$.
我们提出了一个新颖的框架，稀疏启发的数据感知（SiDA），用于高效推理大型MoE模型，其概览如图5所示。SiDA包含两个并行线程，同时运行，即推理线程和哈希构建线程。考虑一系列即将到来的批次，批次 ${\mathbb{X}}_{j}$ 被送入哈希构建线程以构建哈希表 ${\mathbb{H}}_{j}$ ，用于存储批次 ${\mathbb{X}}_{j}$ 的专家激活模式，该模式将被推送到哈希表队列中。同时，推理线程正在处理前一个批次 ${\mathbb{X}}_{i}$ ，并根据哈希表 ${\mathbb{H}}_{i}$ 对MoE层进行动态卸载。

Hash-building thread. The Hash-building thread consists of two components, a hash function and a hash table queue. For each incoming batch ( -a), the hash function will determine experts to be activated for each token at each layer and the corresponding scaling factor $\alpha$ ( -b). The predictions are stored in the hash table ${\mathbb{H}}_{j}$ for the batch ${\mathbb{X}}_{j}$ and pushed to the hash table queue ( -c). The hash function can be a predefined hash function if the MoE model is trained with the Hash layer (Roller et al., 2021). More commonly, for the MoE model using trained router functions, such as Switch Transformers, the hash function will be offline trained. We propose hash function training techniques dedicated to modern MoE models, which will be introduced in later sections.
构建哈希线程。构建哈希线程由两部分组成：一个哈希函数和一个哈希表队列。对于每个传入的批次（-a），哈希函数将确定每个层次上每个令牌要激活的专家及其相应的缩放因子 $\alpha$ （-b）。预测结果存储在批次 ${\mathbb{X}}_{j}$ 的哈希表 ${\mathbb{H}}_{j}$ 中，并推送到哈希表队列（-c）。如果MoE模型是与哈希层一起训练的，哈希函数可以是预定义的哈希函数（Roller等人，2021）。更常见的是，对于使用训练过的路由函数的MoE模型，如Switch Transformers，哈希函数将进行离线训练。我们提出了专门针对现代MoE模型的哈希函数训练技术，这将在后面的章节中介绍。

Inference thread. The inference thread performs two tasks, i.e., dynamically load activated experts and offload inactivated experts according to the hash table built by the hash-building thread, and use the SiDA MoE layers to inference input batches. Specifically, for each incoming batch ${\mathbb{X}}_{i}$ ( -a), the inference thread will first pop the hash table ${\mathbb{H}}_{i}$ from the hash table queue ( -b) and remain idle if ${\mathbb{H}}_{i}$ is not found. Notably, in practice, the inference thread takes a longer time to inference a batch than the hash-building thread to build a hash table for a batch. As a result, the inference thread never idles except at the very beginning. With the popped hash table ${\mathbb{H}}_{i}$, the next step is to dynamically load and offload experts. Based on GPU memory budgets and the expert activation pattern of the current batch, the inference thread will load activated experts to GPU and offload inactivated experts to RAM ( -c). A first-in-first-out (FIFO) scheme is applied on experts if no memory budgets remain. The dynamical loading task of a MoE layer will be done right after the finish of inference on the previous batch following the pipeline parallelism mechanism Huang et al. (2019). Note that, in our system, all routers are offloaded to the main memory and do not participate in the forward pass. Lastly, the incoming batch ${\mathbb{X}}_{i}$ will be forwarded using the SiDA MoE layers specific to ${\mathbb{X}}_{i}$ ( -d).
推理线程。推理线程执行两项任务，即根据哈希构建线程构建的哈希表动态加载激活的专家并卸载未激活的专家，并使用SiDA MoE层对输入批次进行推理。具体来说，对于每个传入的批次 ${\mathbb{X}}_{i}$ （-a），推理线程首先从哈希表队列中弹出哈希表 ${\mathbb{H}}_{i}$ （-b），如果未找到 ${\mathbb{H}}_{i}$ 则保持空闲。值得注意的是，在实践中，推理线程对一个批次进行推理的时间比哈希构建线程为一个批次构建哈希表的时间要长。因此，除了最开始之外，推理线程从不空闲。有了弹出的哈希表 ${\mathbb{H}}_{i}$ ，下一步是动态加载和卸载专家。根据GPU内存预算和当前批次的专家激活模式，推理线程将加载激活的专家到GPU并将未激活的专家卸载到RAM（-c）。如果没有剩余的内存预算，则对专家采用先进先出（FIFO）方案。MoE层的动态加载任务将在上一个批次推理完成后立即进行，遵循Huang等人（2019）提出的流水线并行机制。注意，在我们的系统中，所有路由器都被卸载到主内存中，不参与前向传递。最后，传入的批次 ${\mathbb{X}}_{i}$ 将使用针对 ${\mathbb{X}}_{i}$ （-d）的SiDA MoE层进行转发。

In the design of SiDA, we spot three key challenges.
在SiDA的设计中，我们发现了三个关键挑战。

Challenge 1: How to efficiently obtain experts that are to be offloaded beforehand? Given the observation that experts are activated sparsely, it is trivial to save GPU memory by offloading inactivated experts to RAM. However, this naiv̈e implementation sacrifices the latency since expert activation patterns are inaccessible without the output of the router functions. It incurs large overheads to move experts between CPU and GPU after each router function as it breaks the forwarding pipeline. We propose to use an offline-trained hash function to acquire the expert activation pattern before inference starts for each batch. Furthermore, we design the hash function to run independently of model inference and build a hash-building thread running in parallel with the inference thread to achieve the efficiency requirements. By employing the hash-building thread, SiDA achieves outstanding latency compared to baselines since the expert selection, dynamical offloading, and inference all run in parallel.
挑战1：如何高效地提前获取将要卸载的专家？鉴于专家被激活的情况较为稀疏，通过将未激活的专家卸载到RAM中可以节省GPU内存，这一点并不复杂。然而，这种简单的实现牺牲了延迟，因为在没有路由函数输出的情况下，无法访问专家激活模式。在每个路由函数之后在CPU和GPU之间移动专家会打破前向传播管道，从而产生大量开销。我们提出使用一个离线训练的哈希函数，在每个批次的推理开始前获取专家激活模式。此外，我们设计哈希函数能够独立于模型推理运行，并构建一个与推理线程并行运行的哈希构建线程，以满足效率要求。通过使用哈希构建线程，SiDA在延迟方面与基准相比取得了卓越的表现，因为专家选择、动态卸载和推理都并行运行。

Challenge 2: How to leverage sparse cross-embedding dependency on experts activation to design a lightweight offline trained hash function? Considering the inference efficiency and the GPU memory consumption of the system, the hash function must be a lightweight predictor. However, simple predictors can hardly capture the contextual information of the sequence and can be easily distracted. Hence, it becomes crucial to enforce the predictor to focus on critical information. We empirically verify that there exists a sparse cross-embedding dependency on expert activation, i.e., a limited number of embeddings in the sequence jointly affect expert activation. This sparse cross-embedding dependency sheds light on the success of lightweight predictors. However, it is impractical and inefficient to rule out all possible outcomes to find the cross-embedding dependency for every token. In response to the challenge, we propose a sparse attention mechanism on LSTM that enforces the predictor to focus on the most important embedding automatically.
挑战2：如何利用专家激活的稀疏交叉嵌入依赖来设计一个轻量级的离线训练哈希函数？考虑到系统的推理效率和GPU内存消耗，哈希函数必须是一个轻量级的预测器。然而，简单的预测器很难捕捉序列的上下文信息，并且容易被分散注意力。因此，强制预测器专注于关键信息变得至关重要。我们通过实验证明，存在专家激活的稀疏交叉嵌入依赖，即序列中有限数量的嵌入共同影响专家激活。这种稀疏交叉嵌入依赖为轻量级预测器的成功提供了启示。然而，排除所有可能的结果以找到每个令牌的交叉嵌入依赖是不切实际且低效的。为了应对这一挑战，我们提出了一种在LSTM上的稀疏注意力机制，自动强制预测器专注于最重要的嵌入。

Challenge 3: How to improve the expert selection accuracy and approximate the scaling factor simultaneously? The hash function needs to determine not only the expert activation but also the scaling factor $\alpha$ in Eq. 1. As the scaling factor is derived from the SoftMax logits output from the model, it is natural to apply knowledge distillation (KD), setting the router functions as teacher models and the hash function as the student model. However, it is impossible for the hash function to approximate the scaling factor distribution over all experts by KD due to the limited capacity of the hash function. To solve this challenge, we propose to use a truncated knowledge distillation (TKD), where the KD loss is computed over the top-$T$ experts. However, the TKD cannot guarantee adequate prediction accuracy. We further add a cross-entropy loss to boost the prediction accuracy.
挑战3：如何同时提高专家选择的准确性和估算缩放因子？哈希函数需要确定的不仅是专家激活，还有方程1中的缩放因子。由于缩放因子是从模型输出的SoftMax logits中得出的，自然而然地应用知识蒸馏（KD），将路由函数设为教师模型，哈希函数设为学生模型。然而，由于哈希函数的容量有限，它不可能通过KD来近似所有专家上的缩放因子分布。为了解决这一挑战，我们提出使用截断知识蒸馏（TKD），其中KD损失是在前排专家上计算的。然而，TKD不能保证足够的预测准确性。我们进一步添加了交叉熵损失以提高预测准确性。

We introduce how SiDA deals with each challenge in detail in the following sections.
在接下来的章节中，我们将详细介绍SiDA如何应对每一个挑战。

SiDA proposes a data-aware solution to efficiently obtain the experts to be offloaded beforehand. Specifically, we propose to use a trained hash function that takes the sequence of embedding as input and predicts all the activated experts for each token in the sequence. SiDA, augmented by the data-aware expert activation prediction, enjoys two advantages while compromising little loss of model performance down to less than $1\%$. Firstly, the system can acquire the activation pattern of each sample beforehand and operate dynamically loading and offloading according to the GPU memory budget without interrupting the inference process. Secondly, since the hash function determines the expert activation across all the MoE layers for a sample independently of the inference, the system can build the hash function in a hash-building thread running in parallel with the inference thread. By doing this, we can remove the overhead caused by expert selection from the inference time, which boosts the throughput up to $3.93\times$.
SiDA 提出了一种数据感知的解决方案，以高效地预先获取需要卸载的专家。具体来说，我们提出使用一个训练好的哈希函数，该函数以嵌入序列为输入，并预测序列中每个令牌的所有激活专家。通过数据感知的专家激活预测增强的 SiDA，在几乎不损失模型性能的情况下（降低到小于 0），享有两大优势。首先，系统可以预先获取每个样本的激活模式，并根据 GPU 内存预算动态地加载和卸载，而不中断推理过程。其次，由于哈希函数独立于推理过程确定样本在所有 MoE 层中的专家激活，系统可以在与推理线程并行运行的哈希构建线程中构建哈希函数。通过这样做，我们可以去除推理时间中由专家选择引起的开销，从而将吞吐量提高到 <1>。

Previous works have also been proposed to improve the router function of MoE, such as the Hash layer (Roller et al., 2021) and the Base layer (Lewis et al., 2021). SiDA is orthogonal to these router functions as they can be accommodated in the hash-building thread. For MoE models with trained routers, we propose to train an LSTM as the hash function with the sparse attention boosted with our truncated knowledge distillation, detailed in the following sections.
之前的研究也提出了改进MoE路由器功能的方法，例如哈希层（Roller等，2021年）和基础层（Lewis等，2021年）。SiDA与这些路由器功能是正交的，因为它们可以被容纳在构建哈希的线程中。对于具有训练过的路由器的MoE模型，我们提议使用LSTM作为哈希函数，并通过我们的截断知识蒸馏增强稀疏注意力，详细内容将在以下部分中说明。

The hash function of SiDA is required to predict the expert to be activated and the corresponding scaling factor $\alpha$. Knowledge distillation (KD) (Hinton et al., 2015), which aims to minimize the distance of logits between the teacher and student model, should be the best training strategy for our hash function. However, the capacity of our hash function, 2-layer LSTM, is far less capable than the MoE model. The predictor cannot fully capture the behavior of logits of the router functions in the MoE model. The naiv̈e usage of KD greatly harms the performance of the system.
SiDA的哈希函数需要预测要激活的专家和相应的缩放因子 $\alpha$ 。知识蒸馏（KD）（Hinton等人，2015年），旨在最小化教师模型和学生模型之间的logits距离，应该是我们哈希函数的最佳训练策略。然而，我们的哈希函数，2层LSTM的容量远远小于MoE模型。预测器无法完全捕捉MoE模型中路由函数的logits行为。简单地使用KD极大地损害了系统的性能。

We propose Truncated KD (TKD) to tackle the challenge. Different from the traditional KD, the truncated KD only considers positions with top-$T$ SoftMax logit, which helps the hash function focus more on predicting the scaling factor for experts with a higher chance of being activated. Notably, large $T$ can provide a smooth ground truth for the hash function, while small $T$ enforces the hash function to be more focused on fewer experts. Further, we add the cross entropy loss to ensure the prediction accuracy. The training objective is $\lambda\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{TKD}}(T)$.
我们提出了截断知识蒸馏（TKD）来应对这一挑战。与传统的知识蒸馏不同，截断知识蒸馏仅考虑排名前 $T$ 的SoftMax逻辑值，这有助于哈希函数更加专注于预测有更高激活几率的专家的缩放因子。值得注意的是，较大的 $T$ 可以为哈希函数提供一个平滑的真实值，而较小的 $T$ 则迫使哈希函数更加专注于较少的专家。此外，我们增加了交叉熵损失以确保预测的准确性。训练目标是 $\lambda\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{TKD}}(T)$ 。

We report the GPU memory saving in Fig. 8. For short sentences in SST2, SiDA can achieve over $80\%$ GPU memory reduction. For samples in MRPC whose lengths are clustered between 50 and 80, the GPU memory reduction remains substantial, yielding savings of 6.28GB and 19.84GB GPU memory for Switch-base-128 and Switch-base-256, respectively. Furthermore, even when processing long paragraphs in MultiRC with lengths ranging from 200 to 500, the rate of GPU memory reduction retains over $40\%$ and $20\%$, leading to a save of 4.52GB for Switch-base-128 and 9.92GB for Switch-base-256.
我们在图8中报告了GPU内存节省情况。对于SST2中的短句子，SiDA可以实现超过 $80\%$ 的GPU内存减少。对于MRPC中长度集中在50到80之间的样本，GPU内存减少仍然显著，为Switch-base-128和Switch-base-256节省了6.28GB和19.84GB的GPU内存。此外，即使在处理长度范围从200到500的MultiRC中的长段落时，GPU内存减少的比率仍保持在 $40\%$ 和 $20\%$ 之上，为Switch-base-128和Switch-base-256节省了4.52GB和9.92GB的GPU内存。

Apart from the GPU memory saving, SiDA also achieves overwhelming efficiency in terms of throughput and latency (see Fig. 9). Specifically, SiDA exceeds the average of baselines by $2.60\times$ and $3.93\times$ on throughput for large MoE models such as Swicth-base-128 and Switch-base-256 on SST2. Even for MultiRC containing long sentences, SiDA exceeds the average throughput of baselines by $1.26\times$ on Switch-base-128 and $1.57\times$ on Switch-base-256.
除了节省GPU内存外，SiDA在吞吐量和延迟方面也实现了压倒性的效率（见图9）。具体来说，对于像Swicth-base-128和Switch-base-256这样的大型MoE模型，在SST2上，SiDA的吞吐量超过基准平均值 $2.60\times$ 和 $3.93\times$ 。即使对于包含长句子的MultiRC，SiDA在Switch-base-128上的吞吐量也超过基准平均值 $1.26\times$ ，在Switch-base-256上超过 $1.57\times$ 。

We also investigate the inference latency of SiDA and baselines (see Fig. 10). For large MOE models such as Switch-base-128 and Switch-base-256, SiDA reduces the inference latency to $25\%$ on SST2 and MRPC and to $60\%$ on MultiRC. The improvements come from our design of the hash-building thread that resolves the expert selection overhead.
我们还研究了SiDA及基准模型的推理延迟（见图10）。对于大型MOE模型，如Switch-base-128和Switch-base-256，SiDA将SST2和MRPC的推理延迟降低到 $25\%$ ，将MultiRC的推理延迟降低到 $60\%$ 。这些改进来自于我们设计的哈希构建线程，该线程解决了专家选择的开销问题。

We investigate the efficiency under different GPU memory budgets with different offloading methods on Switch-base-128 and Switch-base-256 since large MoE models are more resource-sensitive. Under a limited GPU memory budget, SiDA will offload and cache inactivated experts in a first-in-first-out manner, while all other baselines implement the model parallelism, where only layers required for inference will be kept on the GPU. The results of throughput versus GPU memory budgets are shown in Fig. 11. SiDA achieves better throughput under all GPU memory budgets across all datasets, demonstrating that SiDA employs a better offloading strategy under limited GPU memory budgets.
我们在Switch-base-128和Switch-base-256上，针对不同的卸载方法，在不同GPU内存预算下调查了大型MoE模型的效率，因为这些模型对资源更为敏感。在有限的GPU内存预算下，SiDA会以先进先出的方式卸载和缓存未激活的专家，而所有其他基准则实现了模型并行，其中只有推理所需的层会保留在GPU上。吞吐量与GPU内存预算的结果显示在图11中。SiDA在所有数据集上，在所有GPU内存预算下都实现了更好的吞吐量，这表明SiDA在有限的GPU内存预算下采用了更好的卸载策略。

We conduct the fidelity analysis to check how much performance SiDA can preserve. As Table. 3 shows, SiDA can preserve up to nearly $99\%$ accuracy leading to a performance degradation down to less than $1\%$ for Switch-base-8. For Switch-base-128, the fidelity is up to $96\%$ leading to a performance loss down to $3\%$. Our results demonstrate the superiority of SiDA, which achieves low inference latency and low GPU memory occupation with negligible loss on the model’s performance.
我们进行了保真度分析，以检查SiDA能保留多少性能。如表3所示，SiDA能够保留高达 $99\%$ 的准确率，使得Switch-base-8的性能下降降至少于 $1\%$ 。对于Switch-base-128，保真度高达 $96\%$ ，导致性能损失降至 $3\%$ 。我们的结果展示了SiDA的优越性，它在模型性能上的损失可以忽略不计，同时实现了低推理延迟和低GPU内存占用。

SiDA adopts a predictor to predict the experts to be activated for each token. We investigate the accuracy of the predictor in the hash-building thread, which we refer to as the hash hits rate. Results can be found in Table 4 where we report top-$3$ accuracy. For very long sentences, such as the MultiRC dataset, the hash hits rate can achieve over $90\%$.
SiDA采用了一个预测器来预测每个标记要激活的专家。我们在构建哈希的线程中调查了预测器的准确性，我们将其称为哈希命中率。结果见表4，我们报告了前 $3$ 准确率。对于非常长的句子，例如MultiRC数据集，哈希命中率可以达到超过 $90\%$ 。