Using Reinforcement Learning for
Load Testing of Video Games
使用强化学习进行视频游戏的负载测试

We injected two artificial “performance bugs” in CartPole and four in MsPacman. The idea behind them is simple: When the agent visits specific areas for the first time during a game, the bugs reveal themselves (simulation of heavy resource loading). A natural way of achieving this goal would have been to introduce the bugs in the source code of the game and to implement the logic to spot FPS drops in the agent accordingly. This, however, would have slowed down the training of the agent. Therefore, we chose to use a more practically sound approach, inspired by the simulation of Heavy-Weight Operation (HWO) operator for performance mutation testing (Delgado-Pérez et al., 2021): We directly assume that the agents observe the bugs when they visit the designated areas and act accordingly.
我们在 CartPole 中注入了两个人工“性能错误”，在 MsPacman 中注入了四个。它们背后的想法很简单：当代理在游戏中第一次访问特定区域时，错误就会显现出来（模拟大量资源加载）。实现这一目标的一种自然方式本应是在游戏源代码中引入错误，并相应地实现逻辑以检测代理中的 FPS 下降。然而，这样做会减慢代理的训练速度。因此，我们选择了一种更为实际的方法，受到性能变异测试中重量级操作（HWO）操作员的启发（Delgado-Pérez 等，2021 年）：我们直接假设代理在访问指定区域时观察到错误并相应地采取行动。

In CartPole, the agent can only move on the $x$ axis (i.e., left or right). When the game starts, the agent is in position $x=0$ (i.e., center of the axis) and it can change its position towards positive (by moving right) or negative (left) $x$ values. The two bugs we injected manifest when $x\in[-0.50,-0.45]$ and $x\in[0.45,0.50]$ — dashed lines in Fig. 2-(a). We use intervals rather than specific values (e.g.,-0.45) because the position of the agent is a float: if it moves to position -0.450001, we want to reward it during the training for having found the injected bug. Concerning MsPacman, we assume that a performance bug manifests when the agent enters the four gates indicated by the white arrows in Fig. 2-(b).
在 CartPole 中，代理只能沿着 $x$ 轴移动（即左或右）。游戏开始时，代理位于位置 $x=0$ （即轴的中心），它可以朝着正方向（向右移动）或负方向（向左移动）改变位置 $x$ 值。我们注入的两个错误在 $x\in[-0.50,-0.45]$ 和 $x\in[0.45,0.50]$ 时显现出来 — 图 2-(a)中的虚线。我们使用区间而不是具体值（例如，-0.45），因为代理的位置是一个浮点数：如果它移动到位置-0.450001，我们希望在训练过程中奖励它找到了注入的错误。关于 MsPacman，我们假设当代理进入图 2-(b)中白色箭头指示的四个门时，性能错误会显现出来。

As detailed in Section 3.1.4, we assess the extent to which RELINE is able to identify the bugs we injected while playing the games. To have a baseline, we compare its results with those of a RL-based agent only trained to play each of the two games (from hereon, rl-baseline), and with a random agent. Since RELINE will be trained with the goal of identifying the bugs (details follow), we expect it to adapt its behavior to not only successfully play the game, but to also exercise more often the “buggy” areas of the games.
如第 3.1.4 节所述，我们评估 RELINE 能够在玩游戏时识别我们注入的错误的程度。为了有一个基准，我们将其结果与仅训练玩两款游戏的基于 RL 的代理（以下简称 rl-baseline）以及随机代理进行比较。由于 RELINE 将被训练以识别错误（详细内容将在后文介绍），我们期望它调整其行为，不仅成功玩游戏，还要更频繁地练习游戏中的“错误”区域。

We trained the rl-baseline agent (i.e., the one only trained to learn how to play) for CartPole using the cross-entropy method (Rubinstein and Kroese, 2004) as RL model. We choose this method because, despite its simplicity, it has been shown to be effective in applications of RL to small environments such as CartPole (Lapan, 2018).
我们使用交叉熵方法（Rubinstein 和 Kroese，2004）作为 RL 模型，为 CartPole 训练了仅训练学习如何玩的 rl-baseline 代理。我们选择这种方法是因为尽管简单，但已被证明在 RL 应用于像 CartPole 这样的小环境中是有效的（Lapan，2018）。

The core of the cross-entropy method is a feedforward neural network (FNN) that takes as input the state of the game and provides as output the action to perform. The state of the game for CartPole is a vector of dimension 4 containing information about the $x$ coordinate of the pole’s center of mass, the pole’s speed, its angle with respect to the platform, and its angular speed. There are two possible actions: go right, go left. Once initialized with random weights, the agent (i.e., the FNN) starts playing while retaining the experience acquired in each episode: The experience is represented by the state, the action, and the reward obtained during each step of the episode.
交叉熵方法的核心是一个前馈神经网络（FNN），它以游戏状态作为输入，并输出要执行的动作。CartPole 的游戏状态是一个包含有关杆质心的 $x$ 坐标、杆的速度、相对于平台的角度和角速度信息的 4 维向量。有两种可能的动作：向右移动，向左移动。一旦用随机权重初始化，代理（即 FNN）开始玩游戏，同时保留每一集中获得的经验：经验由状态、动作和在每一步中获得的奖励表示。

The goal is to keep the pole in balance as long as possible or until the maximum length of an episode (that we set to 1,000 steps) is reached. The game reward function is defined so that the agent receives a +1 reward for each step it manages to keep the pole balanced. The total score achieved is also saved at the end of each episode. After $n=16$ consecutive episodes the agent stops playing, selects the $m=11$ (70%) episodes having the highest score, and uses the experience in those episodes to update the weights of the FNN ($n$ and $m$ have been set according to (Lapan, 2018)).
目标是尽可能保持杆的平衡，直到达到一个我们设定的最大长度（我们设定为 1,000 步）为止。游戏奖励函数被定义为，代理每成功保持杆平衡一步就获得+1 的奖励。在每一集结束时也会保存所获得的总分。在连续 $n=16$ 集之后，代理停止游戏，选择得分最高的 $m=11$ （70%）集，并使用这些集中的经验来更新 FNN 的权重（ $n$ 和 $m$ 已根据（Lapan, 2018）进行设置）。

Instead, we trained the rl-baseline agent for MsPacman using a Deep Q Network (DQN) (Mnih et al., 2013). In our context, a DQN is a Convolutional Neural Network (CNN) that takes as input a set of contiguous screenshots of the game (in our case 4, as done in previous works (Mnih et al., 2013; Mnih et al., 2015)) representing the state of the game and returns, for each possible action defined in the game (five in this case: go up, go right, go down, go left, do nothing), a value indicating the expected reward for the action given the current state (Q value). The multiple screenshots are needed to provide more information to the model about what is happening in the game (e.g., in which direction the agent is moving). The goal of the DQN is the same as the FNN: selecting the best action to perform to maximize the reward given the current state. Differently from the previous model, the DQN is updated not on entire episodes but by randomly batching “experience instances” among 10k steps saved during the most recent episodes. An “experience instance” is saved after each step $\tau$, and is represented by the quadruple ($s_{\tau-1},a_{\tau},s_{\tau},r_{\tau}$), where $s_{\tau-1}$ is the input state, $a_{\tau}$ is the action selected by the agent, $s_{\tau}$ is the resulting state obtained by running $a_{\tau}$ in $s_{\tau-1}$ and $r_{\tau}$ is the received reward.
相反，我们使用深度 Q 网络（DQN）（Mnih 等，2013 年）为 MsPacman 训练了 rl-baseline 代理。在我们的情境中，DQN 是一个卷积神经网络（CNN），它以游戏的一组连续截图作为输入（在我们的案例中为 4 张，与之前的作品相同（Mnih 等，2013 年；Mnih 等，2015 年）），代表游戏的状态，并为游戏中定义的每个可能动作（在本例中为五个：向上、向右、向下、向左、不做任何操作）返回一个值，指示给定当前状态时该动作的预期奖励（Q 值）。需要多个截图以向模型提供有关游戏中发生的情况的更多信息（例如，代理移动的方向）。DQN 的目标与 FNN 相同：选择最佳动作以最大化给定当前状态的奖励。与以前的模型不同，DQN 不是在整个剧集上更新，而是通过在最近剧集中保存的 10k 步中随机批处理“经验实例”来更新。 每个步骤 $\tau$ 之后保存一个“体验实例”，由四元组（ $s_{\tau-1},a_{\tau},s_{\tau},r_{\tau}$ ）表示，其中 $s_{\tau-1}$ 是输入状态， $a_{\tau}$ 是代理选择的动作， $s_{\tau}$ 是通过在 $s_{\tau-1}$ 中运行 $a_{\tau}$ 获得的结果状态， $r_{\tau}$ 是接收到的奖励。

The CNN is initialized with random weights, and the agent starts playing while retaining the experience of each step. When enough experience instances have been collected (10k in our implementation (Lapan, 2018)), the CNN starts updating at each step selecting a random batch of experience instances. The reward function for MsPacman provides a +1 reward every time the agent eats one of the dots and a 0 reward otherwise.
CNN 使用随机权重初始化，代理开始游戏并保留每个步骤的经验。当收集足够的经验实例时（在我们的实现中为 10k（Lapan，2018）），CNN 开始在每个步骤更新，选择一批随机的经验实例。MsPacman 的奖励函数在代理吃掉一个点时提供+1 的奖励，否则提供 0 的奖励。

To train RELINE to play while looking for the injected bugs, we use a simple performance reward function: In both the games, we give a reward of +50 every time the agent, during an episode, spots one of the injected artificial bugs. As previously mentioned, the bugs reveal themselves only the first time the agent visits each buggy position; this means that the performance-based reward is given at most twice for CartPole and four times for MsPacman.
为了训练 RELINE 在寻找注入的错误时进行游戏，我们使用了一个简单的性能奖励函数：在两个游戏中，每当代理在一个 episode 中发现一个注入的人工错误时，我们就给予+50 的奖励。正如之前提到的，错误只在代理第一次访问每个错误位置时显现出来；这意味着基于性能的奖励最多为 CartPole 两次，MsPacman 四次。

We compare RELINE against the two previously mentioned baselines: rl-baseline and the random agent. Both RELINE and rl-baseline have been trained for 3,200 episodes on CartPole and 1,000 on MsPacman. The different numbers are due to differences in the games and in the RL model we exploited. In both cases, we used a number of episodes sufficient for rl-baseline to learn how to play (i.e., we observed a convergence in the score achieved by the agent in the episodes).
我们将 RELINE 与前面提到的两个基线进行比较：rl-baseline 和随机代理。RELINE 和 rl-baseline 在 CartPole 上训练了 3,200 个 episode，在 MsPacman 上训练了 1,000 个 episode。不同的数字是由于游戏和我们利用的 RL 模型的差异。在两种情况下，我们使用了足够的 episode 数量让 rl-baseline 学会如何玩（即，我们观察到代理在 episode 中达到的分数收敛）。

Once trained, the agents have been run on both games for additional 1,000 episodes, storing the performance bugs they managed to identify in each episode. Since different trainings could result in models playing the game following different strategies, we repeated this process ten times. This means that we trained 10 different models for both RELINE and rl-baseline and, then, we used each of the 10 models to play additional 1,000 episodes collecting the spotted performance bugs. Similarly, we executed random agent 10 times for 1,000 episodes each. In this case, no training was needed.
一旦训练完成，代理人在两个游戏中额外运行了 1,000 个回合，存储了他们在每个回合中成功识别的性能缺陷。由于不同的训练可能导致模型按照不同的策略玩游戏，我们重复了这个过程十次。这意味着我们为 RELINE 和 rl-baseline 分别训练了 10 个不同的模型，然后使用这 10 个模型中的每一个来进行额外的 1,000 个回合，收集被发现的性能缺陷。类似地，我们对随机代理人进行了 10 次 1,000 个回合的执行。在这种情况下，不需要训练。

We report descriptive statistics (mean, median, and standard deviation) of the number of performance bugs identified in the 1,000 played episodes by the three approaches. A high number of episodes in which an approach can spot the injected bugs indicate its ability to look for performance bugs while playing the game.
我们报告了三种方法在 1,000 个播放的剧集中识别到的性能缺陷数量的描述性统计数据（均值、中位数和标准偏差）。一个方法能够发现注入的缺陷的大量剧集表明其在玩游戏时寻找性能缺陷的能力。

The training of the rl-baseline agent has been performed using the DQN model previously applied in MsPacman.
rl-baseline 代理的训练已经使用了先前在 MsPacman 中应用的 DQN 模型进行。

We use the previously mentioned PySuperTuxKart (PyS, [n.d.]) to make the agent interact with the game. For the sake of speeding up the training, the screenshots extracted from the game have been resized to 200x150 pixels and converted in grayscale before they are provided as input to the model. Moreover, as previously done for MsPacman, multiple (four) screenshots are fed to the model at each step. Thus, the representation of the state of the game provided to the model is a 4$\times$200$\times$150 tensor. The details of the model and its implementation are available in our replication package (Tufano, 2021).
我们使用了前面提到的 PySuperTuxKart（PyS，[n.d.]）来使代理与游戏互动。为了加快训练速度，从游戏中提取的屏幕截图已被调整为 200x150 像素并在提供给模型之前转换为灰度。此外，与先前为 MsPacman 所做的一样，在每个步骤中向模型提供多个（四个）屏幕截图。因此，提供给模型的游戏状态表示是一个 4 $\times$ 200 $\times$ 150 张量。模型及其实现的详细信息可在我们的复制包中找到（Tufano，2021）。

A critical part of the learning process is the definition of the game reward function. Being SuperTuxKart a racing game, an option could have been to penalize the agent for each additional step required to finish the game. Consequently, to maximize the final score, the agent would have been pushed to reduce the number of steps and, therefore, to drive as fast as possible towards the finish line. However, considering the non-trivial size of the game space, such a reward function would have required a long training time. Thus, we took advantage of the information that can be extracted from the game to help the agent in the learning process.
学习过程的关键部分是定义游戏奖励函数。作为一款赛车游戏，一个选择可能是惩罚代理完成游戏所需的每一步。因此，为了最大化最终得分，代理将被迫减少步数，因此尽可能快地驶向终点。然而，考虑到游戏空间的庞大，这样的奖励函数将需要很长的训练时间。因此，我们利用可以从游戏中提取的信息来帮助代理进行学习过程。

SuperTuxKart provides two coordinates indicating where the agent is in the game: path_done and centering.
SuperTuxKart 提供了两个坐标，指示代理在游戏中的位置：path_done 和 centering。

The former indicates the path traversed by the agent from the starting line of the track, while the latter represents the distance of the agent from the center of the track. In particular, centering equals 0 if the agent is at the center of the track, and it moves away from zero as the agent moves to either side: going towards right results in positive values of the centering value, going left in negative values. We indicate these coordinates with $x$ (centering) and $y$ (path_done), and we define $\delta_{y}$ as the path traversed by the agent in a specific step: Given $y_{i}$ the value for path_done at step $i$, we compute $\delta_{y}$ as $y_{i}-y_{i-1}$. Basically, $\delta_{y}$ measures how fast the agent is advancing towards the finish line.
前者表示代理从赛道起点走过的路径，而后者表示代理距离赛道中心的距离。特别地，如果代理位于赛道中心，则居中等于 0，随着代理向任一侧移动，它远离零：向右移动导致居中值为正值，向左移动导致负值。我们用 $x$ （居中）和 $y$ （路径完成）表示这些坐标，并定义 $\delta_{y}$ 为代理在特定步骤中走过的路径：给定 $y_{i}$ 为步骤 $i$ 中路径完成的值，我们计算 $\delta_{y}$ 为 $y_{i}-y_{i-1}$ 。基本上， $\delta_{y}$ 衡量代理朝着终点线前进的速度。

Given $x$ and $\delta_{y}$ for a given step $i$, we compute the reward function as follows:
给定特定步骤 $i$ 的 $x$ 和 $\delta_{y}$ ，我们按以下方式计算奖励函数：

First, if the agent goes too far from the center of the track ($|x|>\theta$), we penalize it with a negative reward. Otherwise, if the agent is close to the center ($|x|\leq\theta$), we can have two scenarios: (i) if it is not moving towards the finish line ($\delta_{y}\leq 0$), we do not give any reward (the minimum reward is 0); (ii) if it is moving in the right direction ($\delta_{y}>0$), we give a reward proportional to the speed at which it is advancing ($\delta_{y}$), up to a maximum of M.
首先，如果代理人离赛道中心太远（ $|x|>\theta$ ），我们会用负奖励来惩罚它。否则，如果代理人靠近中心（ $|x|\leq\theta$ ），我们可以有两种情况：（i）如果它没有朝着终点线移动（ $\delta_{y}\leq 0$ ），我们不给予任何奖励（最低奖励为 0）；（ii）如果它朝着正确的方向移动（ $\delta_{y}>0$ ），我们会根据其前进速度给予奖励（ $\delta_{y}$ ），最高不超过 M。

In our experimental setup, we set $\theta=20$ because it roughly represents the double of $|x|$ when the agent approaches the sides of the road in the level, and $M=10$ as it is the same maximum reward also given by the performance reward function, as we explain below. Finally, we reward the agent when it crosses the finish line with an additional $+1,000$ bonus.
在我们的实验设置中，当代理人靠近道路边缘时，我们设置 $\theta=20$ ，因为这大致代表了 $|x|$ 的两倍，同时 $M=10$ 也是相同的最大奖励，这也是由性能奖励函数给出的，我们将在下文中解释。最后，当代理人越过终点线时，我们会额外奖励 $+1,000$ 。

To define the performance reward function of RELINE for SuperTuxKart, the first step to perform is to define a way to reliably capture the FPS of the game during the training. In this way, we can reward the agent when it manages to identify low-FPS points. As previously said, we use PySuperTuxKart to interact with the game and such a framework keeps the game frozen while the other instructions of RELINE (e.g., the identification of the action to execute) are run. Since the framework runs the game in the same process in which we run RELINE and since we do not use threads, we can safely use a simple method for computing the time needed to render the four frames: We get the system time before ($T_{\mathit{before}}$) and after ($T_{\mathit{after}}$) we trigger the rendering of the frames and we compute the time needed at step $i$ as $\mathit{rT}_{i}=T_{\mathit{after}}-T_{\mathit{before}}$. Such a value is negatively correlated with the FPS (higher rendering time means lower FPS).
要为 SuperTuxKart 定义 RELINE 的性能奖励函数，执行的第一步是定义一种可靠地捕获游戏 FPS 的方法。通过这种方式，当代理设法识别出低 FPS 点时，我们可以奖励代理。正如之前所说，我们使用 PySuperTuxKart 与游戏交互，这样一个框架会在 RELINE 的其他指令（例如执行的动作的识别）运行时保持游戏冻结。由于框架在我们运行 RELINE 的同一进程中运行游戏，且我们不使用线程，因此我们可以安全地使用一种简单的方法来计算渲染四帧所需的时间：我们在触发帧渲染之前（ $T_{\mathit{before}}$ ）和之后（ $T_{\mathit{after}}$ ）获取系统时间，并在步骤 $i$ 计算所需的时间为 $\mathit{rT}_{i}=T_{\mathit{after}}-T_{\mathit{before}}$ 。这样的值与 FPS 呈负相关（更长的渲染时间意味着更低的 FPS）。

The performance reward function we use is the following:
我们使用的性能奖励函数如下：

We give a performance-based reward of 10 when the agent takes more than $t$ milliseconds to render the frames at a given point (causing an FPS drop). We explain the tuning of $t$ later. We do not give such a reward when $|x|>\theta$ (the kart is far from the center) since we want the agent to spot issues in positions that are likely to be explored by real players (i.e., reasonably close to the track).
当代理在给定点渲染帧时花费超过 $t$ 毫秒（导致 FPS 下降）时，我们会给予基于表现的奖励 10。我们稍后会解释 $t$ 的调整。当 $|x|>\theta$ （卡丁车远离中心）时，我们不会给予这样的奖励，因为我们希望代理能够发现那些可能被真实玩家探索的位置上的问题（即，距离赛道相对较近）。

Finally, in RELINE we do not give a fixed $+1,000$ bonus reward when the agent crosses the finish line but we assign a bonus computed as $10\times\sum_{i=1}^{\mathit{steps}}rp_{i}$, i.e., proportional to the total performance-based reward accumulated by the agent in the episode. This is done to push the agent to visit more low-FPS points during an episode.
最后，在 RELINE 中，当代理越过终点线时，我们不会给予固定的 $+1,000$ 奖励，而是分配一个根据 $10\times\sum_{i=1}^{\mathit{steps}}rp_{i}$ 计算的奖励，即，与代理在该回合中累积的基于表现的奖励总数成比例。这样做是为了促使代理在一个回合中访问更多低 FPS 点。

As done in our preliminary study, we compare RELINE with rl-baseline (i.e., the agent only trained to play the game) and with a random agent.
正如我们在初步研究中所做的那样，我们将 RELINE 与 rl-baseline（即，仅训练玩游戏的代理）和随机代理进行比较。

Training rl-baseline and RELINE. While we used different reward functions for the two RL agents, we applied the same training process for both of them. We trained each model for 2,300 episodes, with one episode having a maximum duration of 90 seconds or ending when the agent crosses the finish line of the racing track (the agent is required to perform a single lap). We set the 90 seconds limit since we observed that, by manually playing the game, $\sim$70 seconds are sufficient to complete a lap. The 2,300 episodes threshold has been defined by computing the average reward obtained by the two agents every 100 episodes and by observing when a plateau was reached by both agents. We found 2,300 episodes to be a good compromise for both agents (graphs plotting the reward function are available in the replication package (Tufano, 2021)).
训练 rl-baseline 和 RELINE。虽然我们为这两个 RL 代理使用了不同的奖励函数，但我们对它们都应用了相同的训练过程。我们为每个模型训练了 2,300 个剧集，每个剧集的最长持续时间为 90 秒，或者当代理越过赛车赛道的终点线时结束（代理需要完成一圈）。我们设置了 90 秒的限制，因为我们观察到，通过手动玩游戏， $\sim$ 70 秒就足以完成一圈。通过计算每 100 个剧集两个代理获得的平均奖励，并观察两个代理何时达到平稳状态，我们发现 2,300 个剧集对于两个代理都是一个很好的折衷点（绘制奖励函数的图表可在复制包（Tufano, 2021）中找到）。

The trained rl-baseline agent has been used to define the threshold $t$ needed for the RELINE’s reward function (i.e., for identifying when the agent found a low-FPS point and should be rewarded).
训练过的 rl-baseline 代理已被用来定义 RELINE 奖励函数所需的阈值 $t$ （即，用于确定代理何时发现低 FPS 点并应获得奖励）。

In particular, once trained, we run rl-baseline for 300 episodes, storing the time needed by the game to render the subsequent four frames after every action recommended by the model.²²2Since we wanted to measure the frames rendering time in a standard scenario in which the agent was driving the kart, we stopped an episode if the agent got stuck against some obstacle.
由于我们想要在标准场景中测量帧渲染时间，其中代理正在驾驶卡丁车，如果代理卡在某个障碍物上，我们就会停止一个情节。
具体来说，一旦训练完成，我们运行 rl-baseline 进行 300 个情节，存储游戏在模型推荐的每个动作后渲染后续四帧所需的时间。 ² This resulted in a total of 48,825 data points $s_{FPS}$, representing the standard FPS of the game in a scenario in which the player is only focused on completing the race as fast as possible.
这导致了总共 48,825 个数据点 $s_{FPS}$ ，代表了游戏的标准 FPS，在这种情况下，玩家只关注尽快完成比赛。

Starting from the 48,825 $s_{FPS}$ data points collected in the 300 episodes played by the trained rl-baseline agent, we apply the five-$\sigma$ rule (Grafarend, 2006) to compute a threshold able to identify outliers. The five-$\sigma$ rule states that in a normal distribution (such as $s_{FPS}$) 99.99% of observed data points lie within five standard deviations from the mean. Thus, anything above this value can be considered as an outlier in terms of milliseconds needed to render the frames. For this reason, we compute $t_{b}=mean(s_{FPS})+5\times sd(s_{FPS})$ as a candidate base threshold to identify low-FPS points. However, $t_{b}$ cannot be directly used as the $t$ value of our reward function. Indeed, we observed that the time needed for rendering frames during the RELINE’s training is slightly higher as compared to the time needed when the trained rl-baseline agent is used to play the game. This is due to the fact that the load on the server (and in particular on the GPU) is higher during training. To overcome this issue, we perform the following steps.
从训练过的 rl-baseline 代理玩的 300 集中收集的 48,825 个数据点开始，我们应用五个标准差准则（Grafarend, 2006）来计算一个能够识别异常值的阈值。五个标准差准则规定，在正态分布（如 $s_{FPS}$ ）中，观察到的数据点中有 99.99%位于距离均值五个标准差之内。因此，任何超过这个值的数据点都可以被视为在渲染帧所需的毫秒数方面的异常值。基于此，我们计算 $t_{b}=mean(s_{FPS})+5\times sd(s_{FPS})$ 作为识别低 FPS 点的候选基准阈值。然而， $t_{b}$ 不能直接用作我们奖励函数的 $t$ 值。事实上，我们观察到，在 RELIN 的训练期间，渲染帧所需的时间略高于使用训练过的 rl-baseline 代理玩游戏时所需的时间。这是因为服务器（尤其是 GPU）的负载在训练期间更高。为了解决这个问题，我们执行以下步骤。

At the beginning of the training, we run 100 warmup episodes in which we collect the time needed to render the four frames after each action performed by the agent. Then, we compute the first ($Q^{\mathit{tr}}_{1}$) and the third ($Q^{\mathit{tr}}_{3}$) quartile of the obtained distribution and compare them to the $Q_{1}$ and $Q_{3}$ of the distribution obtained in the 300 episodes used to define $t_{b}$ (i.e., those played by the trained rl-baseline agent). During the warmup episodes, the agent selects the action to perform almost randomly (it still has to learn): Therefore, it would not be able to explore a substantial area of the game (i.e., of the racing track), thus not providing a distribution of timings comparable with the ones obtained when the trained rl-baseline agent that played the 300 episodes. For this reason, during the 100 warmup episodes of the training, the action to perform is not chosen by the agent currently under training, but by the trained rl-baseline agent (i.e., the same used in the 300 episodes). This does not impact in any way the load on the server that remains the one we have during the training of RELINE since the only change we have is to ask for the action to perform to the rl-baseline agent rather than to the one under training. However, the whole training procedure (e.g., capturing the frames and updating the network) stays the same.
在培训开始时，我们进行了 100 个热身阶段，在这些阶段中，我们收集了代理执行每个动作后渲染四帧所需的时间。然后，我们计算所得分布的第一（ $Q^{\mathit{tr}}_{1}$ ）和第三（ $Q^{\mathit{tr}}_{3}$ ）四分位数，并将其与用于定义 $t_{b}$ 的 300 个阶段中所得分布的 $Q_{1}$ 和 $Q_{3}$ 进行比较（即由训练过的 rl-baseline 代理播放的那些阶段）。在热身阶段，代理几乎是随机选择要执行的动作（它仍然需要学习）：因此，它将无法探索游戏的大部分区域（即赛道），因此无法提供与训练过的 rl-baseline 代理播放 300 个阶段时所获得的时间分布相媲美的分布。因此，在培训的 100 个热身阶段中，要执行的动作不是由当前正在接受培训的代理选择的，而是由训练过的 rl-baseline 代理选择的（即在 300 个阶段中使用的相同代理）。 这并不会对服务器的负载产生任何影响，因为在训练 RELINE 期间，服务器的负载保持不变，因为我们唯一的变化是要求 rl-baseline 代理执行操作，而不是正在训练的代理。然而，整个训练过程（例如，捕获帧和更新网络）保持不变。

We compute the additional “cost” brought by the training in rendering the frames during the game using the formula $\delta=max(Q^{\mathit{tr}}_{1}-Q_{1},Q^{\mathit{tr}}_{3}-Q_{3})$. We use the first and third quartiles since they represent the boundaries of the central part of the distribution, i.e., they should be quite representative of the values in it. We took as $\delta$ the maximum of the two differences to be more conservative in assigning rewards when the agent identifies low-FPS points. The final value $t$ we use in our reward function when training RELINE to load test SuperTuxKart is defined as: $t=t_{b}+\delta=18.36$.³³3We identify as low-FPS points the ones in which the FPS is lower than 218. Such a number is still very high, more than enough for any human player, in practice. Note that we run the game using high-performance hardware and, most importantly, with the lowest graphic settings. The equivalent in normal conditions would be much lower.
我们将 FPS 低于 218 的点标识为低 FPS 点。这个数字仍然非常高，对于任何人类玩家来说都足够了。请注意，我们使用高性能硬件运行游戏，而且最重要的是，使用最低的图形设置。在正常条件下，相应的数值会低得多。
我们通过使用公式 $\delta=max(Q^{\mathit{tr}}_{1}-Q_{1},Q^{\mathit{tr}}_{3}-Q_{3})$ 来计算在游戏过程中训练带来的额外“成本”，用于渲染帧。我们使用第一和第三四分位数，因为它们代表分布中心部分的边界，即它们应该相当代表其中的值。我们将两个差异的最大值作为 $\delta$ ，以便在代理识别低 FPS 点时更加保守地分配奖励。我们在训练 RELINE 以加载测试 SuperTuxKart 时在我们的奖励函数中使用的最终值 $t$ 定义为： $t=t_{b}+\delta=18.36$ 。 ³

Thus, if RELINE is able, during the training, to identify a point in the game requiring more than $t$ milliseconds to render four frames, then it receives a reward as explained in Section 4.1.2.
因此，如果 RELINE 能够在训练期间识别出游戏中需要超过 $t$ 毫秒来渲染四帧的点，那么它将根据第 4.1.2 节中的说明获得奖励。

The training of rl-baseline took $\sim$3 hours, while RELINE requires substantially more time due to the fact that, after each step performed by the agent, we collect and store information about the time needed to render the frames (this is done million of times). This pushed the training for RELINE up to $\sim$30 hours.
rl-baseline 的训练花费了 $\sim$ 3 小时，而 RELINE 需要更多的时间，因为在代理执行每一步之后，我们会收集并存储有关渲染帧所需时间的信息（这样做了数百万次）。这使得 RELINE 的训练时间延长到 $\sim$ 30 小时。

Reliability of Time Measurements. It is important to clarify that the FPS of the game can be impacted by the hardware specifications and the current load of the machine running it. In other words, running the same game on two different machines or on the same machine in two different moments can result in variations of the FPS. For this reason, all the experiments have been performed on the same server, equipped with 2 x 64 Core AMD 2.25GHz CPUs, 512GB DDR4 3200MHz RAM, and an nVidia Tesla V100S 32GB GPU. Also, the process running the training of the agents or the collection of the 48,825 $s_{FPS}$ with the trained rl-baseline agent was the only process running on the machine besides those handled by the operating system (Ubuntu 20.04). On top of that, the process was always run using the chrt --rr 1 option, that in Linux maximizes the priority of the process, reducing the likelihood of interruptions.
时间测量的可靠性。重要的是要澄清，游戏的 FPS 可能会受到硬件规格和运行游戏的机器当前负载的影响。换句话说，在两台不同的机器上运行相同的游戏，或者在同一台机器上的两个不同时刻运行游戏，可能会导致 FPS 的变化。因此，所有实验都是在同一台服务器上进行的，该服务器配备有 2 个 64 核 AMD 2.25GHz CPU、512GB DDR4 3200MHz RAM 和一块 nVidia Tesla V100S 32GB GPU。此外，用于训练代理或收集 48,825 个 $s_{FPS}$ 的过程与经过训练的 rl-baseline 代理是除了由操作系统（Ubuntu 20.04）处理的进程之外，机器上唯一运行的进程。此外，该进程始终使用 chrt --rr 1 选项运行，该选项在 Linux 中最大化了进程的优先级，减少了中断的可能性。

Despite these precautions, it is still possible that variations are observed in the FPS not due to issues in the game, but to external factors (e.g., changes in the load of the machine). To verify the reliability of the collected FPS data, we run a constant agent performing always the same actions in the game for 300 episodes. The set of actions has been extracted from one of the episodes played by the rl-baseline agent, that was able to successfully conclude the race. Then, we plotted the time needed by the game to render the four frames following each action made by the agent. Since we are playing 300 times exactly the same episode, we expect to observe the same trend in terms of FPS for each game. If this is the case, it means that the way we are measuring the FPS is reliable enough to reward the agent when low-FPS points are identified.
尽管采取了这些预防措施，仍然有可能观察到 FPS 的变化，这并不是由于游戏本身的问题，而是由于外部因素（例如，机器负载的变化）。为了验证收集的 FPS 数据的可靠性，我们运行一个恒定的代理程序，在游戏中执行相同的动作 300 次。这组动作是从 rl-baseline 代理程序播放的一个剧集中提取出来的，该代理程序能够成功完成比赛。然后，我们绘制了游戏在代理程序执行每个动作后渲染四帧所需的时间。由于我们完全相同地播放了 300 次剧集，我们期望观察到每场比赛的 FPS 趋势相同。如果是这种情况，这意味着我们测量 FPS 的方式足够可靠，可以在识别到低 FPS 点时奖励代理程序。

Fig. 3 shows the achieved results: The $y$-axis represents the milliseconds needed to render four frames in response to an agent’s action ($x$-axis) performed in a specific part of the game. While, as expected, small variations are possible, the overall trend is quite stable: Points of the game requiring longer time to render frames are consistently showing across the 300 episodes, resulting in a clear trend. We also computed the Spearman’s correlation (Spearman, 1904) pairwise across the 300 distributions, adjusting the obtained $p$-values using the Holm’s correction (Holm, 1979).
图 3 显示了实现的结果： $y$ 轴表示渲染四帧所需的毫秒数，以响应代理的动作（ $x$ 轴）在游戏的特定部分执行。尽管可能会有一些小的变化，但总体趋势相当稳定：需要更长时间来渲染帧的游戏点在 300 个剧集中持续显示，呈现出明显的趋势。我们还计算了斯皮尔曼相关系数（Spearman, 1904），在 300 个分布中成对进行调整，使用 Holm's 校正（Holm, 1979）来调整获得的 $p$ 值。

We found all correlations to be statistically significant (adjusted $p$-values $<$ 0.05) with a minimum $\rho$=0.77 (strong correlation) and a median $\rho$=0.91 (very strong correlation). This confirms the common FPS trends across the 300 episodes.
我们发现所有相关性都具有统计学意义（调整后的 $p$ 值 $<$ 0.05），最小 $\rho$ =0.77（强相关）和中位数 $\rho$ =0.91（非常强相关）。这证实了在 300 个剧集中存在的常见 FPS 趋势。

Running the Three Techniques to Spot Low-FPS Areas. After the 2,300 training episodes, we assume that both the RL-based agents learned how to play the game, and that RELINE also learned how to spot low-FPS points. Then, as also done in our preliminary study, we train both agents for additional 1,000 episodes, storing the time needed to render the frames in every single point they explored during each episode (where a point is represented by its coordinates, i.e., centering=$x$ and path_done=$y$). We do the same also with the random agent.
运行三种技术来发现低 FPS 区域。在进行了 2,300 个训练周期后，我们假设基于 RL 的代理程序都学会了如何玩游戏，而 RELINE 也学会了如何发现低 FPS 点。然后，与我们的初步研究中一样，我们为两个代理程序额外进行了 1,000 个周期的训练，存储了在每个周期内它们探索的每个点渲染帧所需的时间（其中一个点由其坐标表示，即 centering= $x$ 和 path_done= $y$ ）。我们还对随机代理程序执行相同操作。

Data Analysis. The output of each of the three agents is a list of points with the milliseconds each of them required to render the subsequent frames. Since each agent played 1,000 episodes, it is possible that the same point is covered several times by an agent, with slightly different FPS observed (as previously explained, small variations in FPS are possible and expected across different episodes). We classify as low-FPS points those that required more than $t$ milliseconds to render the four subsequent frames more than 50% of times they have been covered by an agent.
数据分析。三个代理程序的输出是一个点列表，其中包含它们分别渲染后续帧所需的毫秒数。由于每个代理程序进行了 1,000 个周期的游戏，可能同一个点会被一个代理程序多次覆盖，观察到稍有不同的 FPS（如前所述，FPS 的小变化在不同周期中是可能且预期的）。我们将那些在被代理程序覆盖超过 50%的时间内需要超过 $t$ 毫秒来渲染四个后续帧的点分类为低 FPS 点。

This means that, if across the 1,000 episodes a point $p$ is exercised 100 times by an agent, at least 51 times the threshold $t$ must be exceeded to consider $p$ as a low-FPS point. In practice, a developer using RELINE for identifying low-FPS points could use a higher threshold to increase the reliability of the findings. However, for the sake of this empirical study, we decided to be conservative.
这意味着，如果在 1,000 个情节中，代理人对一个点 $p$ 进行了 100 次操作，那么至少有 51 次必须超过阈值 $t$ 才能将 $p$ 视为低 FPS 点。在实践中，使用 RELINE 来识别低 FPS 点的开发人员可以使用更高的阈值来提高发现的可靠性。然而，为了这项经验研究的缘故，我们决定保守一些。

Then, we compare the characteristics of the low-FPS points identified by the three approaches. Specifically, we analyze: (i) how many different low-FPS points each approach identified; (ii) the number of times each low-FPS point has been exercised by each agent in the 1,000 episodes; (iii) the confidence of the identified points (i.e., the percentage of times an exercised point resulted in low FPS). Given the low-FPS points identified by each agent, we draw violin plots showing the distribution of timings needed to render the frames when the agent exercised them (the higher the timings, the lower the FPS). We compare these distributions using Mann-Whitney test (Conover, 1998) with $p$-values adjustment using the Holm’s correction (Holm, 1979). We also estimate the magnitude of the differences by using the Cliff’s Delta ($d$), a non-parametric effect size measure (Grissom and Kim, 2005) for ordinal data. We follow well-established guidelines to interpret the effect size: negligible for $|d|<0.10$, small for $0.10\leq|d|<0.33$, medium for $0.33\leq|d|<0.474$, and large for $|d|\geq 0.474$ (Grissom and Kim, 2005).
然后，我们比较了三种方法识别出的低 FPS 点的特征。具体来说，我们分析：(i) 每种方法识别出了多少个不同的低 FPS 点；(ii) 每个低 FPS 点在 1,000 个 episode 中被每个代理练习的次数；(iii) 识别点的置信度（即练习点导致低 FPS 的百分比）。鉴于每个代理识别出的低 FPS 点，我们绘制小提琴图，显示代理在练习时渲染帧所需的时间分布（时间越长，FPS 越低）。我们使用 Mann-Whitney 检验（Conover, 1998）比较这些分布，使用 Holm's 校正（Holm, 1979）进行 $p$ 值调整。我们还通过使用 Cliff's Delta（ $d$ ），一种用于序数数据的非参数效应大小测量方法（Grissom 和 Kim, 2005）来估计差异的大小。我们遵循既定的指导方针来解释效应大小： $|d|<0.10$ 为微不足道， $0.10\leq|d|<0.33$ 为小， $0.33\leq|d|<0.474$ 为中等， $|d|\geq 0.474$ 为大（Grissom 和 Kim, 2005）。