GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians
GaussianAvatars：带有 Rigged 3D Gaussian 的逼真头部头像

NeRF [29] stores the radiance field of a scene in a neural network and provides photorealistic renderings of novel views with volumetric rendering. Later work [49, 39] represents the scene as voxel grids, achieving comparable rendering quality in a shorter time. Efficiency can be further improved by employing compression methods such as voxel hashing [30] or tensor decomposition [6]. PointNeRF [45] uses point clouds as a scene representation, whereas 3D Gaussian Splatting [14] uses anisotropic 3D Gaussians that are rendered by sorting and rasterization, achieving superior visual quality with real-time rendering. Mixture of Volumetric Primitives [26] uses surface-aligned volumes to achieve fast rendering with high visual fidelity. Our method follows 3D Gaussian Splatting [14], benefiting from the expressiveness of anisotropic Gaussians.
NeRF [29] 将场景的辐射场存储在神经网络中，并通过体积渲染提供新颖视图的真实感渲染。后来的工作 [49, 39] 将场景表示为体素网格，在更短的时间内实现了可比的渲染质量。通过采用体素散列[30]或张量分解[6]等压缩方法可以进一步提高效率。 PointNeRF [45] 使用点云作为场景表示，而 3D Gaussian Splatting [14] 使用通过排序和光栅化渲染的各向异性 3D 高斯，通过实时渲染实现卓越的视觉质量。 Mixture of Volumetric Primitives [26] 使用表面对齐的体积来实现具有高视觉保真度的快速渲染。我们的方法遵循 3D 高斯分布 [14]，受益于各向异性高斯的表现力。

A common paradigm to model a dynamic scene is directly storing it in an MLP with 4D coordinates as the input [9, 44] or a 4D tensor with the time dimension or spatial dimensions compressed [38, 7, 4, 1]. These methods can faithfully replay a dynamic scene and produce realistic novel-view rendering but lack an explicit handle to manipulate the content. Another paradigm learns a static canonical space and the maps time steps back to the canonical space via separate MLPs [35, 31, 32]. Instead of using a deformation MLP, a proxy geometry provides more direct controllability. Liu et al. [25] warp points from an observed space to the canonical space based on the movement of the nearest triangle on an SMPL [27] mesh. Peng et al. [34] deform points with the skeleton of SMPL and neural blending weights. Concurrent works create human body avatars with forward deformation [23, 20, 48, 13, 18] and cage-based deformation [54]. Unlike these methods, we directly attach 3D Gaussians to triangles and explicitly move them, obviating the need for a canonical space and enabling effective mesh finetuning.
动态场景建模的常见范例是将其直接存储在 MLP 中，并以 4D 坐标作为输入 [9, 44] 或时间维度或空间维度压缩的 4D 张量 [38, 7, 4, 1]。这些方法可以忠实地重播动态场景并产生逼真的小说视图渲染，但缺乏操作内容的显式句柄。另一种范式学习静态规范空间，并通过单独的 MLP 将时间步映射回规范空间 [35,31,32]。代理几何体不使用变形 MLP，而是提供更直接的可控性。刘等人。 [25] 基于 SMPL [27] 网格上最近三角形的移动将扭曲点从观察空间扭曲到规范空间。彭等人。 [34]使用 SMPL 骨架和神经混合权重对点进行变形。同时进行的工作创建了具有前向变形[23,20,48,13,18]和基于笼子的变形[54]的人体化身。与这些方法不同，我们直接将 3D 高斯函数附加到三角形并显式移动它们，从而无需规范空间并实现有效的网格微调。

Head avatar creation advanced through the uptake of differentiable rendering and scene representations. Thies et al. [41] instigated a shift toward digital avatars with real-time face tracking and authentic face reenactment. Advancements in image synthesis with neural networks [42, 5] boosted the controllability of head avatars from lip-syncing to expression transfer and head motions [40, 15, 47]. Gafni et al. [8] learn a NeRF conditioned on an expression vector from monocular videos. Grassal et al. [10] subdivide and add offsets to FLAME [21] to enhance its geometry and enable a dynamic texture via an expression-dependent texture field. IMavatar [51] learns a 3D morphable head avatar with neural implicit functions, solving for a map from observed to canonical space via iterative root-finding. HeadNeRF [11] learns a NeRF-based parametric head model with 2D neural rendering for efficiency.
通过采用可微分渲染和场景表示，头部头像的创建取得了进步。蒂斯等人。 [41]通过实时面部跟踪和真实的面部重演，推动了向数字化身的转变。神经网络图像合成的进步 [42, 5] 提高了头部化身的可控性，从口型同步到表情转移和头部运动 [40, 15, 47]。加夫尼等人。 [8] 从单目视频中学习以表达向量为条件的 NeRF。格拉萨尔等人。 [10] 对 FLAME [21] 进行细分和添加偏移，以增强其几何形状并通过依赖于表达式的纹理字段启用动态纹理。 IMavatar [51] 使用神经隐式函数学习 3D 可变形头部化身，通过迭代寻根求解从观察空间到规范空间的映射。 HeadNeRF [11] 通过 2D 神经渲染学习基于 NeRF 的参数化头部模型，以提高效率。

INSTA [55] deforms query points to a canonical space by finding the nearest triangle on a FLAME [21] mesh, and combines this with InstantNGP [30] to achieve fast rendering. Like INSTA, we also use triangles to warp the scene, but we build a consistent correspondence between a 3D Gaussian and a triangle instead of querying the nearest one for each timestep. Zheng et al. [52] explore point-based representations with differential point splatting. They define a point set in canonical space and learn a deformation field conditioned on FLAME’s expression vectors to animate the head avatar. While the scale of a point has to be manually set for their method, it is an optimizable parameter for 3D Gaussians. AvatarMAV [46] defines a canonical radiance field and a motion field, both with voxel grids. To model deformations, a set of voxel grid motion bases are blended according to an input 3DMM expression vector [3, 33].
INSTA [55] 通过在 FLAME [21] 网格上查找最近的三角形将查询点变形为规范空间，并将其与 InstantNGP [30] 相结合以实现快速渲染。与 INSTA 一样，我们也使用三角形来扭曲场景，但我们在 3D 高斯和三角形之间建立一致的对应关系，而不是为每个时间步查询最近的对应关系。郑等人。 [52]探索基于点的微分点表示。他们在规范空间中定义一个点集，并学习以 FLAME 表达向量为条件的变形场，以对头部头像进行动画处理。虽然必须为其方法手动设置点的比例，但它是 3D 高斯函数的可优化参数。 AvatarMAV [46] 定义了一个规范的辐射场和一个运动场，两者都带有体素网格。为了对变形进行建模，根据输入 3DMM 表达向量 [3, 33] 混合一组体素网格运动基础。

3D Gaussian Splatting [14] provides a solution to reconstruct a static scene with anisotropic 3D Gaussians given images and camera parameters. A scene is represented by a set of Gaussian splats, each defined by a covariance matrix $\Sigma$ centered at point (mean) $\bm{\mu}$:
3D Gaussian Splatting [14] 提供了一种在给定图像和相机参数的情况下使用各向异性 3D 高斯重建静态场景的解决方案。场景由一组高斯图表示，每个图由以点（平均值） $\bm{\mu}$ 为中心的协方差矩阵 $\Sigma$ 定义：

$$G(\bm{x})=e^{-\frac{1}{2}(\bm{x}-\bm{\mu})^{T}\Sigma^{-1}(\bm{x}-\bm{\mu})}$$

(1)

Note that covariance matrices have physical meaning only when they are semi-definite, which cannot be guaranteed for an optimization process with gradient descent. Therefore, Kerbl et al. [14] first define a parametric ellipse with a scaling matrix $S$ and a rotation matrix $R$, then construct the covariance matrix by:
请注意，协方差矩阵仅在半定时才具有物理意义，这不能保证梯度下降的优化过程。因此，Kerbl 等人。 [14] 首先定义一个带有缩放矩阵 $S$ 和旋转矩阵 $R$ 的参数椭圆，然后通过以下方式构造协方差矩阵：

$$\Sigma=RSS^{T}R^{T}.$$

(2)

Practically, an ellipse is stored as a position vector $\bm{\mu}\in\mathbb{R}^{3}$, a scaling vector $\bm{s}\in\mathbb{R}^{3}$, and a quaternion $\bm{q}\in\mathbb{R}^{4}$. In our paper, we use $\bm{r}\in\mathbb{R}^{3\times 3}$ to notate the corresponding rotation matrix to $\bm{q}$.
实际上，椭圆存储为位置向量 $\bm{\mu}\in\mathbb{R}^{3}$ 、缩放向量 $\bm{s}\in\mathbb{R}^{3}$ 和四元数 $\bm{q}\in\mathbb{R}^{4}$ 。在我们的论文中，我们使用 $\bm{r}\in\mathbb{R}^{3\times 3}$ 来表示 $\bm{q}$ 对应的旋转矩阵。

For rendering, the color $\bm{C}$ of a pixel is computed by blending all 3D Gaussians overlapping the pixel:
对于渲染，像素的颜色 $\bm{C}$ 通过混合与像素重叠的所有 3D 高斯函数来计算：

$$\bm{C}=\sum_{i=1}{\bm{c}_{i}\alpha_{i}^{\prime}\prod_{j=1}^{i-1}}(1-\alpha_{j}^{\prime}),$$

(3)

where $\bm{c}_{i}$ is the color of each point modeled by 3-degree spherical harmonics. The blending weight $\alpha^{\prime}$ is given by evaluating the 2D projection of the 3D Gaussian multiplied by a per-point opacity $\alpha$. The Gaussian splats are sorted by depth before blending to respect visibility order.
其中 $\bm{c}_{i}$ 是通过 3 度球谐函数建模的每个点的颜色。混合权重 $\alpha^{\prime}$ 通过评估 3D 高斯的 2D 投影乘以每点不透明度 $\alpha$ 来给出。高斯图块在混合之前按深度排序以尊重可见性顺序。

The key component of our method is how we build connections between the FLAME [21] mesh and 3D Gaussian splats. Initially, we pair each triangle of the mesh with a 3D Gaussian and let the 3D Gaussian move with the triangle across time steps. In other words, the 3D Gaussian is static in the local space of its parent triangle but dynamic in the global (metric) space as the triangle moves. Given the three vertices of a triangle, we take their mean position $\bm{T}$ as the origin of the local space. Then, we concatenate the direction vector of one edge, the normal vector of the triangle, and their cross product as column vectors to form a rotation matrix $\bm{R}$, which describes the orientation of the triangle in the global space. We also compute a scalar $k$ by the mean length of one edge and its perpendicular in the triangle to describe the triangle scaling.
我们方法的关键组成部分是如何在 FLAME [21] 网格和 3D 高斯图之间建立连接。最初，我们将网格的每个三角形与 3D 高斯配对，并让 3D 高斯与三角形一起跨时间步移动。换句话说，3D 高斯在其父三角形的局部空间中是静态的，但随着三角形的移动在全局（度量）空间中是动态的。给定三角形的三个顶点，我们将它们的平均位置 $\bm{T}$ 作为局部空间的原点。然后，我们将一条边的方向向量、三角形的法向量及其叉积作为列向量连接起来，形成一个旋转矩阵 $\bm{R}$ ，它描述了三角形在全局空间中的方向。我们还通过三角形中一条边及其垂线的平均长度来计算标量 $k$ 来描述三角形缩放。

For the paired 3D Gaussian of a triangle, we define its location $\bm{\mu}$, rotation $\bm{r}$, and anisotropic scaling $\bm{s}$ all in the local space. We initialize the location $\bm{\mu}$ at the local origin, the rotation $\bm{r}$ as an identity rotation matrix, and the scaling $\bm{s}$ as a unit vector. At rendering time, we convert these properties into the global space by:
对于三角形的成对 3D 高斯，我们在局部空间中定义其位置 $\bm{\mu}$ 、旋转 $\bm{r}$ 和各向异性缩放 $\bm{s}$ 。我们将位置 $\bm{\mu}$ 初始化为本地原点，将旋转 $\bm{r}$ 初始化为单位旋转矩阵，将缩放 $\bm{s}$ 初始化为单位向量。在渲染时，我们通过以下方式将这些属性转换为全局空间：

	$\displaystyle\bm{r}^{\prime}$	$\displaystyle=\bm{R}\bm{r},$		(4)
	$\displaystyle\bm{\mu}^{\prime}$	$\displaystyle=k\bm{R}\bm{\mu}+\bm{T},$		(5)
	$\displaystyle\bm{s}^{\prime}$	$\displaystyle=k\bm{s}.$		(6)

We incorporate triangle scaling in this process Eqs. 5 and 6 so that the local position and scaling of a 3D Gaussian are defined relative to the absolute scale of a triangle. This enables an adaptive step size in the metric space with a constant learning rate for parameters defined in the local space. For example, a 3D Gaussian paired with a smaller triangle will move slower in an iteration step than those paired with large triangles. This also makes interpreting the parameters regarding the distance from the triangle center easier.

Only having the same numbers of Gaussian splats as the triangles is insufficient to capture details. For instance, representing a curved hair strand requires multiple splats, while a triangle on the scalp may intersect with several strands. Therefore, we also need the adaptive density control strategy [14], which adds and removes splats based on the view-space positional gradient and the opacity of each Gaussian.

For each 3D Gaussian with a large view-space positional gradient, we split it into two smaller ones if it is large or clone it if it is small. We conduct this in the local space and ensure a newly created Gaussian is close to the old one that triggers this densification operation. Then, it is reasonable to bind a new 3D Gaussian to the same triangle as the old one because it was created to enhance the fidelity of the local region. Therefore, each 3D Gaussian must carry one more parameter, the index of its parent triangle, to enable binding inheritance during densification.

Besides densification, we also use the pruning operation as a part of the adaptive density control strategy [14]. It periodically resets the opacity of all splats close to zero and removes points with opacity below a threshold. This technique is effective in suppressing floating artifacts, however, such pruning can also cause problems in a dynamic scene. For instance, regions of the face that are often occluded (such as eyeball triangles), can be overly sensitive to this pruning strategy, and often end up with few or no attached Gaussians. To prevent this, we keep track of the number of splats attached to each triangle, and ensure that every triangle always has at least one splat attached.

We supervise the rendered images with a combination of $\mathcal{L}_{1}$ term and a D-SSIM term following [14]:

$$\mathcal{L}_{\text{rgb}}=(1-\lambda)\mathcal{L}_{1}+\lambda\mathcal{L}_{\text{D-SSIM}},$$

(7)

with $\lambda=0.2$. This already results in good re-rendering quality without additional supervision, such as depth or silhouette supervision, thanks to the powerful tile rasterizer [14]. We found however, that if we try to animate these splats via FLAME [21] to novel expressions and poses, large spike- and blob-like artifacts appear wildly throughout the scene. This is due to a poor alignment between the Gaussian splats and the triangles.

Position loss with threshold. A basic hypothesis behind 3D Gaussian rigging is that the Gaussian splats should roughly match the underlying mesh. They should also match their locations; for instance a Gaussian representing a spot on the nose should not be rigged to a triangle on the cheek. Although our splats are initialized at triangle centers, and new splats are added nearby these existing ones, it is not guaranteed that the primitives remain close to their parent triangle after the optimization. To address this, we regularize the local position of each Gaussian by:

$$\mathcal{L}_{\text{position}}=\|\max\left(\bm{\mu},\epsilon_{\text{position}}\right)\|_{2},$$

(8)

where $\epsilon_{\text{position}}=1$ is a threshold that tolerates small errors within the scaling of its parent triangle.

Scaling loss with threshold. Aside from position, the scaling of 3D Gaussians is even more essential for the visual quality during animation. Specifically, if a 3D Gaussian is large in comparison to its parent triangle, small rotations of the triangle – barely noticeable at the scale of the triangle – will be magnified by the scale of the 3D Gaussian, resulting in unpleasant jittering artifacts. To mitigate this, we also regularize the local scale of each 3D Gaussian by:

$$\mathcal{L}_{\text{scaling}}=\|\max\left(\bm{s},\epsilon_{\text{scaling}}\right)\|_{2},$$

(9)

where $\epsilon_{\text{scaling}}=0.6$ is a threshold that disables this loss term when the scale of a Gaussian less than $0.6\times$ the scale of its parent triangle. This $\epsilon_{\text{scaling}}$-tolerance is indispensable; without it, the Gaussian splats shrink excessively, causing rendering speeds to deteriorate, as camera rays need to hit more splats to before zero transmittance is reached.

Our final loss function is thus:

$$\mathcal{L}=\mathcal{L}_{\text{rgb}}+\lambda_{\text{position}}\mathcal{L}_{\text{position}}+\lambda_{\text{scaling}}\mathcal{L}_{\text{scaling}},$$

(10)

where $\lambda_{\text{position}}=0.01$ and $\lambda_{\text{scaling}}=1$. Note that we only apply $\mathcal{L}_{\text{position}}$ and $\mathcal{L}_{\text{scaling}}$ to visible splats. Thereby, we only regularize points when the color loss $\mathcal{L}_{\text{rgb}}$ is present. This helps to maintain the learned structure of often-occluded regions such as teeth and eyeballs.

Implementation details. We use Adam [16] for parameter optimization (the same hyperparameter values are used across all subjects). We set the learning rate to 5e-3 for the position and 1.7e-2 for the scaling of 3D Gaussians and keep the same learning rates as 3D Gaussian Splatting [14] for the rest of the parameters. Alongside the Gaussian splat parameters, we also finetune the translation, joint rotation, and expression parameters of FLAME [21] for each timestep, using learning rates 1e-6, 1e-5, and 1e-3, respectively. We train for 600,000 iterations, and exponentially decay the learning rate for the splat positions until the final iteration, where it reaches 0.01$\times$ the initial value. We enable adaptive density control with binding inheritance every 2,000 iterations, from iteration 10,000 until the end. Every 60,000 iterations, we reset the Gaussians’ opacities. We use a photo-metric head tracker to obtain the FLAME parameters, including shape $\bm{\beta}$, translation $\bm{t}$, pose $\bm{\theta}$, expression $\bm{\psi}$, and vertex offset $\Delta\bm{v}$ in the canonical space. We also manually add triangles for the upper and lower teeth, which are rigid to the neck and jaw joints, respectively.

Dataset. We conduct experiments on video recordings of 9 subjects from the NeRSemble [17] dataset. All recordings contain 16 views covering the front and sides of the subject. We take 11 video sequences for each subject, each containing around 200 time steps. We downsample the images to a resolution of $802\times 550$. Participants were instructed to perform specific expressions in 10 sequences and freely perform in the other.

We train our method, plus baselines, using 9 out of 10 expression sequences and 15 out of 16 available cameras. This allows us to quantitatively evaluate both novel expression and novel view synthesis. We use the 11^th (free-performance) sequences to visually assess cross-identity re-enactment capabilities.

We evaluate the reconstruction quality of avatars by novel-view synthesis and animation fidelity by self-reenactment. Fig. 3 shows qualitative comparisons. For novel-view synthesis, all methods produce plausible rendering results. A close inspection of PointAvatar’s [52] results show dotted artifacts, owing its fixed point size. In our case, the anisotropic scaling of 3D Gaussians alleviates this issue.

INSTA [55] shows clean results for the face, however regions around the neck and shoulder can be noisy, as the tracked FLAME meshes are often misaligned in those regions. This causes issues for INSTA, as its warping process is based on the nearest triangle. In the case of our method, each Gaussian splat is rigged to a consistent triangle, regardless of the pose or expression. When the tracked mesh is inaccurate, the positional gradient to a Gaussian splat can consistently back-propagate to the same triangle. This enables misalignments due to incorrect FLAME tracking to be corrected while the 3D Gaussians are being optimized.

AvatarMAV [46] shows comparable quality to other methods in novel-view synthesis but struggles with novel-expression synthesis. This is because it only uses the expression vector of a 3DMM as conditioning. Since the control from the expression vectors to the deformation bases must be learned, it struggles to reproduce expressions that are far from the training distribution. Similar conclusions can be drawn from the quantitative comparison shown in Tab. 1. Our approach outperforms others by a large margin regarding metrics for novel-view synthesis. Our method also stands out in self-reenactment, with significantly lower perceptual differences in terms of LPIPS [50]. Note that self-reenactment is based on tracked FLAME meshes that may not perfectly align with the target images, thus bringing disadvantages to our results with more visual details regarding pixel-wise metrics such as PSNR.

For a real-world test of avatar animation, we conduct experiments on cross-identity reenactment in Fig. 4. Our avatars accurately reproduce eye blinks and mouth movements from source actors showing lively, complex dynamics such as wrinkles. INSTA [55] suffers from aliasing artifacts when the avatars move beyond the occupancy grid of I-NGP [30] optimized for training sequences. The movement of results from PointAvatar [52] is not precise because its deformation space is not guaranteed to be consistent with FLAME. AvatarMAV [46] exhibits large degradations in reenactment due to a lack of deformation priors.

To validate the effectiveness of our method components, we deactivate each of them and report results in Tab. 2.

Adaptive density control with binding inheritance. Without binding inheritance, we lose the ability to add and remove Gaussian splats besides the initial ones. With a limited number of splats, the scaling of each splat increases to occupy the same space, causing blurry renderings. This leads to huge fidelity loss, as shown in the second row of Tab. 2.

Regularization on the scaling of Gaussians splats. Without the scaling loss, the image quality slightly drops (the third row in Tab. 2). But when we use the scaling loss without a small error tolerance, all metrics drastically deteriorate (the fourth row in Tab. 2). This is because all the splats are regularized to be infinitely small, making it hard to construct opaque surfaces to handle occlusions in a scene.

Regularization on the position of Gaussians splats. When turning off the threshold for the position loss, we get slightly worse metrics (the sixth row in Tab. 2). But when turning off the position loss, the metrics on novel-view synthesis become the best in the table (the fifth row in Tab. 2). This is reasonable because Gaussian splats can freely move in the space to achieve minimal re-rendering error. However, the distributions of Gaussian splats are only optimal for the training frames. The avatar shows artifacts such as cracks and fly-around blobs (Fig. 5) when animated with novel expressions and poses. Therefore, the position loss is necessary to enforce a conservative deviation of splats from the mesh surface so that we can animate the reconstructed avatar with unseen facial motion.

FLAME fine-tuning. Thanks to our consistent binding between Gaussian splats and triangles, we can effectively fine-tune FLAME parameters while optimizing Gaussian splats. As shown in the last row in Tab. 2, turning off the fine-tuning negatively affects image quality for novel-view synthesis. For self-reenactment, fine-tuning FLAME parameters also leads to lower perceptual error.