Problem Statement:

Genetic conditions frequently present with pronounced and distinct facial characteristics, which are often the first clue in clinical diagnostics. To this day, a clinical diagnosis relies on an assessment by the clinical experts who are trained to recognize either facial features or a ‘gestalt’ associated with the syndrome. Recently, however, objective phenotyping approaches for assessing the face have proven to be successful in providing automated assistance for clinical diagnosis [1–3]. Previous work on syndrome identification has mostly focused on 2D facial images. While 2D photographs are easier to obtain, 3D images capture facial shape completely and more directly and are not subject to projection distortion, which is inherent to all 2D imaging modalities. Given the increasing accessibility of 3D imaging hardware, including smartphones, studying the potential of 3D shape analysis for syndrome classification is very promising. Previous work in this domain has used linear dimensionality reduction and classification techniques and has typically focused on discriminating one or a few syndrome groups from controls [4,5]. The most comprehensive attempt at multi- syndrome classification deployed these linear techniques on 64 syndrome groups, with classification based on a sparse configuration of 65 anatomical facial landmarks [3].

Deep convolutional neural networks (DCNNs) have recently been deployed for syndrome identification based on 2D images [1, 2]. To learn from 3D data, some approaches transform the data into the Euclidean domain using a 2D UV- or 3D voxel-representation [6,7] or learn from the point cloud using geometric deep learning (GDL) [8]. However, 3D to 2D transformations, voxelization, or ignoring the local connectivity of the mesh by reducing it to a point cloud may cause significant loss of information of the underlying geometry. With recently introduced GDL techniques [9], it is now possible to apply deep learning directly on the non-Euclidean facial surfaces, which are discretized as graphs or

meshes [10–13]. Drawing from this literature, we use spiral convolutional operators which apply local anisotropic filters to features given on a non-Euclidean domain, mimicking the classical convolutional filters used in CNNs [10, 14–16]. These operators are applicable to meshes that share a common topology, as for the 3D facial data in this work. While classification loss functions such as cross-entropy are commonly used for classification tasks, we use deep metric learning instead to learn the distinguishing facial features based on similarity measures. More specifically, we implement a triplet-based architecture and train it with a triplet loss function [18]. The advantage of the triplet loss is that it can efficiently learn a large number of classes, even with a small number of samples per class [19,20].

Since the human face comprises multiple integrated parts that are distinct from each other based on their anatomy, embryological origin, and function, facial dysmorphism can be described by defining atypicalities in each part of the face. In the literature, part-based approaches have been employed, mostly for 2D images, in which separate models are trained for local regions such as the eyes, nose, mouth, and chin, along with the entire face [1, 21]. Inspired by this idea, in this paper, a part-based Geometric Encoder (GE) is proposed to reduce the dimensionality of the variation among 3D meshes. This is executed by learning a supervised lower-dimensional embedding space or metric space by GDL, for the first time, for multiclass syndrome classification. Figure 1 shows the general pipeline. Initially, a low dimensional embedding vector is learned for each facial module using a separate GE block. We experiment with various fusion techniques to combine embeddings or classification scores across multiple facial modules, in order to form a more reliable classification model. Performance of the geometric encoder is benchmarked against principal component analysis (PCA), which is the commonly used unsupervised linear dimensionality reduction technique in 3D facial shape analysis. For each embedding type (PCA and GDL), we assess the contribution of the part-based setup by comparing its performance to using the full face.

For further elaboration on the network design please refer to the materials and methods section of the paper.

Results & Discussion

In today’s era of image-based deep learning, CNNs have only been deployed on 2D images in the context of syndrome classification [1,2]. This is the first study to apply geometric deep learning to multi-class syndrome classification. We derived full-face and propose part-based geometric encoders to encode observations into non-linear metric spaces, supervised to separate among classes. These embeddings were then subjected to classification by LDA. We experimented with the dimensionality of the space and different methods of fusion for the part-based encoders. We compare the results to those using linear embedding spaces derived from PCA.

Figure 4a shows the mean sensitivity of the full-face and the best part-based models for both PCA and GE. It indicates that, for the full-face method, GE significantly outperforms PCA, although this discrepancy becomes smaller as the number of dimensions increases. This is mainly because GEs are trained in a supervised way (in contrast to PCA) therefore the embeddings are trained to preserve the most relevant information to the discrimination task. This is likely to include localized facial features only common to a small proportion of the training set (e.g. to one particular syndrome). PCA ultimately learns these too but it takes additional dimensions to do so, because it only aims to preserve the maximum amount of variation across the entire training dataset. Therefore, its first dimensions will capture broad patterns of facial variation and population structure and not specific discriminative facial features. Figure 6 of supplementary material depicts a 2D visualization of the 14 dimensional part-based PCA and GE projections of the train set (smaller dots) and test set (larger dots) using t-SNE [29]. This plot clearly demonstrates that the GE learns a better structured metric space in which distinct syndromes reside into clusters.