Similarity Encoder: A Neural Network Architecture for Learning Similarity Preserving Embeddings

• Main Author: Horn, Franziska
• Format: Dissertation
• Language: English
• Published: ProQuest Dissertations & Theses 01-01-2020
• Subjects: Acoustics; Approximation; Artificial intelligence; Automation; Classification; Datasets; Eigenvalues; Experiments; Information science; Language; Machine learning; Natural language processing; Neural networks; Noise; Philosophy; Psychology; Ratings & rankings; Recommender systems
• ISBN: 9798522952259
• DOI: 10.14279/depositonce-9956

Matrix factorization is at the heart of many machine learning algorithms, for example, for dimensionality reduction (e.g. kernel PCA) or recommender systems relying on collaborative filtering. Understanding a singular value decomposition (SVD) of a matrix as a neural network optimization problem enables us to decompose large matrices efficiently while dealing naturally with missing values in the given matrix. But most importantly, it allows us to learn the connection between data points' feature vectors and the matrix containing information about their pairwise relations. In this thesis, we introduce a novel neural network architecture termed Similarity Encoder (SimEc), which is designed to simultaneously factorize a given target matrix while also learning the mapping to project the data points' feature vectors into a similarity preserving embedding space. This makes it possible to, for example, easily compute out-of-sample solutions for new data points. Additionally, we demonstrate that SimEcs can preserve non-metric similarities and even predict multiple pairwise relations between data points at once. As the first part of the SimEc architecture, mapping from the original (high dimensional) feature space to the (low dimensional) embedding, can be realized by any kind of (deep) neural network, SimEcs can be used in a variety of application areas. As we will demonstrate, SimEcs can serve as a reliable baseline model in pairwise relation prediction tasks such as link prediction or for recommender systems. The pairwise relations and similarities predicted by a SimEc model can also be explained using layer-wise relevance propagation (LRP). Furthermore, SimEcs can be used to pre-train a neural network used in a supervised learning task, which, for example, improves the prediction of molecular properties when only few labeled training samples are available. Finally, a variant of SimEc, called Context Encoder (ConEc), provides an intuitive interpretation of the training procedure of the CBOW word2vec natural language model trained with negative sampling and makes it possible to learn more expressive embeddings for words with multiple meanings as well as to compute embeddings for out-of-vocabulary words.