Representation Learning in Linear Factor Models

Abstract: A promise of representation learning—an active area of research in machine learning—is that algorithms will, one day, learn to extract the most useful information from modern data sources such as videos, images, or text. This paper analyzes recent theoretical developments in this literature through the lens of a Gaussian Linear Factor Model. We first derive sufficient representations for this model—defined as functions of the covariates that, upon conditioning, render the outcome variable and covariates independent. We then study the theoretical properties of these representations and establish their asymptotic invariance; which means the dependence of the representations on the factors’ measurement error vanishes as the dimension of the covariates grows to infinity. Finally, we use a decision-theoretic approach to understand the extent to which these representations are useful for solving downstream tasks. We show that the conditional mean of the outcome variable given covariates is an asymptotically invariant, sufficient representation, that can solve any task efficiently, not only prediction.