Machine Learning from Human Preferences

Machine learning from human preferences (MLHP) provides mechanisms for optimizing systems that are used by humans - with applications in personalization, finetuning, and assistance. MLHP has been important to successes in modern language modeling, but the technical and conceptual underpinnings are spread across fields: psychometrics, discrete choice, reinforcement learning, information retrieval, and normative theory. This course will provide a careful and practical introduction to the main methods and models of MLHP. It will cover: choice modeling and its axiomatic approaches; imitation and static and active learning of choice; reinforcement learning from human feedback; assistance games, bandits, and recommendation; behavioral biases, style control; aggregation and fairness. This is a graduate-level course. By the end of the course, students should be able to build, analyze, and optimize systems that are used by humans.