References on Bayesian nonparametrics

Last updated: 11/15/17

This post is a collection of references for Bayesian nonparametrics that I’ve found helpful or wish that I had known about earlier. For many of these topics, some of the best references are lecture notes, tutorials, and lecture videos. For general references, I’ve prioritized listing newer references with a more updated or comprehensive treatment of the topics. Many references are missing, and so I’ll continue to update this over time.

Nonparametric Bayes: an introduction

These are references for people who have little to no exposure to the area, and want to learn more. Tutorials and lecture videos offer a great way to get up to speed on some of the more-established models, methods, and results.

  1. Teh (and others). Nonparametric Bayes tutorials.

    This webpage lists a number of review articles and lecture videos that as an introduction to the topic. The main focus is on Dirichlet processes, but a few tutorials cover other topics: e.g., Indian buffet processes, fragmentation-coagulation processes.

  2. Broderick. Nonparametric Bayesian Methods: Models, Algorithms, and Applications. 2017. See also joint tutorial with Michael Jordan at the Simon’s Institute.

    An introduction to Dirichlet processes and the Chinese restaurant process, and nonparametric mixture models. Also comes with R code for generating from and sampling (inference) in these models.

  3. Wasserman and Lafferty. Nonparametric Bayesian Methods. 2010.

    This gives an overview of Dirichlet processes and Gaussian processes and places these methods in the context of popular frequentist nonparametric estimation methods for, e.g., CDF and density estimation, regression function estimation.

Theory and foundations

It turns out there’s a lot of theory and foundational topics.

  1. Orbanz. Lecture notes on Bayesian nonparametrics. 2014.

    A great introduction to foundations of Bayesian nonparametrics, and provides many references for those who want a more in-depth understanding of topics. E.g.: random measures, clustering and feature modeling, Gaussian processes, exchangeability, posterior distributions.

  2. Ghosal and van der Vaart. Fundamentals of Bayesian Nonparametric Inference. Cambridge Series in Statistical and Probabilistic Mathematics, 2017. contents

    The most recent textbook on Bayesian nonparametrics, focusing on topics such as random measures, consistency, contraction rates, and also covers topics such as Gaussian processes, Dirichlet processes, beta processes.

  3. Kleijn, van der Vaart, van Zanten. Lectures on Nonparametric Bayesian Statistics. 2012.

    Lecture notes with some similar topics as (and partly based on) the Ghosal and van der Vaart textbook, including a comprehensive treatment of posterior consistency.

Specific topics

  1. Kingman. Poisson processes. Oxford Studies in Probability, 1993.

    Everything you wanted to know about Poisson processes. (See Orbanz lecture notes above for more references on even more topics on general point process theory.)

  2. Pitman. Combinatorial stochastic processes. 2002.

  3. Aldous. Exchangeability and related topics. 1985.

  4. Orbanz and Roy. Bayesian Models of Graphs, Arrays and Other Exchangeable Random Structures. IEEE TPAMI, 2015.

  5. Jordan. Hierarchical models, nested models, and completely random measures. 2013.

  6. Broderick, Jordan, Pitman. Cluster and feature modeling from combinatorial stochastic processes. 2013.

Background: probability

Having basic familiarity with measure-theoretic probability is fairly important for understanding many of the ideas in this section. Many of the introductory references aim to avoid measure theory (especially for the discrete models), but even this is not always the case, so it is helpful to have as much exposure as possible.

  1. Hoff. Measure and probability. Lecture notes. 2013. pdf

    Gives an overview of the basics of measure-theoretic probability, which are often assumed in many of the above/below references.

  2. Williams. Probability with martingales. Cambridge Mathematical Textbooks, 1991.

  3. Cinlar. Probability and stochastics. Springer Graduate Texts in Mathematics. 2011.

Models and inference algorithms

There are too many papers on nonparametric Bayesian models and inference methods. Below, I’ll list a few “classic” ones, and continue to add more over time. The above tutorials contain many more references.

Dirichlet processes: mixture models and admixtures

Dirichlet process mixture model

  1. Neal. Markov Chain sampling methods for Dirichlet process mixture models. Journal of Computational and Graphical Statistics, 2000. pdf

  2. Blei and Jordan. Variational inference for Dirichlet process mixtures. Bayesian Analysis, 2006. pdf

Hierarchical Dirichlet process

  1. Teh, Jordan, Beal, Blei. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 2006. pdf

  2. Hoffman, Blei, Wang, Paisley. Stochastic variational inference. Journal of Machine Learning Research, 2013. pdf

Nested Dirichlet process & Chinese restaurant process

  1. Rodriguez, Dunson, Gelfand. The Nested Dirichlet process. Journal of the American Statistical Association, 2008. pdf

  2. Blei, Griffiths, Jordan. The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. Journal of the ACM, 2010. pdf

  3. Paisley, Wang, Blei, Jordan. Nested hierarchical Dirichlet processes. IEEE TPAMI, 2015. pdf

Mixtures of Dirichlet processes

  1. Antoniak. Mixture of Dirichlet processes with applications to Bayesian nonparametrics. Annals of Statistics, 1974. pdf

Additional inference methods

  1. Ishwaran and James. Gibbs sampling methods for stick-breaking priors. Journal of the American Statistical Association, 2001. pdf

  2. MacEachern. Estimating normal means with a conjugate style dirichlet process prior. Communications in Statistics - Simulation and Computation, 1994. pdf

  3. Escobar and West. Bayesian density estimation and inference using mixtures. Journal of the American Statististical Association, 1995. pdf

Indian buffet processes and latent feature models

  1. Griffiths and Ghahramani. The Indian buffet process: an introduction and review. Journal of Machine Learning Research, 2011. pdf

  2. Thibaux and Jordan. Hierarchical beta processes and the Indian Buffet process. AISTATS, 2007. pdf | longer

Tweet