Fitting a Normal Distribution (comparison with stan, PyMC)

2

Thanks for the comparison! That was interesting. Here are some tips:

  1. You can write the model more simply as
y = NormalWithSoftplusScale(loc=mu, scale=inv_softplus_sigma, sample_shape=N)
  1. ed.KLqp got to the answer in very few iterations and simply perturbed around the answer over more iterations. You can more stable scale parameters across runs using a smaller learning rate.
  2. Never believe in automatic algorithms. They can give a false sense of security in toy problems and failure in real problems. In particular, ed.HMC here is not automatic: you need to tune the step size parameter and number of leapfrog steps.

In general, successes/failures on a normal-normal model are not illuminating because they may not be indicative of real successes/failures. For example, decide on a real-world task such as Bayesian neural network classification on MNIST. How easy is it in the PPL to specify the program, inference, and evaluation metric? Which is most automatic? Which is most flexible with respect to the model and/or inference? Which is fastest? Which had the best documentation to help with this experiment?