for the bayesian_logistic_regression.py example, I modified the data generation procedure by setting an arbitrary value for w and b (instead of using the np.tanh function). def build_toy_dataset(N, noise_std=0.1): X = (random.uniform(size=(N)) - 0.5) * 100 w0 = np.full((FLAGS.D), 1.0, np.float6…

The data generated by your procedure are linearly separable, and the prior on w is not strong enough (see Gelman 2009). If I instead use w = Normal(loc=tf.zeros([D]), scale=tf.ones([D])) then I get the approximate posterior mean of w is 1.1. The posterior for b is N(.88, 7.67), which suggests yo…

I have numerous questions on your response. What is the significance of having linearly separable data? Does it make it harder to do inference? Gelman has many papers in 2009, could you be more specific. I see you set the scale at 1.0, right on the money. Mine was set in 3.0, which you would think i…

The data being linearly separable means the maximum likelihood estimate of w is infinite. I mean this paper: http://www.stat.columbia.edu/~gelman/research/published/priors11.pdf The main idea of the paper is to use the prior on w to get a sensible posterior on w even in the case of separable data.…

At the moment, I would be very happy if I at least reproduce your results. I am getting tiny numbers like 0.00772677 for qw, regardless of what scale I set for w. And the funny thing is I get a number 10 x that for w -> 0.07774825. For qb, I get 0.008026831, while b is 100X that but negative -> -0.8…

Apparently, the numbers get much closer if I reduce the training data set size. By cutting the sample size from 1000 down to 40, the numbers get really close. How does this work? Normally, I expect higher sample size gets you better results. In this case, the results seemed to be divided by a large …

On the otherhand, the exactly same training data works very well using VI instead of HMC (for all sample sizes). Is there something I need to modify to use HMC properly?

The code I used is here: https://users.rcc.uchicago.edu/~aksarkar/nwas/test.html#org217bacd The main difference is the initialization of qw, qb. It might actually only matter for the first HMC sample, but it radically changes the acceptance rate. qw.eval() is the same as sess.run(qw). Evaluating q…

It seems to be quite tricky to get HMC to go. In the Logistic code you showed, I increased the w dimension D to 3, and initialized w to [1, 2, 3]. it runs fine. [1.108. 2., 3] also runs fine. if I simply change to [1.108, 2.318, 3], I get Acceptance Rate: 0.000. The threshold is between 2.2 and 2.3. …

I think the reason it is so sensitive is that your data is badly behaved (linearly separable). In particular, the scale of X is such that p(y | x) will be either 0 or 1 regardless of the range of values of w0 you’re looking at. I don’t completely understand what you changed, but I suspect no algori…

yes, I can get it to run by change X to random normal. The W and b values are within the margin of error, when the scale is properly set. A narrow scale setting both get the wrong answer and a very strong confidence (low variance). I am curious about the claim that HMC methods are asymptotically acc…

Resolve logistic regression parameters on simulated data

bhomass April 28, 2018, 12:34am 21

I see the subject of NUTS has been brought up before, just not getting the priority right now.

Topic		Replies	Views
Simple Beta-Bernoulli model and HMC inference	1	1092	December 14, 2017
Acceptance Rate 0 for HMC in IRT models	1	1753	January 18, 2019
Having trouble setting up basic HMC model	4	1943	May 29, 2017
Parameter Learning with Simple Bayesian Network; PyMC3 vs. Edward; Edward posteriors not converging around correct parameter values	3	2894	March 27, 2018
MCMC not working for basic model	4	1031	May 8, 2018

Resolve logistic regression parameters on simulated data

Related topics