If I understand your code correctly, you are doing 50 inference updates for each new set of training examples. This is not guaranteed to converge to the posterior over the full data set. Mathematically, the model p(x, z) fits to the current set of training examples to obtain p(z | x_1). Then on a new batch—because the prior hasn’t changed—the model refits to the batch while forgetting about the previous examples. This returns p(z | x_2) and not p(z | x_1, x_2).
If you’d like to do this sort of recursive Bayesian inference, you need to plug in the inferred posterior as the new prior on additional fits. This can be a little involved with building big graphs or being a little smarter about re-assigning tf.Variables in the prior post-inference while fixing them during inference.
Alternatively, if your setting allows it, keep the old data points around and perform inference jointly over the new and old data points.
Also relevant: