I am having problems with loss functions in Edward evaluating to nans. This problem is tricky to debug because of how Tensorflow evaluates computational graphs. There is a debugger for tensorflow called tfdbg. tfdbg has a useful filter to search for infs and nans. It took me a while to figure out how to use it with Edward and there’s no example on the forum yet, so I’ll post the solution here in case others also find it useful.
import edward as ed from tensorflow.python import debug as tf_debug sess = tf_debug.LocalCLIDebugWrapperSession(ed.util.get_session(), ui_type="curses") sess.add_tensor_filter("has_inf_or_nan", tf_debug.has_inf_or_nan) ... with sess.as_default(): ... inference = ed.KLqp(...) inference.run(...) ....
Launching your code from the command line will start the tfdbg command-line debugger. From there, you can run until the filter detects an inf or nan value with:
run -f has_inf_or_nan