Using tfdbg for debugging Edward models

I am having problems with loss functions in Edward evaluating to nans. This problem is tricky to debug because of how Tensorflow evaluates computational graphs. There is a debugger for tensorflow called tfdbg. tfdbg has a useful filter to search for infs and nans. It took me a while to figure out how to use it with Edward and there’s no example on the forum yet, so I’ll post the solution here in case others also find it useful.

import edward as ed
from tensorflow.python import debug as tf_debug

sess = tf_debug.LocalCLIDebugWrapperSession(ed.util.get_session(), ui_type="curses")
sess.add_tensor_filter("has_inf_or_nan", tf_debug.has_inf_or_nan)
...
with sess.as_default():
    ...
    inference = ed.KLqp(...)
    inference.run(...)
    ....

Launching your code from the command line will start the tfdbg command-line debugger. From there, you can run until the filter detects an inf or nan value with:
run -f has_inf_or_nan

4 Likes

Nice work! This would be really neat to have as a snippet somewhere in http://edwardlib.org/tutorials/. Contributions welcome.