# Edward for NLP tasks

**URL:** <https://discourse.edwardlib.org/t/edward-for-nlp-tasks/185>\
**Category:** General\
**Created:** [June 19, 2017, 9:07pm UTC](https://discourse.edwardlib.org/t/edward-for-nlp-tasks/185 "2017-06-19T21:07:27Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![misha](https://avatars.discourse-cdn.com/v4/letter/m/c6cbf5/32.png) [@misha](https://discourse.edwardlib.org/u/misha)\
**Post date:** [June 19, 2017, 9:07pm UTC](https://discourse.edwardlib.org/t/edward-for-nlp-tasks/185/1 "2017-06-19T21:07:27Z")

</div>

Hi All,

I am learning probabilistic machine learning, and I like an idea to do this in TensorFlow. I saw several ML algos in Edward tutorials, but I have a problem to apply a Gaussian Process classifier to the textual data (with lots of features).

I would greatly appreciate any help with it (a sample, reference, etc.)

Thanks!

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [June 20, 2017, 3:35am UTC](https://discourse.edwardlib.org/t/edward-for-nlp-tasks/185/2 "2017-06-20T03:35:09Z")

</div>

I assume you saw the GP classification tutorial? It’s directly applicable to text data.

I’m not sure if people have actually published papers with GP on text before. In practice, one difference with text over other sorts of feature inputs is that it’s very sparse. You can use an embedding matrix/look up table in order to convert the sparse inputs into a set of dense features. Then you place a GP classifier over these features and train the GP and embeddings jointly.

For example:

```python
from edward.models import Bernoulli, MultivariateNormalTriL
from edward.util import rbf

batch_size = 256 # batch size during training
vocabulary = 1000 # vocabulary size
embedding_dim = 100 # dimension of embeddings

X = tf.placeholder(tf.float32, [batch_size, vocabulary])
W = tf.Variable(tf.random_normal([vocabulary, embedding_dim]), name="embedding_matrix")
Z = tf.nn.embedding_lookup(W, X)

f = MultivariateNormalTriL(loc=tf.zeros(N), scale_tril=tf.cholesky(rbf(Z)))
y = Bernoulli(logits=f)

```

The embedding idea comes from [word2vec](https://www.tensorflow.org/tutorials/word2vec#motivation_why_learn_word_embeddings). My guess is this would do really well as a bag of words classifier. And it’s extendable in all the usual ways (e.g., sequence-based, attention, deeper or just no GPs).

---

<div class="post-metadata">

**Author:** ![misha](https://avatars.discourse-cdn.com/v4/letter/m/c6cbf5/32.png) [@misha](https://discourse.edwardlib.org/u/misha)\
**Post date:** [June 20, 2017, 8:09pm UTC](https://discourse.edwardlib.org/t/edward-for-nlp-tasks/185/3 "2017-06-20T20:09:05Z")

</div>

Thank you, Dustin! I am working on it…

---

<div class="post-metadata">

**Author:** ![misha](https://avatars.discourse-cdn.com/v4/letter/m/c6cbf5/32.png) [@misha](https://discourse.edwardlib.org/u/misha)\
**Post date:** [June 21, 2017, 3:33pm UTC](https://discourse.edwardlib.org/t/edward-for-nlp-tasks/185/4 "2017-06-21T15:33:51Z")

</div>

Dustin,

I am using gensim.models.doc2vec to compress features, but something is missing in inference = ed.KLqp({f: qf}, data={…, y: y\_ph}). Without an additional param, loss values during training are too big. Please look at this code…

N = X\_train.shape[0] # number of data points  
M = 256 # batch size  
D = X\_train.shape[1] # number of features

data = generator([X\_train, y\_train], M)

X = tf.placeholder(tf.float32, [M, D])  
y\_ph = tf.placeholder(tf.int32, [M])  
f = MultivariateNormalTriL(loc=tf.zeros(M), scale\_tril=tf.cholesky(rbf(X)))  
y = Bernoulli(logits=f)

qf = Normal(loc=tf.Variable(tf.random\_normal([M])),  
…scale=tf.nn.softplus(tf.Variable(tf.random\_normal([M]))))

n\_batch = int(N / M)  
n\_epoch = 5

inference = ed.KLqp({f: qf}, data={…, y: y\_ph})  
inference.initialize(n\_iter=n\_batch \* n\_epoch, n\_samples=5, scale={y: N / M})  
tf.global\_variables\_initializer().run()

for \_ in range(inference.n\_iter):  
…X\_batch, y\_batch = next(data)  
…info\_dict = inference.update({X: X\_batch, y\_ph: y\_batch})  
…inference.print\_progress(info\_dict)

Thanks!

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [June 21, 2017, 6:35pm UTC](https://discourse.edwardlib.org/t/edward-for-nlp-tasks/185/5 "2017-06-21T18:35:38Z")

</div>

You’re trying to do stochastic variational inference with a Gaussian process. Each of the function outputs (a parameter in `qf`) is associated to a different data point. However, all data points depend on each other through `f`'s covariance matrix so your approach doesn’t apply. You either have to train the GP on the full data set or be a little more sophisticated with inducing variables (see Hensman et al., 2013).