Compute policy gradient using cross entropy loss

[weiliu620]

This is not an issue of the code per se, but I am learning RL and am wondering how the policy gradient is calculated in pg_reinforce.py. In this line

self.cross_entropy_loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
logits=self.logprobs, labels=self.taken_actions)

A loss defined as the cross entropy between the action distribution from policy, and the actual action taken. But standard text such as Sutton define the policy gradient in the the REINFORCE algorithm simply as the 'log grad' of the policy function.

What is the difference of the two definitions? It seems cross entropy loss is a more general definition because it includes not just the action taken but all remaining actions. Can you give a reference of your method?

This is great project. Thank you for sharing this.

[Utsavz]

Did you get an answer? @weiliu620 Currently I am learning RL. Please help

[luofuli]

@Utsavz @weiliu620

• The 'log grad' of the policy function is the log(p(x)).
• The loss function of cross-entropy is \sum q * log(p).

These two are the same because q is a one-hot vector (the index of x is 1, others are 0). That's is two say,when the q is a one-hot vector, \sum q * log(p) = log(p(x)), where x refers to the sampled example.