The Extra Softmax, Max Entropy, and Exploring the Entropy Tradeoff

%pylab
%matplotlib inline
import torch

Using matplotlib backend: TkAgg
Populating the interactive namespace from numpy and matplotlib

import jovian

Purpose

We've been seeing that using an extra softmax helps our method. Here I'm going to show what that extra softmax is doing -- it simulates maximum entropy by pushing the target very close to the center of the simplex -- and propose an experiment for testing the effect of the target entropy on our method.