Labs · Scores and attention

Softmax

Logits become probabilities; temperature, masking and scale decide how peaked they are and whether a gradient survives.

pi=ezi/T∑jezj/Tp_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}8 logits

A near-tie at the top, one outlier, two negatives.

What softmax computes

Reads /api/labs/softmax — loading

Computing…

Softmax — Labs — TransformerLab