Labs · Vectors

Pooling with and without the mask

One sentence vector from a batch's token rows — [CLS], mean or max — and what the padding rows do to it unless the mask keeps them out.

\bar h = \frac{\sum_i m_i h_i}{\sum_i m_i}

8 rows × 4 dims · 2 of them padding

Two padding rows holding leftovers: the unmasked mean drags towards them, the masked one does not.

One vector from the batch — with the mask, and without

Reads /api/labs/pooling — loading

Computing…