Account

P10 · Generative AI

language modelling → Transformer (reference tutorial) → fine-tuning, evaluation

Generative AI built from first principles: tokens, embeddings, attention and the full Transformer assembled step by step, culminating in a live BERT pipeline you can probe sentence by sentence.

The end-to-end picture first: every module in this pillar is one stage of the same journey — raw sentence in, meaning-bearing vectors out. The diagram below is the map; the corpus editor under it sets the sentences every lab computes on.

Raw Textstringstext-representationTokenizetoken IDs + masktext-representationEmbedseq × d_modelvectorspositional-encodingPositionalEncodeseq × d_model+ positionattentionAttentionseq × d_modelcontextualattentionMulti-HeadAttentionseq × d_modelmulti-contexttransformer-architectureEncoderBlockseq × d_modelencoder outputtransformer-architecturePool(d_model,)sentence vector
End-to-end: raw text becomes a sentence vector in eight stages. The Atlas module walks the inside of each one; Console & Inspect run the live BERT pipeline behind them.

Change a sentence above and every figure downstream recomputes from it — the tokens, the embeddings, the attention weights, the sentence vector. That is the point of fixing one small corpus for the whole pillar: nothing here is illustrated with numbers you have to take on trust, and a stage that seems mysterious can be interrogated by changing its input and watching what moves.

Orientation — concepts before the maths

Text Representation

Positional Encoding

Attention

Transformer Architecture

LLM Essentials

LLM Language Practice

P10 · Generative AI — TransformerLab