Transformers
The attention-based neural architecture behind modern LLMs — studied hands-on through the IBM Generative AI Engineering certificate.
The transformer is a neural network architecture built on self-attention. Encoder models (BERT-style), decoder models (GPT-style) and encoder–decoder models all derive from it, and it underpins every modern LLM.
How I use it
I studied transformers in depth in the IBM Generative AI Engineering Professional Certificate professional certificate: Generative AI Language Modeling with Transformers (GPT-style language models and translation), Generative AI Engineering with Transformers & LLMs and Generative AI and LLMs: Architecture and Data Preparation, working with PyTorch and Hugging Face. This grounds my application-layer work — agents, RAG (Retrieval-Augmented Generation) and provider integrations — in how the models actually work.
Why it matters
Understanding attention, context windows and tokenization explains practical limits I design around in agents — such as why Akashi windows and summarizes older context, and why retrieval beats stuffing everything into a prompt.
Related
- Ale Zanello · LLMs · NLP · Deep Learning · Fine-Tuning