LLMs are revolutionizing the AI world, and attention is everywhere!
Today, I'll clearly explain how self-attention works! π
A Thread π§΅π
Today, I'll clearly explain how self-attention works! π
A Thread π§΅π
Computer are good with numberβοΈ
In NLP we convert the sequence of words into token & then token to embeddings.
You can think of embedding as a meaningful representation of each token using a bunch of numbers.
Check this out π
In NLP we convert the sequence of words into token & then token to embeddings.
You can think of embedding as a meaningful representation of each token using a bunch of numbers.
Check this out π
Now, for a language model to perform at a human level, it's not sufficient for it to process these tokens independently.
It's also important to understand the relationship between them!
Check this π
It's also important to understand the relationship between them!
Check this π
In the self-attention, relationships between tokens are expressed as probability scores.
Each token assigns the highest score to itself and additional scores to other tokens based on their relevance.
Check this out π
Each token assigns the highest score to itself and additional scores to other tokens based on their relevance.
Check this out π
To understand how self-attention works we first need to understand 3 terms:
- Query Vector
- Key Vector
- Value Vector
These vectors are created by multiplying the input embedding by three weight matrices that are trainable.
Check this out π
- Query Vector
- Key Vector
- Value Vector
These vectors are created by multiplying the input embedding by three weight matrices that are trainable.
Check this out π
Self-attention allows models to learn long-range dependencies between different parts of a sequence.
After acquiring keys, queries, and values, we merge them to create a new set of context-aware embeddings.
Take a look at this!π
After acquiring keys, queries, and values, we merge them to create a new set of context-aware embeddings.
Take a look at this!π
Implementing self-attention using PyTorch, doesn't get easier! π
It's very intuitive! π‘
Check this out π
It's very intuitive! π‘
Check this out π
Understanding LLMs & LLMOps is going to be a high leverage skill in future!
@LightningAI provides state of the art tutorials on LLMs & LLMOps, I have personally learnt a lot from there.
Everything is free & Open-sourced π₯
Check their blog π
lightning.ai
@LightningAI provides state of the art tutorials on LLMs & LLMOps, I have personally learnt a lot from there.
Everything is free & Open-sourced π₯
Check their blog π
lightning.ai
That's a wrap!
If you interested in:
- Python π
- Data Science π
- AI/ML π€
- LLMs π§
I'm sharing daily content over here, follow me β @akshay_pachaar if you haven't already!!
Cheers!! π
If you interested in:
- Python π
- Data Science π
- AI/ML π€
- LLMs π§
I'm sharing daily content over here, follow me β @akshay_pachaar if you haven't already!!
Cheers!! π
Loading suggestions...