Basic neural nets are like a 'bag of points' arranged in layers. each point has a 'weight' that takes the input, applies math using that weight to get an output. (It learns by adjusting the weight so that the 'inputs' will calculate to 'expected' outputs.
The next stage is called 'LSTM' (Long-Short Term Memory), this is like the 'bag of points' but has a 'Long memory' gate, a 'short memory' gate, and a 'forget' gate. So, when you type a phrase, it looks at the words before and tries to guess at what's coming next and will 'forget' options so that it doesn't go back to the previous guess.
Transformers take the whole document and create 'key-value' pairs that are also trained like the 'bag of points'. the benefit of this is that the 'transformer' can begin to learn the context based on the other features. This is a tricky one to explain, but is the first type of AI system that's capable of grasping context of what is 'scanned'.
He's talking about Neural Nets / AI.
Basic neural nets are like a 'bag of points' arranged in layers. each point has a 'weight' that takes the input, applies math using that weight to get an output. (It learns by adjusting the weight so that the 'inputs' will calculate to 'expected' outputs.
The next stage is called 'LSTM' (Long-Short Term Memory), this is like the 'bag of points' but has a 'Long memory' gate, a 'short memory' gate, and a 'forget' gate. So, when you type a phrase, it looks at the words before and tries to guess at what's coming next and will 'forget' options so that it doesn't go back to the previous guess.
Transformers take the whole document and create 'key-value' pairs that are also trained like the 'bag of points'. the benefit of this is that the 'transformer' can begin to learn the context based on the other features. This is a tricky one to explain, but is the first type of AI system that's capable of grasping context of what is 'scanned'.