Director of AI at Tesla. Previously a research scientist at OpenAI and CS PhD student at Stanford. I like to train deep neural nets on large datasets 🧠🤖💥
Andrej Karpathy @karpathy
·
Mar 15, 2022
Excellent and unintuitive read on GPUs. The chip doing the compute has tiny amount of memory & is connected to the main memory literally through a straw. Most of the energy goes to data movement too. Many repercussions. E.g. latency better predicted by # activations than # flops