2) The details say that the performance numbers require 2.7x the overall power, so around 13.5W. Power needed for all the memory is not explicit in the numbers.
3) The compiler is also simulated and it's a prototype, so it's not a drop-in replacement for anything like Nvidia ISAAC
4) 5W seems to refer to sub-4B models, so not the 100B param ones.
This mentions targets for inference of 10 to 30tps and I (albeit just skimming through) didn't find any info about ingestion speed. They also mention that the 1st gen chip is used for vision, though I suspect it's only for specific cases and not anything general.
Overall I like power efficient optimizations like this and let's hope that this is adopted broadly. Although the compute in memory sounds like a big increase in memory prices
1) The performance numbers are simulated
2) The details say that the performance numbers require 2.7x the overall power, so around 13.5W. Power needed for all the memory is not explicit in the numbers.
3) The compiler is also simulated and it's a prototype, so it's not a drop-in replacement for anything like Nvidia ISAAC
4) 5W seems to refer to sub-4B models, so not the 100B param ones.
Overall I like power efficient optimizations like this and let's hope that this is adopted broadly. Although the compute in memory sounds like a big increase in memory prices