I think the limitation is chip size/cost. SRAM is a lot less dense than RAM. According to Google this is typically used for registers and caches, which are only megabytes large.

They're using hundreds of chips. Based on the data sheet I would estimate this demo uses 173 KW. It may be 100x energy to get 10x speedup.

100s of chips for who knows how many clients. The mobile phone will have to do calculations just for 1 client.

Yes, we pipeline requests so multiple users are being handled by the same hardware at one time.

Thanks for the clarification. So, would you say that Groq has a potential to have let's say OpenAI speeds on handheld devices at reasonable energy consumption? Or is that not really what this tech's strength is maybe?

The industry as a whole is a very long way away from that. The power requirements are too high for mobile.