> Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines?

I'm genuinely unsure if my own brain is any different.