Could you give an example of "conceptual understanding" that humans have which is necessary for safe autonomous driving? How does autonomous driving differ from, say, playing poker when it comes to implementing a machine learning solution?

Playing poker is trivial to automate, it’s essentially fully solved.

Driving is often surprisingly social. I had many instances where I observed somebody pulling out of a driveway in front of me and something about how their head was turned or whatnot meant I subconsciously “knew” in a split second that they were about to go full speed without looking, so I had to slow down. Stop sign tie-breaking gets complicated. Pedestrian/bike rider eye contact is a thing. “Do I know that they know that I know what their intentions are?” And so on.

It’s not as complex as fully solving human communication, but it seems like a huge challenge when mixing human and automated driving on the same roads.

But I also don’t know if there were similar transition concerns when horse buggies were being replaced with cars and people had wrong expectations?

Maybe we should paint these cars in bright orange and have flashing lights on all the time so that people learn to expect the different behaviour. (The huge Lidar rig on top of some self driving cars works to that purpose)

The reason we humans can understand very complex situations with very little information very quickly (such as navigating through traffic) is that we not only perceive that "stuff" is out there and moving, but we also instantly classify that stuff, and in so doing, understand what it can and cant do almost instantly. A large semi truck can only maneuver in certain ways and certain speeds, a car moving erratically likely has a driver that is somehow impaired, a child is fully upredictable and could jump in the street, etc. We know about not just that stuff is out there, but what that stuff is, and how it causally can behave given how things have been behaving up to that point.