Holding hands may also be from when babies latch on and don't let go because of the necessity of holding on to the parent. My friend's 2 year old grabbed my hand recently and it reminded me of their iron grip.

Any touch can be erotic, depending on the context. That's why I'm skeptical about kissing being special. People have sex with their entire bodies, not just using genitals.