That doesn't seem surprising. When it was still at a "quirky" level, it didn't bother investors or much of the tech community, because it was all excusable as someone blowing off steam in a way that wasn't directly interfering with his work. Some people didn't like it because he was an asshole, but mostly just in a "ugh, that guy" kind of way that didn't interfere with them later deciding to buy a Tesla.

Then he bought Twitter, and suddenly his behavior was actively hurting his companies -- Tesla because he had to sell stock which affected prices and because he's tarnishing the brand by going political, and Twitter because he was making erratic decisions that drove away users and advertisers. So investors started to care.