"the impression I always got was that the engineers/algorithm guys were always second fiddle to the traders"

This is mostly a stereotype of working at a large iBank. Proprietary trading shops and hedge funds have very rarely worked that way, especially after the rise of electronic trading. In fact, most often now, if someone is coming from one of those environments with the title "trader" they were most likely a clerk that was doing a lower paying job that just wasn't worth automating.

My time in finance generally and algo trading specifically was much more "nerd amongst nerds" than any other development job I've ever had.