I couldn't quickly remember how to not send my public keys at all, so I did "sudo -u nobody ssh ..." instead.