I've used crawlera for a few years, but not at the same scale. Might be worth checking out, anyway: https://scrapinghub.com/crawlera/