1. Instead of creating a urls global variable, use start_requests function.
2. Don't use beautifulsoup to parse, use CSS or XPATH.
3. If you are going into multiple pages over and over again, use CrawlSpider with Rule.
It's way faster and has better support for CSS selectors.
This is my current setup as well, been scraping on and off for 20+ years now.
[0] http://go-colly.org/
By "not Colly" I just wanted to remark that in Go is relatively easy to write a crawler from scratch.
1. Instead of creating a urls global variable, use start_requests function.
2. Don't use beautifulsoup to parse, use CSS or XPATH.
3. If you are going into multiple pages over and over again, use CrawlSpider with Rule.