scrapy httpcache刷新重试


stackoverflow这个帖子写的很清楚了:I crawl a webpage page by page, by following urls in pagination. In some pages, website detects that I use a bot and gives me an error in html. Since it is a successful request, it caches the page and when I run it again, I get the same error.