I am trying to webscrape a website with scrapy (with a 10s download delay + AUTOTHROTTLE_ENABLED = True + ROBOTSTXT_OBEY = True). When I run the command:
scrape crawl myspider -o mydata.csv
I get this output for multiple "lines" until 405:
[protego] DEBUG: Rule at line 4 without any user agent to enforce it on.
and also:
[scrapy.spidermiddlewares.httperror] INFO: Ignoring response <405 HTTP status code is not handled or not allowed
Why can't scrapy scrape this website?
and this is the dumping stats (if useful):
2019-11-03 00:33:55 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 417,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 23402,
'downloader/response_count': 2,
'downloader/response_status_count/405': 2,
'elapsed_time_seconds': 13.42723,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 11, 2, 23, 33, 55, 12777),
'httperror/response_ignored_count': 1,
'httperror/response_ignored_status_count/405': 1,
'log_count/DEBUG': 61,
'log_count/INFO': 11,
'memusage/max': 51818496,
'memusage/startup': 51818496,
'response_received_count': 2,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/405': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2019, 11, 2, 23, 33, 41, 585547)}
1 Answer
You need to emulate exactly same request as a real browser does
headers = {
'Connection': 'keep-alive',
'Cache-Control': 'max-age=0',
'DNT': '1',
'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36',
'Sec-Fetch-User': '?1',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
'Sec-Fetch-Site': 'same-origin',
'Sec-Fetch-Mode': 'navigate',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9',
}
yield scrapy.Request(' headers=headers)
