When you want to scrape javascript generated content from a website you will realize that Scrapy or other web scraping libraries cannot run javascript code while scraping. ... If you need to enable JS to reach the data you want there's not much you can do but use a headless or lightweight browser to load data for scraping.

How do I enable JavaScript on Scrapy?

3 Answers. Scraping hub which is the same company behind Scrapy, has special instances to run your spiders with splash enabled. Then yield SplashRequest instead of Request in your spider like this: import scrapy from scrapy_splash import SplashRequest class MySpider(scrapy.

Can you scrape websites with JavaScript?

Thanks to Node. js, JavaScript is a great language to use for a web scraper: not only is Node fast, but you'll likely end up using a lot of the same methods you're used to from querying the DOM with front-end JavaScript.