I'm trying to scrap the content of all the categories of a web page. For each category, I need to click on the category button, wait until the content of the category is displayed, and scrap the content.

To wait until the content is displayed, I'm using this code:

driver = webdriver.Chrome()

driver.get(')

content = driver.page_source
soup = BeautifulSoup(content)

categories = driver.find_elements(By.XPATH, "//li[@data-yy='category']")

for category in categories:
  category.click()

  WebDriverWait(driver, timeout=30, poll_frequency=2).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-yy='item']")))

  content = driver.page_source
  soup = BeautifulSoup(content)

  # Followed by BeautifulSoup code...

It works well for the first 15 categories, but for the 16th category, the program crashes while the page is loading. No matter what category the 16h one is, no matter the order of the categories, the 16th one always crashes, without going to timeout, and it gives me the following exception:

from unknown error: cannot determine loading status
from tab crashed

Stacktrace:
0   chromedriver                        0x0000000104b8d598 chromedriver + 4404632
1   chromedriver                        0x0000000104b14fa3 chromedriver + 3911587
2   chromedriver                        0x00000001047c1d20 chromedriver + 425248
...
26  chromedriver                        0x0000000104b948fe chromedriver + 4434174
27  libsystem_pthread.dylib             0x00007ff80c1294e1 _pthread_start + 125
28  libsystem_pthread.dylib             0x00007ff80c124f6b thread_start + 15

WebDriverException                        
Traceback (most recent call last)
scraping_attributes.ipynb Cellule 6 in <cell line: 4>()
     10   continue
     12 category.click()
---> 14 WebDriverWait(driver, timeout=30, poll_frequency=2).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-yy='item']")))
     16 content = driver.page_source
     17 soup = BeautifulSoup(content)

File /Library/Frameworks/ in WebDriverWait.until(self, method, message)
     79 while True:
     80     try:
---> 81         value = method(self._driver)
     82         if value:
     83             return value

File /Library/Frameworks/ in visibility_of_element_located.<locals>._predicate(driver)
    123 def _predicate(driver):
    124     try:
--> 125         return _element_if_visible(driver.find_element(*locator))
    126     except StaleElementReferenceException:
    127         return False

File /Library/Frameworks/ in WebDriver.find_element(self, by, value)
    853     by = By.CSS_SELECTOR

When I try to use time.sleep(10) instead of the WebDriverWait it works. But it takes much more time...

• Can this be a memory issue ? It always seem to be at the 16th category.
• Or maybe it's linked to the website itself ?
• Have any of you ever faced this issue while using Selenium in Python ?

Thank you!