Selenium Web Scraping With Python - Capturing "View More" Contents

1408 views python
5

I am new to this stuff - using Selenium to scrape customer reviews and other data of interest (number of stars given, date of review, how many found it useful etc)

I have it all working except some reviews are lengthy and have a "See More" button to reveal more text. I can't seem to capture this text. I am using driver.find_elements_by_xpath

Here is an example of the webpage with the "View More" on the first post: https://www.consumeraffairs.com/insurance/metlife_disabil.html?page=3

I try to scrape by numerous variations of: review_SeeMore = driver.find_elements_by_xpath('(//div[@class="js-collapsed"]/p)')

I don't think I should have to but I also tried clicking the button to scrape it as: review_SeeMore = driver.find_elements_by_xpath('(//div[@class="js-expanded"]/p)')

Can somebody please help. It's been driving me crazy for several hours

answered question

1 Answer

5

You could click on the link using selenium and then get the contents:

links = driver.find_elements_by_css_selector('a.exp-btn')

for link in links:
    link.click()

posted this

Have an answer?

JD

Please login first before posting an answer.