1

I'm scrapping a page in selenium (using the edge driver) and I've noticed discrepancies between the information under the elements tab in the dev window found here:

enter image description here

and the data returned from browser.page_source(), is there a way to get the data from the elements tab?

Isiah
  • 195
  • 2
  • 15
  • 1
    Does [this](https://stackoverflow.com/a/71763545/7429447) or [this](https://stackoverflow.com/a/71812637/7429447) or [this](https://stackoverflow.com/a/71699106/7429447) discussion answers your question? – undetected Selenium Apr 19 '22 at 21:43

1 Answers1

0

I think some websites block or restrict the selenium user agent.

Change user agent:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()

options.add_argument("user-agent=Something")

driver = webdriver ...

You can get random user-agent with fake-useragent module.

from fake_useragent import UserAgent

user_agent = UserAgent()
print(user_agent.random)

install:

pip install fake-useragent
Hamidreza Khorammfar
  • 661
  • 1
  • 8
  • 19