Python Selenium Get Data From Two Combo Boxes

Question

I have two combo boxes from which I need to get data. Let's call them 'Manufacturers' and 'Models'

When you select a manufacturer from one box the other box populates with the models that the manufacturer produces. This works fine manually but when I select a manufacturer programmatically with Selenium the 'Model' box does not re-populate. Here's the code.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
driver = webdriver.Firefox()
#Base URL
driver.get("https://example.com/")
def scrape():    
    #get manufacturer list
    select1 = driver.find_element_by_name("mnfr")
    makes = [x.text for x in select1.find_elements_by_tag_name("option")]
    print(makes)
    #get models list
    for m in makes:
        select1a = Select(driver.find_element_by_name("makeCodeListPlaceHolder"))           
        select1a.select_by_visible_text(m)        
        select2 = driver.find_element_by_name("models")
        models = [x.text for x in select2.find_elements_by_tag_name("option")]
        print(models)
scrape()

It's not easy to know without seeing the page but does `select1a.select_by_visible_text(m).click()` get you what you're after? — JimmyA, Mar 12 '19 at 21:37
why can't you just get the list of options rather `x.text` and click on option element, which will select the list item from `select1` listbox. — supputuri, Mar 12 '19 at 23:36
@DebanjanB Why was this answer marked as duplicate? The "original" answer was in Java. The question I asked was in Python. — Mike C., Mar 13 '19 at 13:55
@DebanjanB Only when the "binding arts" are not the same language. I can read between the lines, but maybe it would help someone else if the solution were explained in the language in which the question were asked. I speak English, but I'd hate for someone to teach me how to play a guitar using only Russian, although the guitar is played the same regardless of what language the instructor speaks. My two cents. — Mike C., Mar 13 '19 at 14:10
Have you tried adding a wait after setting the first dropdown? It's possible the page takes a fraction of a second to load the options into the second dropdown and so your code is running so fast that it grabs the second dropdown and prints 0 options before the options are even loaded. — JeffC, Mar 13 '19 at 19:49

score 1 · Answer 1 · answered Mar 12 '19 at 21:50

So there are some oddities with Selenium clicks not triggering all DOM events attached to an object. It normally occurs due to, let's say, un-optimally written front-end code.

But not making any judgements here, as Selenium can just encounter problems that it really shouldn't have.

So here is one suggestion. Go into the browser dev tools > console, and try the following example. I will write this in JQuery, but if you do not have JQuery, feel free to inject it on the page, or change the code to regular document.getElement code.

$("#FirstComboBoxOption").click();

Does that trigger all the expected events? Does the second Combobox populate correctly?

There is no dishonor using that to click an element. I use Selenium's click whenever possible, but Selenium's click is not literally identical to a real human click anyway, as the mouse/user32.dll is not used to perform a click. So using javascript, while not ideal, is not wrong if it is the only way to trigger all the events on the object.

The best situation is that the event logic is hooked up in a way that this is not necessary, but I am going to assume that is not an option here. Instead, I would do this (note that I use C# for Selenium, so this is a general guess on the syntax. Consider it kinda pseudo code)

browser.execute_script("$('#FirstComboBoxOption').click()");

Next, I am going to assume this doesn't work for you, so here is the next option.

First, find all events attached to the first combo box, that ostensibly are triggering the population of the second combo box. You can do this, using JQuery, like this:

var elem = $('#FirstComboBoxOption')[0];
$._data(elem, "events");

Do a little playing around to make sure you know what events are going to be triggered by a real click. Then, do the following:

$('#FirstComboBoxOption').trigger('SomeExpectedEvent');

Do one line for each of the events, if there is more than one event. I've only ever had to do the event example above once. It is a last resort. There really isn't anything wrong with it in the grand scheme of things, but whenever possible, it is obviously a superior option to trigger things using means as close to a real user's interactions as possible.

Is this a browser? Because if it is a browser, you have access to the javascript. Go to the Developer Tools. Then go to the console. You can run javascript from there. You do not need to see the existing javascript. It can be minified; doesn't matter. Since you have Firefox in the example, go here: 1) Menu, top right (hamburger menu) 2) Select Web Developer 3) Select Web Console — Asyranok, Mar 12 '19 at 21:56
Sorry. I didn't realize you could inject javascript like that. The JQuery didn't work in my instance but I'll play around with it. Thank you. — Mike C., Mar 12 '19 at 22:14

score 0 · Answer 2 · answered Mar 12 '19 at 23:41

Have you tried the below approach?

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
driver = webdriver.Firefox()
#Base URL
driver.get("https://example.com/")
def scrape():    
    #get manufacturer list
    select1 = driver.find_element_by_name("mnfr")
    makes = select1.find_elements_by_tag_name("option") # get the option elements
    print(makes)
    #get models list
    for m in makes:
        #click on list option
        m.click()
        select2 = driver.find_element_by_name("models")
        models = [x.text for x in select2.find_elements_by_tag_name("option")]
        print(models)
scrape()

Python Selenium Get Data From Two Combo Boxes

2 Answers2