Selenium

Chrome does not accept a proxy username and password on its command line, which is why so many Selenium proxy guides stop working. Recent Selenium 4 releases solve it through WebDriver BiDi: Selenium answers the gateway's login request for the browser.

Install#

pip install -U selenium

Selenium downloads a matching ChromeDriver on its own. Google Chrome must be installed.

Basic setup#

from selenium import webdriver
from selenium.webdriver.common.by import By

options = webdriver.ChromeOptions()
options.add_argument("--proxy-server=http://geo.gw.nyxon.sh:8080")
options.add_argument("--headless=new")
options.enable_bidi = True

driver = webdriver.Chrome(options=options)
driver.network.add_auth_handler("USERNAME-country-us", "PASSWORD")

driver.browsing_context.navigate(
    context=driver.current_window_handle,
    url="https://api.ipify.org?format=json",
    wait="complete",
)
print(driver.find_element(By.TAG_NAME, "body").text)

driver.get("https://example.com/")
print(driver.title)
driver.quit()

Three parts make it work:

• --proxy-server carries the host and port only, never the credentials.
• options.enable_bidi = True opens the BiDi connection when the browser starts.
• driver.network.add_auth_handler gives Selenium the username and password to answer the gateway with.

If driver.network has no add_auth_handler, your Selenium is too old. Upgrade it with the command above.

Load the first page with BiDi#

The first page must load through driver.browsing_context.navigate, as above. When the first login request arrives during driver.get, the handler's answer does not reach Chrome, and the page waits until Selenium raises a TimeoutException.

You need this once per browser. After the first login, Chrome remembers the credentials for the proxy, and driver.get works for every page that follows.

One browser per country or session#

Targeting and sessions go in the username, and the handler applies to the whole browser. For several countries or identities, start one browser for each:

import secrets
from selenium import webdriver
from selenium.webdriver.common.by import By


def new_browser(country, minutes=10):
    sid = secrets.token_hex(4)
    user = f"USERNAME-country-{country}-session-{sid}-time-{minutes}"
    options = webdriver.ChromeOptions()
    options.add_argument("--proxy-server=http://geo.gw.nyxon.sh:8080")
    options.add_argument("--headless=new")
    options.enable_bidi = True
    driver = webdriver.Chrome(options=options)
    driver.network.add_auth_handler(user, "PASSWORD")
    driver.browsing_context.navigate(
        context=driver.current_window_handle,
        url="https://api.ipify.org?format=json",
        wait="complete",
    )
    return driver


for country in ["us", "de", "gb"]:
    driver = new_browser(country)
    print(country, driver.find_element(By.TAG_NAME, "body").text)
    driver.quit()

Chrome keeps its connections to a site open, so pages on one site usually come from the same IP even without a session. For a fresh IP, start a new browser with a new session ID.

Skip the proxy for some sites#

List hosts that should go direct in --proxy-bypass-list, separated by semicolons. A leading *. matches subdomains:

options.add_argument("--proxy-bypass-list=*.internal.example;status.example.com")

SOCKS5#

Chrome supports SOCKS5 proxies only without a password, so use the HTTP port with Selenium. It still carries HTTPS sites end to end: Chrome opens a tunnel through the gateway and runs TLS inside it.

What no longer works#

• Credentials inside --proxy-server. Chrome refuses the proxy and every page shows ERR_NO_SUPPORTED_PROXIES, while driver.get returns as if nothing happened.
• Proxy login extensions loaded with --load-extension. Google Chrome now ignores that switch and logs --load-extension is not allowed in Google Chrome, ignoring.
• Selenium Wire. The project is archived and no longer maintained.

Errors#

Most of these arrive as exceptions in the form unknown error: net::ERR_.... The blank page and ERR_NO_SUPPORTED_PROXIES raise nothing, so check the page content as well as exceptions.

Starting a new project? Playwright takes proxy credentials at launch with no handler.