Python Selenium Headless Chrome: User Agent and Proxy

When you run Selenium with headless Chrome, the browser sends a default user agent that identifies it as an automated Chromium instance. Many sites block or throttle that agent. Similarly, if you need to route traffic through a proxy, headless Chrome requires explicit configuration through ChromeOptions. This article shows how to set both a custom user agent and a proxy in Selenium's headless Chrome mode, and what to watch out for when doing so.

Setting Up Headless Chrome with Selenium

Before changing the user agent or proxy, you need a working headless Chrome instance. The minimal setup uses webdriver.Chrome with Options and the --headless argument.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless")

driver = webdriver.Chrome(options=options)

This launches Chrome without a visible window. Headless mode has changed over Chrome releases, so if you rely on a specific Chrome version, check its documentation for the exact flag.

With this base, you can add arguments for the user agent and proxy.

Changing the User Agent in Headless Chrome

The default user agent in headless Chrome often includes the string HeadlessChrome instead of Chrome, which makes it easy for servers to detect automation. To replace it, pass the --user-agent argument.

options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")

This sets the User-Agent header for requests made by the browser, including navigation and resource loads. It also changes the value returned by navigator.userAgent, so JavaScript sees the same custom string.

Use a realistic user agent that matches the operating system and browser version you want to emulate. Copying an existing agent string from a real browser is common, but be aware that the string alone does not make your traffic indistinguishable from a real browser. Headless Chrome still has other detectable properties, such as navigator.webdriver being true by default. To suppress that, you can add the --disable-blink-features=AutomationControlled argument.

options.add_argument("--disable-blink-features=AutomationControlled")

This does not change the user agent but removes the most obvious automation flag. Combine it with a custom user agent for a more realistic fingerprint.

Configuring a Proxy for Headless Chrome

You can configure a proxy for Chrome by passing the --proxy-server command-line argument, which accepts the proxy address and port.

options.add_argument("--proxy-server=http://proxy.example.com:8080")

For an HTTP proxy, use the http:// scheme. For a SOCKS proxy, use socks5://.

options.add_argument("--proxy-server=socks5://proxy.example.com:1080")

When you set this argument, traffic from the browser, including HTTP and HTTPS requests, goes through the proxy. There is no need to configure separate proxy settings inside the browser profile.

If you need to bypass the proxy for certain hosts, Chrome supports the --proxy-bypass-list argument. The list is comma-separated. For example, to exclude localhost and an internal domain:

options.add_argument("--proxy-bypass-list=localhost,127.0.0.1,*.internal.example.com")

This is useful when you want the proxy only for external traffic.

Combining User Agent and Proxy in One Session

Both arguments are independent, so you can add them to the same Options object.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless")
options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("--proxy-server=http://proxy.example.com:8080")

driver = webdriver.Chrome(options=options)

This configuration gives you a headless browser that sends a custom user agent and routes all traffic through the specified proxy. The order of arguments does not matter; Chrome applies all of them.

One thing to keep in mind is that the proxy is set at the browser level, not per request. If you need different proxies for different requests within the same session, a common approach is to create separate driver instances, each with its own proxy.

Handling Proxy Authentication

Many proxies require a username and password. The --proxy-server argument does not have a separate credential option, but you can try embedding credentials in the URL.

Embedding credentials in the proxy URL works for HTTP proxies in many cases:

options.add_argument("--proxy-server=http://username:password@proxy.example.com:8080")

However, this approach is not reliable across all Chrome versions and proxy types. Some proxies respond with a 407 status and expect the browser to show an authentication dialog. In headless mode, that dialog cannot appear, and the request fails.

A more robust method is to use a Chrome extension that handles proxy authentication. You create a small extension with a background script that sets the proxy and provides credentials. Then you load that extension with Selenium.

import json
import tempfile
import pathlib
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

manifest = {
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Proxy Auth",
    "permissions": ["proxy", "tabs", "storage", "webRequest", "webRequestBlocking"],
    "background": {
        "scripts": ["background.js"]
    }
}

background_js = """
var config = {
    mode: "fixed_servers",
    rules: {
        singleProxy: {
            scheme: "http",
            host: "proxy.example.com",
            port: 8080
        },
        bypassList: ["localhost"]
    }
};
chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
    return {
        authCredentials: {
            username: "username",
            password: "password"
        }
    };
}
chrome.webRequest.onAuthRequired.addListener(
    callbackFn,
    {urls: ["<all_urls>"]},
    ["blocking"]
);
"""

with tempfile.TemporaryDirectory() as tmp:
    pathlib.Path(tmp, "manifest.json").write_text(json.dumps(manifest))
    pathlib.Path(tmp, "background.js").write_text(background_js)

    options = Options()
    options.add_argument("--headless")
    options.add_argument("--load-extension=" + tmp)

    driver = webdriver.Chrome(options=options)

This extension uses the Chrome proxy API to set a fixed proxy and the webRequest API to respond to authentication challenges. The --load-extension argument loads the extension in headless mode. Test the extension with your specific Chrome version because headless mode and extension support have changed across releases. Also note that this example uses Manifest V2; if your Chrome version no longer supports Manifest V2, adjust the manifest to match current extension policies.

The extension approach can be more reliable for proxies that require authentication in Chrome versions that support it, but it adds complexity. For public proxies or proxies without authentication, the simple --proxy-server argument is sufficient.

Common Failure Modes and How to Diagnose Them

When you set a user agent or proxy, you may encounter issues that are not immediately obvious. Here are the most common problems and how to identify them.

Proxy connection refused or timeout

If the proxy address is wrong or the proxy is down, Chrome will fail to load pages. The error appears in the browser console or as a WebDriverException with a message like ERR_PROXY_CONNECTION_FAILED. Check that the proxy host and port are correct and reachable from your network.

Proxy authentication required

If the proxy requires credentials and you did not provide them, you may get a 407 status or a page that never loads. Use the extension method described above to handle authentication.

User agent not applied

If you inspect the request headers and see the default headless user agent, the --user-agent argument might be overridden by another setting. Verify the value that the browser is using by printing driver.execute_script("return navigator.userAgent") after starting the driver.

Headless mode detection

Even with a custom user agent, sites may detect headless Chrome through other signals. The navigator.webdriver property is the most common. Adding --disable-blink-features=AutomationControlled helps, but it is not a complete solution. Some sites use more advanced fingerprinting, such as checking for missing plugins or the behavior of the window.chrome object. In such cases, you may need a more sophisticated approach, such as a patched browser or a tool like Puppeteer with stealth plugins.

Proxy bypass not working

The --proxy-bypass-list argument uses a comma-separated list of hostnames. If you use semicolons or spaces, Chrome may ignore the list. Also, the list applies only to the proxy set via --proxy-server, not to extensions.

Operational Considerations for Production Scraping

When you run headless Chrome with a custom user agent and proxy in production, the setup has several operational implications.

Resource usage

Each webdriver.Chrome instance consumes memory and CPU. Headless mode reduces visual overhead but still runs a full browser engine. If you need many concurrent sessions, consider using a pool of driver instances or a browser automation framework that supports parallel execution.

Proxy reliability

A single proxy is a single point of failure. If the proxy goes down, all requests fail. For production scraping, use a pool of proxies and rotate them per request or per session. This also helps distribute load and reduce the chance of IP-based rate limiting.

User agent rotation

Using the same user agent for all requests makes your traffic easier to fingerprint. Rotate user agents from a list, ideally matching the operating system and browser version you claim to use. However, rotating too frequently may trigger anti-bot systems. A common pattern is to change the user agent per session, not per request.

Logging and observability

When you set a proxy, you lose the direct connection between the browser and the target server. This makes debugging harder. Log the proxy address and user agent for each driver session. If a request fails, you can correlate the failure with the specific proxy and agent combination.

Compatibility with Chrome versions

Chrome updates frequently, and command-line arguments sometimes change. The --proxy-server and --user-agent arguments have been stable for many years, but headless mode and extension support have changed. Test your setup after Chrome updates to ensure nothing broke.

Security

If you embed proxy credentials in the --proxy-server URL, they may appear in process listings or logs. Avoid this in production. Use the extension method or a secure configuration store. Also, be careful with proxies that are not under your control; they can see all unencrypted traffic.