Web Scraping

Mobile Proxies with Puppeteer, Playwright & Selenium

The three major automation tools each handle proxies a little differently, and authentication is where everyone gets stuck. Working code for all three.

Published · Updated · 7 min read

The one rule before the code

Chromium, which powers Puppeteer, Playwright's default browser, and Selenium's ChromeDriver, has two proxy quirks that cause 90% of failed setups: it won't read credentials from the proxy URL, and it doesn't support authenticated SOCKS5 at all. The fix is the same everywhere: use your proxy's HTTP port and supply credentials through the tool's auth mechanism. (Where the protocols differ and why it doesn't matter here: SOCKS5 vs HTTP Proxy.)

Puppeteer

const browser = await puppeteer.launch({
  args: ['--proxy-server=http://your-endpoint:port'],
});

const page = await browser.newPage();
await page.authenticate({ username: 'user', password: 'pass' });

await page.goto('https://api.ipify.org');  // should print the carrier IP

page.authenticate() must run before the first navigation, since it answers the proxy's auth challenge. Call it on every new page.

Playwright

const browser = await chromium.launch({
  proxy: {
    server: 'http://your-endpoint:port',
    username: 'user',
    password: 'pass',
  },
});

// or per-context, one identity per context:
const context = await browser.newContext({
  proxy: { server: 'http://your-endpoint:port', username: 'user', password: 'pass' },
});

Playwright has the cleanest story: credentials are a first-class option, and setting the proxy per context lets one browser process run several isolated identities, each with its own cookies and its own proxy.

Selenium

# Python, unauthenticated or IP-whitelisted proxy:
options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=http://your-endpoint:port')
driver = webdriver.Chrome(options=options)

# With username/password, plain Selenium can't answer the auth
# dialog. Use selenium-wire instead, same API, auth just works:
from seleniumwire import webdriver
driver = webdriver.Chrome(seleniumwire_options={
    'proxy': {'http': 'http://user:pass@your-endpoint:port',
              'https': 'http://user:pass@your-endpoint:port'}
})

Selenium is the awkward one: vanilla Selenium has no way to answer a proxy auth prompt in headless mode. The practical options are selenium-wire (shown above), a provider that supports IP-whitelisting, or generating a tiny Chrome extension that injects credentials. selenium-wire is the least painful.

Verify, throttle, rotate

  • Verify first: load api.ipify.org before any real target and assert the IP is your proxy's, not your server's. A silent proxy failure that leaks your real IP is worse than a crash.
  • Throttle like a human: random 2–6s delays between navigations, realistic viewport and user agent. A mobile carrier IP buys tolerance; patient request patterns keep it.
  • Rotate between jobs, not requests: trigger a fresh carrier IP via API when a run finishes or a target starts pushing back, the endpoint stays the same, so your code doesn't change. When to rotate (and when not to) is its own topic: How Often Should You Rotate Proxy IPs?
  • Believe the browser over curl: if a request succeeds from the shell and stalls in Chrome through the same endpoint, that asymmetry usually isn't the proxy at all. It's MTU, and the reason it only shows up in the browser is QUIC.

Headless flags leak: default headless Chrome advertises itself in the user agent. Set a normal UA, a real viewport, and prefer the newer headless mode. The proxy hides your network, not your browser's confession.

Frequently asked questions

Why does my proxy work in curl but fail in headless Chrome?

Almost always authentication. Chromium doesn’t accept user:pass inside the --proxy-server flag, and it doesn’t support authentication for SOCKS5 proxies at all. Use the HTTP proxy port with page.authenticate() (Puppeteer) or the proxy credentials option (Playwright), and it works.

Should I use HTTP or SOCKS5 for browser automation?

For Chromium-based automation, use the HTTP proxy port, Chromium can’t do authenticated SOCKS5. The traffic is still end-to-end encrypted (HTTPS passes through a tunnel), so you lose nothing. SOCKS5 shines in tools that support its auth, like many HTTP clients and custom scripts.

One browser instance per proxy, or can I share?

One proxy per browser context is the clean pattern: each context gets its own cookies, fingerprint surface, and network identity. Playwright makes this easy since proxy can be set per context. Sharing one IP across many parallel contexts multiplies your request rate on that IP, the opposite of looking human.

How do I rotate the IP between scraping runs?

With a dedicated mobile proxy, trigger rotation via the provider’s API or dashboard between jobs, the phone reconnects and picks up a fresh carrier IP while your proxy endpoint stays the same. That means no proxy-list juggling in code: same host, same port, new identity.

One endpoint, endless fresh IPs

Point your automation at a PocketProxy HTTP endpoint on a real 4G/5G carrier IP, rotate by API between runs without touching your code.