TL;DR
A 407 Proxy Authentication Required error means the proxy server, not the target website, rejected your request because it lacks valid proxy credentials. In web scraping it shows up when the username and password are missing or wrong, the proxy plan expired, or your IP is not on the provider's allowlist.
Reading a 407 Correctly
Start from what 407 rules out: the target site never saw your request. The failure sits entirely between you and your proxy provider, so header tuning and backoff cannot fix it. That makes 407 one of the fastest errors to diagnose in a scraping stack, as long as you do not mistake it for a block.
The Four Usual Causes
Malformed credentials. Proxy URLs embed auth as http://user:pass@host:port. Special characters in the password must be URL-encoded, and a stray newline from an environment variable is enough to fail authentication.
Wrong auth mode. Providers support username/password auth, IP allowlisting, or both. If your account is set to IP allowlist and you connect from a new machine (or your office IP changed), every request 407s until the new IP is registered.
Expired or exhausted plan. Some providers answer over-quota traffic with 407 rather than 402, because the auth layer is where they enforce billing.
Session parameters in the username. Residential providers pack options into the username (user-session-abc123-country-us). A typo in these flags fails auth even with a correct password.
Debugging a 407
Test the proxy in isolation before touching scraper code:
curl -x "http://USER:PASS@proxy.example.com:8000" \
-w "%{http_code}\n" -o /dev/null -s \
https://httpbin.org/ipA 200 here means the proxy works and the bug is in how your scraper builds the proxy URL. A 407 here means the problem is the account: check the credentials, the auth mode, and the plan status in the provider dashboard, in that order.
Key Takeaways
- 407 comes from the proxy layer; the target site is not involved.
- Check credentials encoding, auth mode (password vs IP allowlist), and plan status.
- Test with a bare
curl -xcall to separate account problems from code problems.
How ScrapeGraphAI Handles This
ScrapeGraphAI removes the proxy layer from your side of the stack entirely. Requests to the scrape endpoint run through managed, pre-authenticated proxy pools, so credential rotation, allowlists, and pool health are handled by the platform and 407s disappear from your logs.