# How to Rotate Proxies in Python: A Practical Guide

Sooner or later, every serious scraping or data-collection project runs into the same problem: a single proxy (or your own IP address) isn't enough. It goes down, it gets slow, or you simply need to see a site from several locations. The answer is **proxy rotation**: spreading your requests across a pool of proxies instead of relying on just one.

In this guide you'll learn how to rotate proxies in Python, starting with a few lines of `requests` code and building up to a reusable rotator class that retries failed requests, benches broken proxies, and works with async code too.

> **Heads-up:** all example proxy addresses below (`203.0.113.x`) come from a range reserved for documentation. Replace them with proxies you actually have access to.

* * *

## Why Rotate Proxies?

There are plenty of legitimate reasons to use a pool instead of a single proxy:

-   **Reliability.** If one proxy dies, your job keeps running through the others.
    
-   **Geo-testing.** Check how a page, price, or ad renders from different countries.
    
-   **Spreading permitted workloads.** Large, authorized data collection can be distributed across many exit points instead of funneling through one.
    
-   **Privacy.** Avoid tying all your traffic to a single address.
    

## Use Proxies Responsibly

Rotation is a reliability tool, not a license to ignore a site's rules. Keep these principles in mind:

-   **Follow the site's Terms of Service and** `robots.txt`**.** Rotating your IP doesn't make prohibited scraping permitted.
    
-   **Treat "slow down" as an instruction.** If a site rate-limits you (HTTP 429), the right response is to back off, not to hit it from a fresh IP. The code in this post honors `Retry-After` for exactly that reason.
    
-   **Don't rotate your way past a block.** A persistent `403 Forbidden` means the site doesn't want your traffic. Look for an official API or ask for permission.
    
-   **Never use proxies to access accounts, data, or systems you aren't authorized to use.**
    

## Proxy Basics

### Types of proxies

| Type | Strengths | Trade-offs |
| --- | --- | --- |
| Datacenter | Fast and inexpensive | Easy for sites to identify as non-residential |
| Residential | IPs belong to real home connections | Slower and pricier |
| Mobile | Carrier-grade IPs | Usually the most expensive |
| Free public lists | No cost | Unreliable, slow, and potentially unsafe (see below) |

> **A word of caution on free proxies:** whoever runs a proxy can see (and potentially tamper with) any unencrypted traffic passing through it, and can log metadata even for HTTPS. Never send passwords, API keys, or personal data through a proxy you don't trust.

### Proxy URL format

Python libraries describe a proxy with a single URL:

```text
scheme://username:password@host:port
```

The scheme is usually `http` (works for both HTTP and HTTPS sites via tunneling) or `socks5`. Authentication is optional.

## Using a Single Proxy with `requests`

Install the library first:

```bash
pip install requests
```

Then pass a `proxies` dictionary to any request:

```python
import requests

proxy_url = "http://user:pass@203.0.113.10:8080"
proxies = {"http": proxy_url, "https": proxy_url}

response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)
print(response.json())      # {'origin': '203.0.113.10'}
```

The [httpbin.org/ip](https://httpbin.org/ip) endpoint echoes the IP address it sees, which makes it handy for confirming that your traffic really goes through the proxy.

A few details worth knowing:

-   Both dictionary keys usually point to the same `http://` proxy URL. The key says which _target_ scheme the proxy is used for; the proxy URL's own scheme describes how you talk to the _proxy_.
    
-   `requests` also reads the `HTTP_PROXY` and `HTTPS_PROXY` environment variables, but a `proxies=` argument you pass explicitly takes precedence.
    
-   Always set a `timeout`. Dead proxies often hang rather than fail fast.
    

### Special characters in credentials

If your username or password contains characters like `@`, `:` or `/`, URL-encode them. Otherwise the proxy URL is parsed incorrectly and `requests` typically fails with an `InvalidURL` error:

```python
from urllib.parse import quote

username = quote("my.user@example.com", safe="")
password = quote("p@ss:word/1", safe="")
proxy_url = f"http://{username}:{password}@203.0.113.10:8080"

print(proxy_url)
```

Better still, don't hardcode credentials at all. Load them from environment variables or a secrets manager.

### SOCKS proxies

For SOCKS support, install the extra and change the scheme:

```bash
pip install "requests[socks]"
```

```python
proxy_url = "socks5h://user:pass@203.0.113.10:1080"
```

The `socks5h` variant resolves DNS through the proxy as well, which avoids leaking your lookups to your own resolver.

## Rotation Strategies

### 1\. Random choice

The simplest approach: pick any proxy for each request.

```python
import itertools
import random

PROXIES = [
    "http://user:pass@203.0.113.10:8080",
    "http://user:pass@203.0.113.11:8080",
    "http://user:pass@203.0.113.12:8080",
]

def random_proxy():
    return random.choice(PROXIES)
```

It's easy, but you can't guarantee an even spread, and the same proxy may be chosen several times in a row.

### 2\. Round robin

`itertools.cycle` walks through the list in order and loops forever, so every proxy gets an equal share:

```python
proxy_cycle = itertools.cycle(PROXIES)

def next_proxy():
    return next(proxy_cycle)
```

### 3\. Let the provider rotate for you

Many paid services expose a single "gateway" endpoint that assigns a different exit IP behind the scenes. In that case your code needs no rotation logic at all:

```python
gateway = "http://user:pass@gateway.example-proxy.com:8000"
proxies = {"http": gateway, "https": gateway}
```

Check your provider's documentation. Some rotate on every request, others keep a "sticky" IP for a few minutes, which is useful when a multi-step flow (like paging through results) should come from one address.

### Which should you use?

| Strategy | Best for | Weakness |
| --- | --- | --- |
| Random | Quick scripts | Uneven distribution |
| Round robin | Fair, predictable spread | Doesn't notice dead proxies |
| Provider gateway | Large pools with zero maintenance | Costs money, less control |
| Rotator with health tracking | Anything you'll run repeatedly | A bit more code (below) |

## Building a Reusable Rotator

Random and round-robin both share a flaw: they keep handing out a proxy even after it has stopped working. A better rotator tracks failures and temporarily benches proxies that misbehave.

```python
import time
from dataclasses import dataclass
from urllib.parse import urlsplit

@dataclass
class Proxy:
    url: str
    failures: int = 0
    banned_until: float = 0.0

    @property
    def label(self) -> str:
        """Host and port only, so credentials never end up in logs."""
        return urlsplit(self.url).netloc.rpartition("@")[2]

class ProxyRotator:
    def __init__(self, proxy_urls, max_failures=3, cooldown=300):
        self.proxies = [Proxy(url) for url in proxy_urls]
        self.max_failures = max_failures      # consecutive failures before benching
        self.cooldown = cooldown              # seconds a benched proxy sits out
        self._counter = 0

    def get(self) -> Proxy:
        """Return the next healthy proxy in round-robin order."""
        now = time.time()
        healthy = [p for p in self.proxies if p.banned_until <= now]
        if not healthy:
            raise RuntimeError("No healthy proxies left")
        proxy = healthy[self._counter % len(healthy)]
        self._counter += 1
        return proxy

    def report_success(self, proxy: Proxy) -> None:
        proxy.failures = 0

    def report_failure(self, proxy: Proxy) -> None:
        proxy.failures += 1
        if proxy.failures >= self.max_failures:
            proxy.banned_until = time.time() + self.cooldown
            proxy.failures = 0
```

What this gives you:

-   **Round-robin rotation** over whichever proxies are currently healthy.
    
-   **Automatic benching.** After `max_failures` failures in a row, a proxy sits out for `cooldown` seconds, then rejoins the pool.
    
-   **Safe logging.** The `label` property prints only host and port, never your username and password.
    

> This class isn't thread-safe. If you share one rotator across threads, wrap `get()`, `report_success()` and `report_failure()` in a `threading.Lock`.

## Fetching with Automatic Failover

Now pair the rotator with a request function that switches proxies when one fails:

```python
import random
import time

import requests

HEADERS = {"User-Agent": "MyLearningScraper/1.0 (educational project)"}
PROXY_ERROR_STATUS = {407, 502, 504}     # what a proxy typically returns when it fails
SITE_BUSY_STATUS = {429, 500, 503}       # the site itself is asking you to back off

def fetch(url, rotator, max_attempts=5, timeout=10):
    """GET a URL, switching to a different proxy whenever one fails."""
    last_error = None

    for attempt in range(1, max_attempts + 1):
        proxy = rotator.get()
        try:
            response = requests.get(
                url,
                headers=HEADERS,
                proxies={"http": proxy.url, "https": proxy.url},
                timeout=timeout,
            )
        except requests.RequestException as exc:   # ProxyError, timeouts, SSL errors...
            last_error = exc
            rotator.report_failure(proxy)
            print(f"Attempt {attempt}: {proxy.label} failed ({type(exc).__name__})")
            time.sleep(random.uniform(0.5, 1.5))
            continue

        if response.status_code in SITE_BUSY_STATUS:     # the site says "slow down": respect it
            retry_after = response.headers.get("Retry-After", "")
            delay = int(retry_after) if retry_after.isdigit() else 2 ** attempt
            print(f"Attempt {attempt}: site returned {response.status_code}, waiting {delay}s")
            time.sleep(min(delay, 60))
            continue

        if response.status_code in PROXY_ERROR_STATUS:   # bad proxy login, or proxy can't reach the site
            rotator.report_failure(proxy)
            print(f"Attempt {attempt}: {proxy.label} returned {response.status_code}")
            time.sleep(random.uniform(0.5, 1.5))
            continue

        rotator.report_success(proxy)
        response.raise_for_status()                # 403, 404...: don't rotate, just raise
        return response

    raise RuntimeError(f"All {max_attempts} attempts failed") from last_error
```

The key decisions here:

-   **Network-level errors** (connection refused, timeouts, TLS failures) count against the proxy and trigger an immediate switch.
    
-   **407, 502 and 504 responses** are what proxies typically return when your login is wrong or they can't reach the site, so they count against the proxy.
    
-   **429, 500 and 503 come from the _site_,** which is asking you to slow down or is having trouble. The code waits (honoring `Retry-After` when provided) and doesn't blame the proxy, so a site outage can't bench your whole pool.
    
-   **Other 4xx errors** such as 403 or 404 are raised straight away. Retrying them through a different proxy would just be hammering a site that already said no.
    

## Plugging It into a BeautifulSoup Scraper

If you've built a scraper with BeautifulSoup, swapping in rotation is a one-line change: replace your `requests.get` call with `fetch`.

```python
from bs4 import BeautifulSoup

PROXY_URLS = [
    "http://user:pass@203.0.113.10:8080",
    "http://user:pass@203.0.113.11:8080",
    "http://user:pass@203.0.113.12:8080",
]

rotator = ProxyRotator(PROXY_URLS)

response = fetch("https://books.toscrape.com/", rotator)
soup = BeautifulSoup(response.content, "html.parser")
print(soup.title.get_text(strip=True))
```

Because the rotator keeps its state between calls, you can loop over hundreds of pages and the load will spread evenly across your healthy proxies.

## Filtering Out Dead Proxies Before You Start

Proxy lists, especially free ones, are full of dead entries. It's cheaper to weed them out up front than to burn retries on them mid-crawl.

```python
from concurrent.futures import ThreadPoolExecutor

import requests

def load_proxies(path):
    """Read one proxy URL per line, ignoring blank lines and # comments."""
    with open(path, encoding="utf-8") as f:
        lines = (line.strip() for line in f)
        return [line for line in lines if line and not line.startswith("#")]

def check_proxy(proxy_url, timeout=8):
    """Return the exit IP if the proxy works, otherwise None."""
    try:
        response = requests.get(
            "https://httpbin.org/ip",
            proxies={"http": proxy_url, "https": proxy_url},
            timeout=timeout,
        )
        response.raise_for_status()
        return response.json()["origin"]
    except (requests.RequestException, ValueError, KeyError):
        return None

def healthy_proxies(candidates, workers=10):
    """Test proxies in parallel and keep only the ones that respond."""
    with ThreadPoolExecutor(max_workers=workers) as pool:
        exit_ips = list(pool.map(check_proxy, candidates))
    return [url for url, ip in zip(candidates, exit_ips) if ip]
```

Use it like this:

```python
candidates = load_proxies("proxies.txt")
rotator = ProxyRotator(healthy_proxies(candidates))
```

## Rotating Proxies Asynchronously

When you need many requests in flight at once, `aiohttp` accepts a `proxy` argument on each request, which makes rotation straightforward:

```bash
pip install aiohttp
```

```python
import asyncio
import itertools

import aiohttp

PROXIES = [
    "http://user:pass@203.0.113.10:8080",
    "http://user:pass@203.0.113.11:8080",
    "http://user:pass@203.0.113.12:8080",
]
proxy_cycle = itertools.cycle(PROXIES)

async def fetch(session, url, proxy, semaphore):
    async with semaphore:                          # cap how many requests run at once
        try:
            timeout = aiohttp.ClientTimeout(total=10)
            async with session.get(url, proxy=proxy, timeout=timeout) as resp:
                resp.raise_for_status()
                return url, resp.status, await resp.text()
        except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
            return url, None, type(exc).__name__

async def main(urls):
    semaphore = asyncio.Semaphore(5)
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, url, next(proxy_cycle), semaphore) for url in urls]
        return await asyncio.gather(*tasks)

if __name__ == "__main__":
    results = asyncio.run(main(["https://httpbin.org/ip"] * 6))
    for url, status, body in results:
        print(status, body[:60])
```

Concurrency multiplies your request rate, so keep the semaphore small and stay within what the target site allows.

### A quick note on `httpx`

If you prefer `httpx`, create a client with the `proxy` argument:

```python
import httpx

with httpx.Client(proxy=random_proxy(), timeout=10) as client:
    print(client.get("https://httpbin.org/ip").json())
```

Older tutorials use a `proxies=` argument, which was removed in `httpx` 0.28. Use `proxy=` on current versions. For rotation, build a new client per proxy.

## Best Practices

-   **Add delays and jitter.** Pause between requests, ideally with a random component, so you don't hit a site in lockstep.
    
-   **Rotate for reliability, not to dodge limits.** Honor `429` and `Retry-After`, and stop if you keep getting `403`.
    
-   **Always set timeouts.** A proxy that hangs is worse than one that fails.
    
-   **Keep sessions consistent.** If a site uses cookies, create one `requests.Session` per proxy so each exit IP carries a coherent set of cookies.
    
-   **Monitor your pool.** Log successes and failures per proxy (by `label`, never with credentials) and replace the chronic offenders.
    
-   **Protect credentials.** Keep them out of source control and logs.
    
-   **Prefer HTTPS targets.** TLS keeps the content of your traffic private from the proxy operator.
    

## Troubleshooting

| Symptom | Likely cause | What to try |
| --- | --- | --- |
| ProxyError / connection refused | Proxy is down, or wrong host or port | Test it with check_proxy; verify the address |
| 407 Proxy Authentication Required | Wrong credentials, or special characters not URL-encoded | Re-check them and use urllib.parse.quote |
| ConnectTimeout / ReadTimeout | Proxy is slow or overloaded | Lower the timeout and let the rotator bench it |
| SSLError | Proxy intercepts TLS, or the wrong proxy scheme is used | Use http:// for an HTTP proxy URL; avoid proxies that tamper with certificates |
| Frequent 429 | You're sending requests too fast | Increase delays and respect Retry-After |
| Persistent 403 | The site is refusing your traffic | Review its terms; look for an API or ask permission |

## Wrapping Up

Proxy rotation in Python comes down to three ideas: describe each proxy with a URL, pick the next one with a sensible strategy (random, round robin, or a provider gateway), and handle failures so a single bad proxy never sinks your run. The `ProxyRotator` and `fetch` function above give you a solid foundation: round-robin rotation, automatic failover, benching with a cooldown, and respect for rate limits.

From here, you could extend the rotator with weighted selection (favoring faster proxies), per-proxy statistics, or a lock for multithreaded use.

Happy scraping, and please do it politely!

* * *

_Found this helpful? Share it with a fellow developer or leave your questions in the comments._

---

*Published via [ZyVOP](https://zyvop.com/how-to-rotate-proxies-in-python-a-practical-guide-y9ccf?utm_source=hashnode&utm_medium=crosspost&utm_campaign=syndication) — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.*
