# Rotating User Agents and Headers in Python: A Practical Guide

Every request your scraper sends carries a small introduction: _who am I, and what can I handle?_ Left alone, a Python script introduces itself as `python-requests/2.34.2`, and some sites reject that on sight, even when they would happily serve a polite scraper.

The usual advice is to "rotate your user agent", and the usual code picks a random string from a list. That is a start, but on its own it is often _worse_ than doing nothing. A Chrome user agent that arrives without the other headers Chrome sends is a client that doesn't exist in the real world. This guide shows a better approach: rotate **complete, consistent header profiles**, **per session** rather than per request, and **check what you actually send**.

> **Version note:** tested with Python 3.13, `requests` 2.34, `httpx` 0.28, Scrapy 2.20, `curl_cffi` 0.17 and `fake-useragent` 2.2. The browser versions in the code (Chrome 152, Firefox 155) are a few releases behind the current ones (Chrome 157, with 158 rolling out, and Firefox 160 in mid-November 2026). Browsers update every few weeks, so treat the versions as placeholders to refresh.

* * *

## What the Server Sees

You can't improve what you can't see, so start by building a tiny server that echoes back the headers it receives. Save this as `echo_server.py`:

```python
import json
from http.server import BaseHTTPRequestHandler, HTTPServer

class Echo(BaseHTTPRequestHandler):
    def do_GET(self):
        body = json.dumps(dict(self.headers.items()), indent=2).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):    # keep the terminal quiet
        pass

HTTPServer(("127.0.0.1", 8000), Echo).serve_forever()
```

Run it in one terminal with `python echo_server.py`, install the libraries in another (`pip install requests httpx`), and look at what a plain `requests` call sends:

```python
import requests

print(requests.get("http://127.0.0.1:8000", timeout=10).text)
```

```json
{
  "Host": "127.0.0.1:8000",
  "User-Agent": "python-requests/2.34.2",
  "Accept-Encoding": "gzip, deflate, br, zstd",
  "Accept": "*/*",
  "Connection": "keep-alive"
}
```

(Your `Accept-Encoding` depends on which compression libraries are installed. That detail matters later.) Compare this with what Chrome sends for an ordinary page visit: a dozen headers, including language preferences, a detailed `Accept` list and several `sec-` headers. A server that looks at all of them sees an obvious difference.

## What a Browser Sends

You don't need to memorize every header, but you should know the ones that matter:

| Header | What it says | Notes |
| --- | --- | --- |
| User-Agent | Browser, major version and OS family | Chrome freezes the minor version (152.0.0.0) and Safari freezes the macOS version (10_15_7), so little else varies |
| Accept | Content types the client can handle | Browsers send a long, browser-specific list; libraries send */* |
| Accept-Language | Preferred languages | Libraries often send none at all |
| Accept-Encoding | Compression the client can decode | Must match what your library can actually decompress |
| sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform | Client Hints repeating the browser, device type and OS | Sent by Chrome and other Chromium browsers, not by Firefox or Safari |
| Sec-Fetch-Site, -Mode, -Dest, -User | How the request was triggered | none for a typed URL, same-origin for a click within a site |
| Referer | The page you came from | Absent on a first visit |

The important idea is **consistency**. The user agent, the Client Hints, the platform and the browser version must all tell the same story.

## Level 1: Rotate the User-Agent

The simplest version picks a user agent at random for each request:

```python
import random

import requests

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:155.0) Gecko/20100101 Firefox/155.0",
]

def get(url):
    headers = {"User-Agent": random.choice(USER_AGENTS)}
    return requests.get(url, headers=headers, timeout=10)

for _ in range(3):
    print(get("http://127.0.0.1:8000").json()["User-Agent"])
```

This works as far as it goes, and a lenient site may need nothing more. But look at what else arrives: `Accept: */*`, no `Accept-Language`, no `sec-ch-ua`. Every one of those requests claims to be a modern browser while sending the headers of a Python library.

## Level 2: Rotate Whole Header Profiles

A **profile** is a complete header set copied from a real browser. To capture one, open your browser's developer tools, go to the **Network** tab, load a page, right-click the first request and choose **Copy as cURL**. Keep headers like the ones in the profile below, and drop `Cookie`, `Host` and `Accept-Encoding`, which your library should manage itself.

Create `profiles.py`:

```python
import re

CHROME_WINDOWS = {
    "sec-ch-ua": '"Chromium";v="152", "Not?A_Brand";v="24", "Google Chrome";v="152"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-User": "?1",
    "Sec-Fetch-Dest": "document",
    "Accept-Language": "en-US,en;q=0.9",
}

CHROME_MACOS = {
    **CHROME_WINDOWS,
    "sec-ch-ua-platform": '"macOS"',
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
}

PROFILES = [CHROME_WINDOWS, CHROME_MACOS]
```

These two are the headers that the Chrome 152 profile of `curl_cffi` (more on that library below) sent to my echo server, so use them as a starting point and replace them with a capture from your own browser when you can. Add Firefox or Safari profiles the same way. They have no `sec-ch-ua` headers, which is exactly why you copy them whole instead of building them by hand.

Hand-built or mixed-up profiles are easy to get wrong, so add a quick linter to the same file. It catches the mismatches that give a fake identity away:

```python
def check_profile(headers):
    """Return a list of ways this header set contradicts itself."""
    h = {name.lower(): value for name, value in headers.items()}
    ua = h.get("user-agent", "")
    problems = []

    if "Chrome/" not in ua:
        if "sec-ch-ua" in h:
            problems.append("sec-ch-ua headers without a Chrome user agent")
        return problems

    if "sec-ch-ua" not in h:
        return ["Chrome user agent but no sec-ch-ua headers"]

    major = re.search(r"Chrome/(\d+)", ua).group(1)
    if f'v="{major}"' not in h["sec-ch-ua"]:
        problems.append(f"user agent says Chrome {major} but sec-ch-ua disagrees")

    platform = h.get("sec-ch-ua-platform", "").strip('"')
    expected = "Windows" if "Windows" in ua else "macOS" if "Macintosh" in ua else "Linux"
    if platform != expected:
        problems.append(f"user agent says {expected} but sec-ch-ua-platform says {platform or 'nothing'}")
    return problems

for profile in PROFILES:
    assert check_profile(profile) == []
```

## Level 3: Rotate Sessions, Not Requests

A real visitor doesn't change browsers between page views. If you rotate on _every_ request, one cookie-carrying session appears to switch from Windows Chrome to a Mac a second later. Rotate per **session** instead: choose a profile, keep it for a few dozen requests, then start over with a fresh identity and fresh cookies. Save this as `browser_session.py` next to `profiles.py`:

```python
import random
from urllib.parse import urlsplit

import requests

from profiles import PROFILES

class BrowserSession:
    """A requests session that behaves like one consistent browser."""

    def __init__(self, profiles=PROFILES, rotate_after=(20, 40)):
        self.profiles = profiles
        self.rotate_after = rotate_after
        self._new_identity()

    def _new_identity(self):
        self.profile = random.choice(self.profiles)
        self.session = requests.Session()               # a new session means new cookies
        self.session.headers.update(self.profile)       # replaces User-Agent and Accept
        self.limit = random.randint(*self.rotate_after)
        self.count = 0
        self.last_url = None

    def get(self, url, **kwargs):
        if self.count >= self.limit:
            self._new_identity()

        headers = {}
        if self.last_url:                               # behave like a click from the last page
            same_site = urlsplit(url).netloc == urlsplit(self.last_url).netloc
            headers["Referer"] = self.last_url
            headers["Sec-Fetch-Site"] = "same-origin" if same_site else "cross-site"

        kwargs.setdefault("timeout", 10)
        response = self.session.get(url, headers=headers, **kwargs)
        self.count += 1
        self.last_url = url
        return response

if __name__ == "__main__":
    browser = BrowserSession()
    for page in range(1, 4):
        reply = browser.get(f"http://127.0.0.1:8000/page-{page}").json()
        print(reply["User-Agent"][:60], "| Referer:", reply.get("Referer"))
```

Three details are worth noticing:

-   `session.headers.update(profile)` overwrites the library's `User-Agent` and `Accept` and adds everything else. It leaves `Accept-Encoding` to `requests` for a reason covered below.
    
-   **The first request has no** `Referer` and `Sec-Fetch-Site: none`, like a typed address. Later requests look like clicks.
    
-   **A new** `Session` **is a new identity.** Cookies don't leak from one "visitor" to the next.
    

If you also use proxies (see our guide to rotating proxies in Python), give each identity its own proxy and keep the pair together. Five different browsers behind one IP address looks just as odd as one browser jumping between five IP addresses mid-session.

## The Same Idea in httpx

The profile dictionaries work unchanged with `httpx`:

```python
import random

import httpx

from profiles import PROFILES

profile = random.choice(PROFILES)

with httpx.Client(headers=profile, timeout=10) as client:
    first = client.get("http://127.0.0.1:8000/page-1")
    print(first.json()["User-Agent"])
```

`httpx` keeps its own defaults (`Connection`, `Accept-Encoding`) and replaces `User-Agent` and `Accept` with yours. Like `requests`, it sends headers in its own order rather than a browser's.

## Rotating Profiles in Scrapy

In Scrapy, a downloader middleware is the right place. Put `profiles.py` inside your project package (here it's called `myproject`), then create the middleware in `myproject/middlewares.py`:

```python
import random

from myproject.profiles import PROFILES

class RotateProfileMiddleware:
    """Give every request a coherent browser header set."""

    def process_request(self, request):
        profile = request.meta.setdefault("profile", random.choice(PROFILES))
        for name, value in profile.items():
            request.headers[name] = value
```

Enable it in `settings.py`:

```python
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotateProfileMiddleware": 543,
}
```

The number matters: 543 runs _after_ Scrapy's built-in header middlewares (at 400 and 500), so your values replace their defaults. This also overrides the `USER_AGENT` setting. Because the chosen profile is stored in `request.meta["profile"]`, a retried or redirected request keeps the same identity. New requests created with `response.follow()` don't inherit `meta`, so to pin a whole chain of requests to one identity, pass it along explicitly:

```python
yield response.follow(url, callback=self.parse_page, meta={"profile": response.meta["profile"]})
```

Scrapy normalizes header names (`sec-ch-ua` goes out as `Sec-Ch-Ua`), one more small difference from a real browser that headers alone can't fix.

## What About fake-useragent?

The [fake-useragent](https://pypi.org/project/fake-useragent/) package gives you a ready-made list of user agents, and it's handy for quick experiments:

```python
from fake_useragent import UserAgent

ua = UserAgent(browsers=["Chrome", "Firefox"], os=["Windows", "Mac OS X", "Linux"], platforms=["desktop"])
print(ua.random)
```

Treat it with care. When I tested version 2.2.0:

-   **The default is mostly mobile.** Mobile devices make up about 69% of its weighted data; in 3,000 draws from plain `UserAgent().random`, 2,070 were mobile, including in-app browsers such as the Google and Facebook apps. Always pass `platforms=["desktop"]` unless you want mobile pages.
    
-   **The data is old.** Its newest Chrome is 136 and its newest Firefox 138, both from spring 2025, while real browsers were at 157 and 160. It also contains Chrome versions as old as 65.
    
-   **Failures are easy to miss.** Asking for a `min_version` it has no data for doesn't raise an exception. It logs a short error and returns a fallback Edge 122 user agent.
    

And a user agent alone is only one line of a profile. Drop one of its strings into a profile and the linter shows the problem:

```python
from fake_useragent import UserAgent

from profiles import CHROME_WINDOWS, check_profile

ua = UserAgent(browsers=["Chrome"], os=["Windows"], platforms=["desktop"])
mixed = {**CHROME_WINDOWS, "User-Agent": ua.random}
print(check_profile(mixed))
# ['user agent says Chrome 135 but sec-ch-ua disagrees']   (the version varies)
```

Your own captured profiles, refreshed every few months, beat a random list.

## The Accept-Encoding Trap

This is the one header you shouldn't copy from your browser. Chrome advertises `gzip, deflate, br, zstd`. If you paste that into a `requests` setup that doesn't have the `brotli` package installed, the server may answer in Brotli and `response.text` comes back as binary garbage. I reproduced exactly this. Leave `Accept-Encoding` to the library, as the code above does, or install `brotli` and `zstandard` first.

## Beyond Headers

Headers are only one signal. Sites can also look at the TLS handshake your client performs, HTTP/2 details, the order and capitalization of headers, your IP reputation and whether you run their JavaScript. Python's `requests` can't change most of that, and no header profile can hide it. Some libraries address the lower layers. `curl_cffi`, for example, impersonates a browser's connection and sends a matching, complete header set:

```python
from curl_cffi import requests as cffi_requests

response = cffi_requests.get("http://127.0.0.1:8000", impersonate="chrome")
print(response.json()["user-agent"])
```

If a site has gone to the trouble of blocking automated traffic, more disguise only starts an arms race.

## Using This Responsibly

Header rotation is a tool for being _compatible_, not for being _invisible_.

-   **Prefer honesty where you can.** A descriptive agent such as `MyBot/1.0 (+https://example.com/bot; me@example.com)` lets site owners contact you instead of blocking you. Use it unless a site rejects it for no good reason.
    
-   **Never rotate to dodge a rule.** `robots.txt` rules are addressed to crawlers by name, and a `*` group applies to everyone. Changing your identity to slip past a `Disallow`, a rate limit or a ban you were given is circumvention, and it can turn a terms-of-service problem into a bigger one. Our guide "Is Web Scraping Legal?" explains why.
    
-   **Don't raise your volume.** Rotation doesn't change how many requests you send. Keep your delays and respect `429` responses.
    
-   **Legitimate uses exist.** Making a permitted scraper work with a site that rejects library agents, or testing how your own site responds to different browsers.
    

## Troubleshooting

| Symptom | Likely cause | What to try |
| --- | --- | --- |
| 403 even with a full browser profile | The block uses your IP, TLS fingerprint or a JavaScript check, not your headers | Slow down and re-read the site's rules before trying anything more |
| response.text is binary garbage | You advertised br or zstd without the matching package | Drop Accept-Encoding or install brotli and zstandard |
| Parser breaks on some requests | You're getting mobile and desktop HTML at random | Use desktop-only profiles |
| CAPTCHAs appear after rotating | Identity changes mid-session, or one IP shows many browsers | Rotate per session and pair each identity with one proxy |
| check_profile reports a mismatch | Header set was mixed from different sources | Recapture the whole set from one browser |
| Scrapy ignores your USER_AGENT setting | The middleware overwrites it | That's intended; edit the profiles instead |

## Where to Go from Here

You now have a pattern you can trust: capture real header sets, lint them, rotate them per session, and check what actually leaves your machine with the echo server. From here:

-   Combine profiles with **rotating proxies** and a polite delay (see our guides on both).
    
-   If a site needs JavaScript, use a real browser instead of imitating one (see our Playwright guide).
    

Happy scraping, and please do it politely!

* * *

_Found this helpful? Share it with a fellow developer or leave your questions in the comments._

---

*Published via [ZyVOP](https://zyvop.com/rotating-user-agents-and-headers-in-python-a-practical-guide-7p6sc?utm_source=hashnode&utm_medium=crosspost&utm_campaign=syndication) — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.*
