Skip to main content

Rotating User Agents and Headers in Python: A Practical Guide

Learn how to rotate User-Agent strings and HTTP headers in Python using requests, httpx, and Scrapy with consistent browser profiles.

Updated 13 min readMarkdown
Rotating User Agents and Headers in Python: A Practical Guide

Every request your scraper sends carries a small introduction: who am I, and what can I handle? Left alone, a Python script introduces itself as python-requests/2.34.2, and some sites reject that on sight, even when they would happily serve a polite scraper.

The usual advice is to "rotate your user agent", and the usual code picks a random string from a list. That is a start, but on its own it is often worse than doing nothing. A Chrome user agent that arrives without the other headers Chrome sends is a client that doesn't exist in the real world. This guide shows a better approach: rotate complete, consistent header profiles, per session rather than per request, and check what you actually send.

Version note: tested with Python 3.13, requests 2.34, httpx 0.28, Scrapy 2.20, curl_cffi 0.17 and fake-useragent 2.2. The browser versions in the code (Chrome 152, Firefox 155) are a few releases behind the current ones (Chrome 157, with 158 rolling out, and Firefox 160 in mid-November 2026). Browsers update every few weeks, so treat the versions as placeholders to refresh.


What the Server Sees

You can't improve what you can't see, so start by building a tiny server that echoes back the headers it receives. Save this as echo_server.py:

import json
from http.server import BaseHTTPRequestHandler, HTTPServer

class Echo(BaseHTTPRequestHandler):
    def do_GET(self):
        body = json.dumps(dict(self.headers.items()), indent=2).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):    # keep the terminal quiet
        pass

HTTPServer(("127.0.0.1", 8000), Echo).serve_forever()

Run it in one terminal with python echo_server.py, install the libraries in another (pip install requests httpx), and look at what a plain requests call sends:

import requests

print(requests.get("http://127.0.0.1:8000", timeout=10).text)
{
  "Host": "127.0.0.1:8000",
  "User-Agent": "python-requests/2.34.2",
  "Accept-Encoding": "gzip, deflate, br, zstd",
  "Accept": "*/*",
  "Connection": "keep-alive"
}

(Your Accept-Encoding depends on which compression libraries are installed. That detail matters later.) Compare this with what Chrome sends for an ordinary page visit: a dozen headers, including language preferences, a detailed Accept list and several sec- headers. A server that looks at all of them sees an obvious difference.

What a Browser Sends

You don't need to memorize every header, but you should know the ones that matter:

Header What it says Notes
User-Agent Browser, major version and OS family Chrome freezes the minor version (152.0.0.0) and Safari freezes the macOS version (10_15_7), so little else varies
Accept Content types the client can handle Browsers send a long, browser-specific list; libraries send /
Accept-Language Preferred languages Libraries often send none at all
Accept-Encoding Compression the client can decode Must match what your library can actually decompress
sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform Client Hints repeating the browser, device type and OS Sent by Chrome and other Chromium browsers, not by Firefox or Safari
Sec-Fetch-Site, -Mode, -Dest, -User How the request was triggered none for a typed URL, same-origin for a click within a site
Referer The page you came from Absent on a first visit

The important idea is consistency. The user agent, the Client Hints, the platform and the browser version must all tell the same story.

Level 1: Rotate the User-Agent

The simplest version picks a user agent at random for each request:

import random

import requests

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:155.0) Gecko/20100101 Firefox/155.0",
]

def get(url):
    headers = {"User-Agent": random.choice(USER_AGENTS)}
    return requests.get(url, headers=headers, timeout=10)

for _ in range(3):
    print(get("http://127.0.0.1:8000").json()["User-Agent"])

This works as far as it goes, and a lenient site may need nothing more. But look at what else arrives: Accept: */*, no Accept-Language, no sec-ch-ua. Every one of those requests claims to be a modern browser while sending the headers of a Python library.

Level 2: Rotate Whole Header Profiles

A profile is a complete header set copied from a real browser. To capture one, open your browser's developer tools, go to the Network tab, load a page, right-click the first request and choose Copy as cURL. Keep headers like the ones in the profile below, and drop Cookie, Host and Accept-Encoding, which your library should manage itself.

Create profiles.py:

import re

CHROME_WINDOWS = {
    "sec-ch-ua": '"Chromium";v="152", "Not?A_Brand";v="24", "Google Chrome";v="152"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-User": "?1",
    "Sec-Fetch-Dest": "document",
    "Accept-Language": "en-US,en;q=0.9",
}

CHROME_MACOS = {
    **CHROME_WINDOWS,
    "sec-ch-ua-platform": '"macOS"',
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
}

PROFILES = [CHROME_WINDOWS, CHROME_MACOS]

These two are the headers that the Chrome 152 profile of curl_cffi (more on that library below) sent to my echo server, so use them as a starting point and replace them with a capture from your own browser when you can. Add Firefox or Safari profiles the same way. They have no sec-ch-ua headers, which is exactly why you copy them whole instead of building them by hand.

Hand-built or mixed-up profiles are easy to get wrong, so add a quick linter to the same file. It catches the mismatches that give a fake identity away:

def check_profile(headers):
    """Return a list of ways this header set contradicts itself."""
    h = {name.lower(): value for name, value in headers.items()}
    ua = h.get("user-agent", "")
    problems = []

    if "Chrome/" not in ua:
        if "sec-ch-ua" in h:
            problems.append("sec-ch-ua headers without a Chrome user agent")
        return problems

    if "sec-ch-ua" not in h:
        return ["Chrome user agent but no sec-ch-ua headers"]

    major = re.search(r"Chrome/(\d+)", ua).group(1)
    if f'v="{major}"' not in h["sec-ch-ua"]:
        problems.append(f"user agent says Chrome {major} but sec-ch-ua disagrees")

    platform = h.get("sec-ch-ua-platform", "").strip('"')
    expected = "Windows" if "Windows" in ua else "macOS" if "Macintosh" in ua else "Linux"
    if platform != expected:
        problems.append(f"user agent says {expected} but sec-ch-ua-platform says {platform or 'nothing'}")
    return problems

for profile in PROFILES:
    assert check_profile(profile) == []

Level 3: Rotate Sessions, Not Requests

A real visitor doesn't change browsers between page views. If you rotate on every request, one cookie-carrying session appears to switch from Windows Chrome to a Mac a second later. Rotate per session instead: choose a profile, keep it for a few dozen requests, then start over with a fresh identity and fresh cookies. Save this as browser_session.py next to profiles.py:

import random
from urllib.parse import urlsplit

import requests

from profiles import PROFILES

class BrowserSession:
    """A requests session that behaves like one consistent browser."""

    def __init__(self, profiles=PROFILES, rotate_after=(20, 40)):
        self.profiles = profiles
        self.rotate_after = rotate_after
        self._new_identity()

    def _new_identity(self):
        self.profile = random.choice(self.profiles)
        self.session = requests.Session()               # a new session means new cookies
        self.session.headers.update(self.profile)       # replaces User-Agent and Accept
        self.limit = random.randint(*self.rotate_after)
        self.count = 0
        self.last_url = None

    def get(self, url, **kwargs):
        if self.count >= self.limit:
            self._new_identity()

        headers = {}
        if self.last_url:                               # behave like a click from the last page
            same_site = urlsplit(url).netloc == urlsplit(self.last_url).netloc
            headers["Referer"] = self.last_url
            headers["Sec-Fetch-Site"] = "same-origin" if same_site else "cross-site"

        kwargs.setdefault("timeout", 10)
        response = self.session.get(url, headers=headers, **kwargs)
        self.count += 1
        self.last_url = url
        return response

if __name__ == "__main__":
    browser = BrowserSession()
    for page in range(1, 4):
        reply = browser.get(f"http://127.0.0.1:8000/page-{page}").json()
        print(reply["User-Agent"][:60], "| Referer:", reply.get("Referer"))

Three details are worth noticing:

  • session.headers.update(profile) overwrites the library's User-Agent and Accept and adds everything else. It leaves Accept-Encoding to requests for a reason covered below.

  • The first request has no Referer and Sec-Fetch-Site: none, like a typed address. Later requests look like clicks.

  • A new Session is a new identity. Cookies don't leak from one "visitor" to the next.

If you also use proxies (see our guide to rotating proxies in Python), give each identity its own proxy and keep the pair together. Five different browsers behind one IP address looks just as odd as one browser jumping between five IP addresses mid-session.

The Same Idea in httpx

The profile dictionaries work unchanged with httpx:

import random

import httpx

from profiles import PROFILES

profile = random.choice(PROFILES)

with httpx.Client(headers=profile, timeout=10) as client:
    first = client.get("http://127.0.0.1:8000/page-1")
    print(first.json()["User-Agent"])

httpx keeps its own defaults (Connection, Accept-Encoding) and replaces User-Agent and Accept with yours. Like requests, it sends headers in its own order rather than a browser's.

Rotating Profiles in Scrapy

In Scrapy, a downloader middleware is the right place. Put profiles.py inside your project package (here it's called myproject), then create the middleware in myproject/middlewares.py:

import random

from myproject.profiles import PROFILES

class RotateProfileMiddleware:
    """Give every request a coherent browser header set."""

    def process_request(self, request):
        profile = request.meta.setdefault("profile", random.choice(PROFILES))
        for name, value in profile.items():
            request.headers[name] = value

Enable it in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotateProfileMiddleware": 543,
}

The number matters: 543 runs after Scrapy's built-in header middlewares (at 400 and 500), so your values replace their defaults. This also overrides the USER_AGENT setting. Because the chosen profile is stored in request.meta["profile"], a retried or redirected request keeps the same identity. New requests created with response.follow() don't inherit meta, so to pin a whole chain of requests to one identity, pass it along explicitly:

yield response.follow(url, callback=self.parse_page, meta={"profile": response.meta["profile"]})

Scrapy normalizes header names (sec-ch-ua goes out as Sec-Ch-Ua), one more small difference from a real browser that headers alone can't fix.

What About fake-useragent?

The fake-useragent package gives you a ready-made list of user agents, and it's handy for quick experiments:

from fake_useragent import UserAgent

ua = UserAgent(browsers=["Chrome", "Firefox"], os=["Windows", "Mac OS X", "Linux"], platforms=["desktop"])
print(ua.random)

Treat it with care. When I tested version 2.2.0:

  • The default is mostly mobile. Mobile devices make up about 69% of its weighted data; in 3,000 draws from plain UserAgent().random, 2,070 were mobile, including in-app browsers such as the Google and Facebook apps. Always pass platforms=["desktop"] unless you want mobile pages.

  • The data is old. Its newest Chrome is 136 and its newest Firefox 138, both from spring 2025, while real browsers were at 157 and 160. It also contains Chrome versions as old as 65.

  • Failures are easy to miss. Asking for a min_version it has no data for doesn't raise an exception. It logs a short error and returns a fallback Edge 122 user agent.

And a user agent alone is only one line of a profile. Drop one of its strings into a profile and the linter shows the problem:

from fake_useragent import UserAgent

from profiles import CHROME_WINDOWS, check_profile

ua = UserAgent(browsers=["Chrome"], os=["Windows"], platforms=["desktop"])
mixed = {**CHROME_WINDOWS, "User-Agent": ua.random}
print(check_profile(mixed))
# ['user agent says Chrome 135 but sec-ch-ua disagrees']   (the version varies)

Your own captured profiles, refreshed every few months, beat a random list.

The Accept-Encoding Trap

This is the one header you shouldn't copy from your browser. Chrome advertises gzip, deflate, br, zstd. If you paste that into a requests setup that doesn't have the brotli package installed, the server may answer in Brotli and response.text comes back as binary garbage. I reproduced exactly this. Leave Accept-Encoding to the library, as the code above does, or install brotli and zstandard first.

Beyond Headers

Headers are only one signal. Sites can also look at the TLS handshake your client performs, HTTP/2 details, the order and capitalization of headers, your IP reputation and whether you run their JavaScript. Python's requests can't change most of that, and no header profile can hide it. Some libraries address the lower layers. curl_cffi, for example, impersonates a browser's connection and sends a matching, complete header set:

from curl_cffi import requests as cffi_requests

response = cffi_requests.get("http://127.0.0.1:8000", impersonate="chrome")
print(response.json()["user-agent"])

If a site has gone to the trouble of blocking automated traffic, more disguise only starts an arms race.

Using This Responsibly

Header rotation is a tool for being compatible, not for being invisible.

  • Prefer honesty where you can. A descriptive agent such as MyBot/1.0 (+https://example.com/bot; [email protected]) lets site owners contact you instead of blocking you. Use it unless a site rejects it for no good reason.

  • Never rotate to dodge a rule. robots.txt rules are addressed to crawlers by name, and a * group applies to everyone. Changing your identity to slip past a Disallow, a rate limit or a ban you were given is circumvention, and it can turn a terms-of-service problem into a bigger one. Our guide "Is Web Scraping Legal?" explains why.

  • Don't raise your volume. Rotation doesn't change how many requests you send. Keep your delays and respect 429 responses.

  • Legitimate uses exist. Making a permitted scraper work with a site that rejects library agents, or testing how your own site responds to different browsers.

Troubleshooting

Symptom Likely cause What to try
403 even with a full browser profile The block uses your IP, TLS fingerprint or a JavaScript check, not your headers Slow down and re-read the site's rules before trying anything more
response.text is binary garbage You advertised br or zstd without the matching package Drop Accept-Encoding or install brotli and zstandard
Parser breaks on some requests You're getting mobile and desktop HTML at random Use desktop-only profiles
CAPTCHAs appear after rotating Identity changes mid-session, or one IP shows many browsers Rotate per session and pair each identity with one proxy
check_profile reports a mismatch Header set was mixed from different sources Recapture the whole set from one browser
Scrapy ignores your USER_AGENT setting The middleware overwrites it That's intended; edit the profiles instead

Where to Go from Here

You now have a pattern you can trust: capture real header sets, lint them, rotate them per session, and check what actually leaves your machine with the echo server. From here:

  • Combine profiles with rotating proxies and a polite delay (see our guides on both).

  • If a site needs JavaScript, use a real browser instead of imitating one (see our Playwright guide).

Happy scraping, and please do it politely!


Found this helpful? Share it with a fellow developer or leave your questions in the comments.


Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

About the author

  • One platform to write, cross-post to Dev.to, Medium, WordPress, Hashnode, and Bluesky, and protect your SEO with canonical links. Zero paywalls and full content ownership.

No comments yet. Be the first to comment.