docker logs -f is fine for one container. Then you end up with a Compose stack of eight services, or three hosts, and a bug that only shows up when the API, the worker, and the database are all unhappy at the same moment. Now you're flipping between terminal tabs, trying to line up timestamps by eye.

Dozzle solves that one problem. It's a small web app that shows container logs live in your browser: open the page, click a container, watch the lines arrive. This post goes from the two-minute install through search, alerts, multiple hosts, Kubernetes, and locking it down. It's based on v11.

What it is, and what it isn't

Dozzle is a live tail, nothing more. It doesn't store logs. It reads from the Docker API, the same place docker logs reads from, so what you see is whatever Docker still holds, and how much that is depends on your logging driver's rotation settings. Once Docker drops a line, Dozzle can't show it.

Keep in mind: Dozzle is a live viewer, not a log store. If you need history, it has to come from Docker's log settings or a separate logging stack.

The upside of being that simple is that the image is only a few megabytes compressed and there's next to nothing to configure before logs appear. It works with Docker, Swarm, and Kubernetes, and with Colima and Podman too. Podman needs its remote socket enabled first.

The limits are worth knowing up front. The project says it's been tested with hundreds of containers, but it has no offline searching, and it points people who need full search toward tools like Loggly, Papertrail, or Kibana. Dozzle is for watching what's happening right now, not for digging through last week.

What changed in v10 and v11

A few things worth knowing if you last used Dozzle a while ago:

• v10 introduced alerts with webhook delivery. Today they cover logs, resource metrics, and container events.

• v11 is the biggest visual overhaul so far: flat, neutral panels, with color saved for things that need your attention. It also brought GitHub and OIDC sign-in, recognition of more log formats, and alerts that persist across reloads.

• v11.1 added a separate oidc auth provider that reads users and roles from the token, a login-first setup wizard for fresh installs, and generate-certs for giving agents their own certificate.

One upgrade catch: session tokens are now signed with a random secret kept in the data directory, so everyone gets signed out once after upgrading.

Quick start

The one-liner:

docker run -d --name dozzle \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v dozzle_data:/data \
  -p 8080:8080 \
  amir20/dozzle:latest

Open http://localhost:8080 and your containers should be listed. For something you plan to keep running, a Compose file is easier to maintain:

services:
  dozzle:
    image: amir20/dozzle:latest   # pin a specific version tag in production
    container_name: dozzle
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./dozzle-data:/data
    environment:
      DOZZLE_NO_ANALYTICS: "true"

Some notes on that file:

• Mount /data. Alert and destination settings are stored there, so without a volume they vanish on restart. User settings and your users.yml live there too.

• Dozzle sends anonymous usage analytics by default. DOZZLE_NO_ANALYTICS turns that off.

• Pin the image tag. With Dozzle moving fast (v11 signed everyone out on upgrade), latest can bite you at a bad time.

Two habits worth having from day one: mount /data so your settings survive restarts, and pin the image tag instead of using latest.

Getting around the interface

The sidebar lists your containers and groups Compose services by stack name automatically. v11 rebuilt it around collapsible groups with counts, and each container's icon carries a status badge. Container names are fuzzy-searchable, so on a busy host you type a few letters and jump straight to the service.

Logs stream in the main pane. Dozzle detects JSON logs and pretty-prints them, and if your entries have a level field they're colored by severity. In v11, warn and error rows get a light tint so they stand out as you scroll, and a live indicator plus a floating scroll readout show where you are in the container's lifetime. If you only care about problems, one click hides the info and debug lines.

Split view is the feature that actually replaces terminal tabs. It puts several containers side by side, so when the API returns a 500 you can watch the database and cache logs at the same timestamp. In v11 the pinned columns are stored in the URL, which means a side-by-side view is just a link you can send to a teammate.

Each container also gets small CPU and memory charts. They're basic, but enough to tell whether a container is struggling.

Searching and querying logs

For quick filtering there's regex search over the logs. For anything more analytical there's a SQL engine.

The SQL engine runs DuckDB compiled to WebAssembly inside your browser, so your logs never leave your machine. Dozzle loads your JSON logs into a virtual logs table that you can query. You open it from the menu or with Ctrl/Cmd+Shift+F, and it only works on JSON-structured logs. The docs still label it beta.

It queries what's already loaded in the browser, not Docker's full history. That makes it good for ad-hoc debugging, but don't expect trend analysis from it. WebAssembly caps it at 4 GB of memory, and if you run out you refresh the page.

-- How noisy is each severity right now?
SELECT level, COUNT(*) AS n
FROM logs
GROUP BY level;

-- Slowest failing requests (field names depend on your JSON logs)
SELECT message.path, message.status, message.duration
FROM logs
WHERE message.status >= 500
ORDER BY message.duration DESC
LIMIT 20;

-- Errors per minute
SELECT date_trunc('minute', timestamp) AS minute, COUNT(*) AS error_count
FROM logs
WHERE level = 'error'
GROUP BY minute
ORDER BY minute DESC;

If you already emit structured logs, this can replace a lot of docker logs | jq | grep pipelines.

Grouping and naming containers

Dozzle groups by stack by default. To make your own groups, add the dev.dozzle.group label, and containers that share a group name end up together in the UI. There's also a dev.dozzle.name label if you want a friendlier display name.

services:
  api:
    image: myorg/api:1.4.2
    labels:
      dev.dozzle.group: shop
      dev.dozzle.name: shop-api

Under Swarm, if Dozzle sees the service-name label, it switches to a swarm view that joins all tasks of the same service.

Limiting what Dozzle can see

DOZZLE_FILTER restricts which containers Dozzle can see at all. Filters are passed straight to Docker, in the same style as docker ps --filter, so DOZZLE_FILTER=label=color shows only containers that carry that label. They can also be set per agent and per user, and they stack: a container has to match all of them to show up.

Be careful with filters that exclude stopped containers, like status=running. The container that just crashed is often the one you need to read, and a filter like that hides it completely.

Security

Mounting the Docker socket gives a container effectively root-level access to the host, and the :ro in the examples above doesn't change that. It only marks the socket file read-only on disk, so API calls still pass through and create, delete, and update operations stay possible. If you don't need actions, put a socket proxy such as tecnativa/docker-socket-proxy between Dozzle and the daemon to limit what it can do.

An unauthenticated Dozzle on a reachable network also shows every container's logs to anyone who finds it, and logs often contain tokens and personal data.

Rule of thumb: no authentication, no exposure beyond localhost.

Built-in auth

Start by generating a users file:

docker run -it --rm amir20/dozzle generate admin \
  --password 'change-me' \
  --email admin@example.com \
  --name "Admin" > users.yml

Put users.yml in your mounted /data directory and set DOZZLE_AUTH_PROVIDER: simple. Passwords are stored bcrypt-hashed. Each user can also have a filter, which restricts which containers they can see by label, and roles, which control what they can do: shell, actions, download, notifications, and cloud. A user with no roles listed gets all of them, so set roles explicitly for anyone who shouldn't have full access. The instance-wide flags for shell and actions still have to be on before those roles do anything.

GitHub and OIDC (v11)

v11 lets you sign in with GitHub or any OIDC provider, such as Authentik, Keycloak, Pocket ID, or Google. It sits on top of the simple provider, so users.yml stays the allowlist, no accounts are created automatically, and password login keeps working. If you'd rather manage users and roles in your identity provider, v11.1 added a separate oidc provider that reads them from the token.

environment:
  DOZZLE_AUTH_PROVIDER: simple
  DOZZLE_AUTH_GITHUB_CLIENT_ID: <your-client-id>
  DOZZLE_AUTH_GITHUB_CLIENT_SECRET: <your-client-secret>

Forward-proxy auth

In production, Dozzle can trust identity headers from a proxy like Authelia, Authentik, or Cloudflare Access. That's the better route if you want centralized multi-factor auth, but it comes with one hard rule: Dozzle believes the Remote-User header on every request. Publish only the proxy and keep Dozzle on an internal network (expose, not ports), because anyone who can reach Dozzle directly can set that header and log in as whoever they like. Also map roles from your proxy, for example DOZZLE_AUTH_HEADER_ROLES: Remote-Groups for Authelia groups, since without a mapping every authenticated user gets all roles.

Actions and shell are opt-in

Container start/stop/restart actions (DOZZLE_ENABLE_ACTIONS) and shell access (DOZZLE_ENABLE_SHELL) are off by default. If you turn either on, get authentication in place first. They give the web UI the same power as docker stop and docker exec.

Reverse proxy

Dozzle streams logs over Server-Sent Events and uses WebSockets for shell and attach. That gives a reverse proxy three jobs: don't buffer responses, forward the WebSocket upgrade headers, and don't compress text/event-stream. Buffering makes logs arrive in bursts or not at all. A minimal nginx location:

location / {
    proxy_pass http://127.0.0.1:8080;

    chunked_transfer_encoding off;
    proxy_buffering off;
    proxy_cache off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 3600s;
}

The long read timeout matters too, because logs stop after a few seconds when the proxy's timeouts are short. Behind Traefik, the default compress middleware breaks SSE, so exclude text/event-stream. In Caddy, flush_interval -1 turns off response buffering. And if you mount Dozzle under a sub-path with DOZZLE_BASE, make sure the proxy passes the full path through instead of stripping the prefix.

Proxy tip: if logs arrive in bursts or not at all, response buffering is the first thing to turn off.

Keep it updated

Dozzle's security page lists several advisories from 2026, including these high-severity ones:

• an unauthenticated SSRF through the webhook test endpoint on default deployments without auth

• cross-site WebSocket hijacking on the exec and attach endpoints, which got around authentication for setups with shell enabled (versions up to 10.5.1)

• a label-based access bypass in the agent that allowed unauthorized shell access

So: turn on auth, keep the container patched, and keep it off the open internet.

Monitoring multiple hosts with agents

To see several machines in one UI, run Dozzle in agent mode on each remote host and point a central instance (the hub) at them. Agents listen on port 7007, and the hub connects to them over TLS.

# On each remote host
services:
  dozzle-agent:
    image: amir20/dozzle:latest
    command: agent
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - "7007:7007"   # keep this on a private network

# On the central host
services:
  dozzle:
    image: amir20/dozzle:latest
    volumes:
      - ./data:/data
    ports:
      - "8080:8080"
    environment:
      DOZZLE_AUTH_PROVIDER: simple   # expects users.yml in ./data
      DOZZLE_REMOTE_AGENT: "10.0.1.10:7007|web-1|production,10.0.1.11:7007|web-2|production"

The connection string looks like endpoint|name|group. All three parts are optional, and groups show up as collapsible sections in the sidebar, each with a button that merges the group's logs into one view. If the hub only needs to show remote hosts, you can skip mounting the local socket there. If you run Swarm, you don't need agents at all, because Dozzle discovers the cluster on its own.

Treat the agent port as sensitive. The TLS certificate Dozzle ships with is identical in every copy of the image, so it encrypts the connection but doesn't prove who is on the other end. Anyone who can reach port 7007 can connect their own Dozzle to your agent, read every log on that host, and run commands inside its containers. The agent also ignores DOZZLE_ENABLE_SHELL and DOZZLE_ENABLE_ACTIONS, because those flags only control what the UI offers. Keep 7007 on a private network (on a shared Docker network you don't need to publish it at all), and if anything you don't control can reach it, generate your own certificate with generate-certs so agents only accept your hub.

Important: anyone who can reach port 7007 can read every log and run commands inside that host's containers. Keep it on a private network.

Alerts

Since v10, Dozzle can tell you when something breaks instead of waiting for you to notice. It watches logs, resource metrics, and lifecycle events, evaluates your rules on your own instance, and sends notifications to a webhook, Slack, Discord, or ntfy.

Each alert has a container expression, which decides which containers to watch, and a trigger expression. Triggers come in three types: log, metric, and event. Setup lives on the Notifications page: add a destination first, then create rules. Webhook destinations come with built-in Slack, Discord, and ntfy payloads, and you can write custom Go text/template payloads for anything else. There's a Test button, so you can confirm delivery before saving.

Some example rules, written in the expression style the docs use:

# 5xx responses from production APIs
Container: name contains "api" && labels["env"] == "production"
Log:       message.status >= 500

# Memory pressure on the database
Container: name == "postgres"
Metric:    memory > 85

# Any OOM kill, anywhere
Container: true
Event:     name == "oom"

Metric alerts evaluate a smoothed average over a sample window and have a cooldown between triggers, so a brief spike doesn't flood your channel. For die events, the docs' example excludes exit codes 0, 130, 143, and 137, since those show up on routine stops and update cycles.

Dozzle Cloud is optional. Your rules always live on your self-hosted instance, but if you link it, delivery features such as grouping repeated failures, summaries, muting, and mobile channels are configured there.

Alerts are deliberately simple. There are no escalation policies or on-call rotations, so treat them as a safety net for staging and homelabs, not as a production pager.

Kubernetes

For Kubernetes, run Dozzle with DOZZLE_MODE=k8s. The docs include a full RBAC manifest; at minimum it needs read access to pods, pod logs, and nodes. Logs work without the Kubernetes Metrics API (metrics-server), but CPU and memory stay empty without it. Give it a persistent volume for /data so your alert config survives restarts.

env:
  - name: DOZZLE_MODE
    value: "k8s"
  - name: DOZZLE_NAMESPACE
    value: "prod,staging"   # optional; defaults to all namespaces
  - name: DOZZLE_FILTER
    value: "env=prod"       # optional label filter

The docs still call Kubernetes support a newer feature that may have limitations compared to the Docker version, and the release notes bear that out. v11.1.1 alone includes Kubernetes hardening, alerts for CronJob pods, and fixes for duplicate ReplicaSets and finished Jobs. If you run Dozzle on Kubernetes, keep it up to date.

Letting AI assistants read your logs (MCP)

Dozzle can expose an MCP endpoint so coding assistants can inspect your containers. It's disabled by default. Enable it with DOZZLE_ENABLE_MCP=true and it's served at /api/mcp from the same container. Every tool is read-only: listing containers and hosts, fetching and searching logs, and pulling CPU and memory history.

One warning: with no auth provider configured, the endpoint is publicly accessible, so set up authentication first. Once auth is on, MCP clients have to present credentials too.

When your app logs to files instead of stdout

Dozzle only sees what Docker captures, which means stdout and stderr, exactly like docker logs. Files inside a container are invisible to it.

The best fix is to log to the console, or symlink the log file to /dev/stdout, as the official nginx image does. If you can't, the docs suggest a small sidecar that tails the file:

docker run -d --name app-log --network none \
  --label dev.dozzle.name=app-log \
  --log-opt max-size=10m --log-opt max-file=3 \
  -v /var/log/myapp:/logs:ro \
  alpine tail -n 1000 -F /logs/app.log

Use -F instead of -f so the tail reopens the path after log rotation. Mount the directory, not the single file, because a single-file bind mount stays attached to the old inode.

Troubleshooting

• Empty stream for a container that's clearly running: if it uses a remote logging driver such as splunk, fluentd, or awslogs, check whether cache-disabled is set to true (and look at daemon.json too). That setting blocks the local cache Dozzle reads from.

• Logs arrive in bursts, or stop after a few seconds, behind a proxy: response buffering is on, text/event-stream is being compressed, or the read timeout is too short. See the reverse proxy section.

• Shell disconnects immediately: the proxy isn't forwarding the WebSocket upgrade headers.

• Won't start after following an old tutorial: DOZZLE_USERNAME and DOZZLE_PASSWORD are no longer supported. Use users.yml instead.

• Alerts vanish after a restart: /data isn't mounted as a volume.

• Signed out on every restart: if /data isn't writable, Dozzle falls back to an in-memory session secret (and warns about it), so sessions drop whenever it restarts.

• Everyone logged out after upgrading to v11: expected, and it only happens once.

When to outgrow Dozzle

Dozzle answers "what is this container saying right now?" It can't answer which deploy introduced this spike, did the error rate stay high overnight, or what happened to this request across three services last week. Those need retention, correlation, and analysis over time, which a real-time viewer doesn't give you. When you reach that point, add a proper logging or observability stack, like Loki, an OpenTelemetry pipeline, or a hosted platform, and keep Dozzle for the quick look.

Checklist before you rely on it

1. Pin the image version and update on a schedule.

2. Mount /data as a persistent volume.

3. Turn on authentication (users.yml, OIDC/GitHub, or a forward proxy) before exposing it beyond localhost.

4. Leave actions and shell off unless you need them, and put a socket proxy in front of the Docker socket if you don't need actions.

5. For multiple hosts, use agents instead of exposing a Docker socket, keep port 7007 on a private network, and generate your own agent certificate if it's reachable from anywhere else.

6. Set log rotation (max-size, max-file) so there's enough history to look at.

7. Log to stdout, in JSON if you can, so you get level coloring, SQL queries, and structured alerts.

References

• Dozzle docs and What's New in v11

• Alerts, Agent Mode, and Kubernetes

• Reverse Proxy & Base Path, Simple authentication, and Filters

• SQL Engine, MCP Integration, and Log Files on Disk

• Container Groups and supported environment variables

• Authentication, Forward Proxy, and Docker's dual logging docs

• Security advisories and release notes

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

The request isn't doing anything unusual: fetch a product from PostgreSQL, serialize it, return JSON. The data changes infrequently, but the same query runs thousands of times.

Redis looks like an obvious fix.

async function getProduct(id: string) {
  const key = `product:${id}`;

  const cached = await redis.get(key);

  if (cached) {
    return JSON.parse(cached);
  }

  const product = await productRepository.findOneBy({ id });

  if (product) {
    await redis.set(key, JSON.stringify(product), {
      EX: 3600,
    });
  }

  return product;
}

On a cache hit, an expensive database read disappears from the request path.

For example:

Database read:  ~800ms
Redis hit:       ~30ms

Those numbers are illustrative, but the improvement can be substantial when the original operation is expensive.

So far, Redis is doing exactly what we wanted.

The interesting problems start when the underlying data changes.

The Database Is Correct. The Response Isn't.

Consider product 123.

PostgreSQL contains:

{
  "id": "123",
  "name": "Mechanical Keyboard",
  "price": 129
}

Redis still contains an older copy:

{
  "id": "123",
  "name": "Mechanical Keyboard",
  "price": 99
}

The application checks Redis first:

const cached = await redis.get("product:123");

if (cached) {
  return JSON.parse(cached);
}

Nothing fails.

The database is healthy. Redis is healthy. The request returns 200 OK.

It also returns the wrong price.

The obvious fix is invalidation.

await productRepository.save(product);
await redis.del(`product:${product.id}`);

For a single cache key, that's straightforward.

Most applications don't have a single cache key.

One Row Can Exist in Several Cached Responses

Product 123 might appear in:

product:123
products:list
products:category:keyboards
products:featured
search:mechanical-keyboard

Updating one database row can make every one of those entries stale.

The write path now needs to know about the read paths:

await productRepository.save(product);

await Promise.all([
  redis.del(`product:${product.id}`),
  redis.del("products:list"),
  redis.del(`products:category:${product.categoryId}`),
  redis.del("products:featured"),
]);

And that still doesn't necessarily handle search results, filtered lists, pagination, or other cached representations containing the product.

This is where caching starts affecting application design.

Without caching:

update product -> database

With several cached representations:

update product
      |
      +--> database
      +--> product cache
      +--> category cache
      +--> list cache
      +--> featured cache
      +--> search cache

Reads became cheaper, but correctness now depends on invalidating every relevant representation.

The more places the same data is cached, the harder that becomes.

Redis Failure Is Not the Same as a Cache Miss

A common cache-aside implementation assumes this:

const cached = await redis.get(key);

if (cached) {
  return JSON.parse(cached);
}

return database.findProduct(id);

But if Redis is unavailable, redis.get() doesn't necessarily return null.

It may throw.

It may also spend time reconnecting or waiting for a network timeout before failing.

So a real fallback needs to treat Redis as a dependency that can fail.

async function getProduct(id: string) {
  const key = `product:${id}`;

  try {
    const cached = await redis.get(key);

    if (cached) {
      return JSON.parse(cached);
    }
  } catch (error) {
    logger.warn({ error, key }, "Redis read failed");
  }

  const product = await productRepository.findOneBy({ id });

  if (!product) {
    return null;
  }

  try {
    await redis.set(key, JSON.stringify(product), {
      EX: 3600,
    });
  } catch (error) {
    logger.warn({ error, key }, "Redis write failed");
  }

  return product;
}

This allows the database path to continue when Redis fails.

It doesn't mean the system is safe.

Suppose Redis normally absorbs 90% of product reads.

10,000 requests/minute

Redis hits:        9,000
PostgreSQL reads:  1,000

Now Redis becomes unavailable.

10,000 requests/minute

Redis hits:            0
PostgreSQL reads: 10,000

The database didn't fail.

Its workload changed by an order of magnitude.

This is why testing a Redis outage with one request proves very little.

The interesting test is Redis unavailable under normal production traffic.

There is another detail worth handling: the Redis client itself needs sensible connection and command timeouts. A fallback doesn't help much if every request waits several seconds for Redis before reaching PostgreSQL.

TTLs Can Move Load Instead of Removing It

Suppose thousands of cache entries are populated during a batch import or deployment.

They all receive the same TTL:

const CACHE_TTL = 3600;

await redis.set(key, value, {
  EX: CACHE_TTL,
});

If many entries are created around 10:00, many become eligible for expiration around 11:00.

Traffic that was previously hitting Redis starts rebuilding those entries from the database.

Instead of database work being spread across the hour, some of it becomes concentrated around expiration.

A small amount of TTL jitter helps avoid unnecessary synchronization:

function ttlWithJitter(baseSeconds: number) {
  const jitter = Math.floor(Math.random() * 300);

  return baseSeconds + jitter;
}

await redis.set(key, value, {
  EX: ttlWithJitter(3600),
});

Entries created together no longer have identical expiration times.

Jitter doesn't solve cache stampedes, though.

One Expired Key Can Trigger Many Queries

Consider a popular cache entry:

homepage:products

It expires.

Before any request has rebuilt it, 100 requests arrive.

Each one executes:

const cached = await redis.get(key);

if (!cached) {
  return loadFromDatabase();
}

All 100 requests observe the same miss.

The cache normally prevents repeated work, but during that window it provides no coordination between callers.

This is a cache stampede.

For a single Node.js process, duplicate work can be coalesced with an in-flight promise:

const pending = new Map<string, Promise<unknown>>();

async function loadOnce<T>(
  key: string,
  loader: () => Promise<T>,
): Promise<T> {
  const existing = pending.get(key);

  if (existing) {
    return existing as Promise<T>;
  }

  const request = loader().finally(() => {
    pending.delete(key);
  });

  pending.set(key, request);

  return request;
}

Now concurrent requests inside that process can share one rebuild.

But process-local coordination has an important limitation.

Suppose the application runs four instances:

             Load Balancer
          /       |       \
       API-1    API-2    API-3    API-4
         |        |        |        |
         +--------+--------+--------+
                          |
                     PostgreSQL

Each instance has its own pending map.

The same expired key can therefore trigger one rebuild per instance.

Four instances may produce four expensive queries instead of 100. That's much better, but it isn't globally coordinated.

For expensive or high-traffic keys, options include distributed locking, stale-while-revalidate, background refresh, or other forms of cross-instance request coordination.

Which one makes sense depends on how expensive stale data and duplicate work are for that particular cache.

Sometimes Redis Is Hiding the Real Problem

Suppose a database query takes 400ms.

Caching it might reduce most requests to a few milliseconds.

Before adding Redis, run:

EXPLAIN ANALYZE
SELECT ...

Maybe the query is scanning 800,000 rows because an index is missing.

After fixing the query:

Before index: ~400ms
After index:   ~30ms

Those numbers are an example, but the point matters: the cache would have hidden a query that should have been fixed.

The same applies to N+1 queries.

Or fetching entire relations when the response needs three fields.

Or repeatedly calling an external service when the data could have been included in the original request.

Caching a slow operation and optimizing a slow operation are different things.

Before deciding to cache an endpoint, it helps to know why the endpoint is expensive.

Not Everything Expensive Should Be Cached

A useful cache candidate usually has some combination of:

• expensive computation or I/O;

• frequent reads;

• relatively infrequent changes;

• tolerance for some amount of staleness;

• predictable invalidation.

A public category list might fit well.

categories:all

It is requested frequently and probably changes infrequently.

A user dashboard is different.

dashboard:user:4821

It might contain:

permissions
subscription state
usage limits
billing information
recent activity
account settings

Some of those values can change independently, and some may need stronger freshness guarantees than others.

Caching the entire dashboard as one object may save database work, but it also turns several independent data sources into one invalidation problem.

The question isn't only whether caching makes the endpoint faster.

The question is whether the performance gain is worth the new correctness problem.

Cache Keys Are Part of the Architecture

Cache keys often start simple:

product:123

Then requirements arrive.

Different currencies:

product:123:USD
product:123:EUR

Different locales:

product:123:en
product:123:fr

Different tenants:

tenant:42:product:123
tenant:91:product:123

Different permissions or response variants can add more dimensions.

At that point, cache-key design isn't an implementation detail.

A missing dimension can return stale data.

A missing tenant identifier can return someone else's data.

A key format that is difficult to invalidate can turn a simple update into a broad cache purge.

The key needs to represent every input that can materially change the cached response.

That is easy to say and surprisingly easy to get wrong.

What a Safer Cache-Aside Path Looks Like

A cache-aside read path is still simple conceptually:

The difficult parts are outside that diagram:

• deciding what deserves to be cached;

• choosing keys that represent the actual response;

• invalidating every affected representation;

• preventing synchronized expiration;

• controlling expensive rebuilds;

• handling Redis latency and failure;

• ensuring the database can survive reduced cache effectiveness.

Redis solves none of those automatically.

Redis Wasn't the Problem

Redis can remove an enormous amount of repeated work from a system.

It can also make a poorly understood performance problem harder to see.

If an endpoint is slow, start with the endpoint.

Measure the query.

Look at the query plan.

Check how much data is being fetched.

Check downstream calls.

Look for repeated computation.

Then decide whether the remaining work is something worth caching.

The better question isn't:

Should this use Redis?

It's:

What work are we avoiding, how long can we reuse its result, and what happens when that result is stale or unavailable?

If those answers are clear, Redis can be remarkably effective.

If they aren't, a 30ms cache hit may simply be hiding the next production problem.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

Here is a NestJS controller that most developers would look at and consider secure:

@Controller('invoices')
@UseGuards(JwtAuthGuard)
export class InvoicesController {
  constructor(private readonly invoicesService: InvoicesService) {}

  @Get(':id')
  getInvoice(@Param('id') id: string) {
    return this.invoicesService.findOne(id);
  }
}

async findOne(id: string): Promise<Invoice> {
  const invoice = await this.repo.findOneBy({ id });
  if (!invoice) throw new NotFoundException();
  return invoice;
}

The JWT guard works. An anonymous attacker gets a 401. An authenticated attacker with a free-tier account gets whatever they ask for:

GET /invoices/INV-10022  →  200 OK  { userId: 99, amount: 14990 }
GET /invoices/INV-10023  →  200 OK  { userId: 156, amount: 3200 }
# Every invoice in the database, iterated with a for loop

The vulnerability is not the identifier. It is the missing check that the identifier belongs to the user making the request.

This is IDOR (Insecure Direct Object Reference) — also called BOLA (Broken Object Level Authorization) in API security contexts. It falls under A01:2025 in the OWASP Top 10 and API1:2023 in the OWASP API Security list. It is consistently one of the most common causes of API data breaches in SaaS applications — not because it is sophisticated, but because it is a logic bug that automated tools do not catch.

Authentication and authorization answer different questions:

Nearly every NestJS tutorial covers authentication in depth. Object-level authorization is almost never discussed. That is the gap.

2. Five Places IDOR Hides in NestJS

URL parameters are the obvious case. These five are the ones that slip through even after a team has been briefed.

1. Request body IDs — trusting user-supplied ownership

// VULNERABLE: userId comes from the client
@Put('profile')
@UseGuards(JwtAuthGuard)
async updateProfile(@Body() dto: UpdateProfileDto) {
  return this.service.update(dto.userId, dto); // attacker sends userId: 999
}

2. Nested resource parents not validated

// VULNERABLE: checks the comment exists, not that postId belongs to the user
@Get('posts/:postId/comments/:commentId')
@UseGuards(JwtAuthGuard)
async getComment(@Param('postId') postId: string, @Param('commentId') commentId: string) {
  return this.commentsService.findOne(postId, commentId);
}

3. Bulk and batch operations with mixed IDs

// VULNERABLE: processes any IDs sent, including other users' records
@Delete('invoices/bulk')
@UseGuards(JwtAuthGuard)
async bulkDelete(@Body() dto: BulkDeleteDto) {
  return this.service.deleteMany(dto.ids); // no ownership filter on the array
}

4. Export and download endpoints

// VULNERABLE: export endpoints receive heavy feature testing, light security testing
@Get('reports/:reportId/export')
@UseGuards(JwtAuthGuard)
async exportReport(@Param('reportId') id: string, @Res() res: Response) {
  const pdf = await this.reportsService.generatePdf(id);
  res.send(pdf); // streams any report to any authenticated user
}

5. Pagination and list endpoints without user scope

// VULNERABLE: returns all records from the table, paginated
@Get('invoices')
@UseGuards(JwtAuthGuard)
async listInvoices(@Query() page: PaginationDto) {
  return this.repo.findAndCount({ take: page.limit, skip: page.offset });
  // ← no WHERE userId filter — attacker pages through everything
}

The list endpoint case is frequently missed because it looks like a collection route, not a resource lookup. Without a userId scope, a single authenticated user can page through the entire table.

3. Fix 1: Block Mass Assignment at the DTO Layer

TypeScript types disappear at runtime. A UpdateInvoiceDto that doesn't declare a userId field still accepts one from the client at the wire level unless you explicitly block it.

Enable whitelist validation globally:

// main.ts
app.useGlobalPipes(new ValidationPipe({
  whitelist:            true,   // strip fields not in the DTO
  forbidNonWhitelisted: true,   // reject requests that include extra fields
  transform:            true,
}));

Never include ownership fields in write DTOs:

// CORRECT — userId is absent. It comes from the JWT, never from the client.
export class CreateInvoiceDto {
  @IsString() @IsNotEmpty() description: string;
  @IsNumber() @IsPositive()  amount:      number;
  @IsEnum(Currency)          currency:    Currency;
}

export class UpdateInvoiceDto extends PartialType(CreateInvoiceDto) {
  // inherits the same safe fields — still no userId, orgId, or role
}

Always inject userId from the verified token:

@Post()
@UseGuards(JwtAuthGuard)
createInvoice(
  @Body() dto: CreateInvoiceDto,
  @CurrentUser() user: JwtPayload,   // from JWT signature — not client input
) {
  return this.service.create({ ...dto, userId: user.id });
}

The @CurrentUser() decorator reads from req.user, which is populated by JwtAuthGuard from the verified token — never from anything the client controls:

// decorators/current-user.decorator.ts
export const CurrentUser = createParamDecorator(
  (_: unknown, ctx: ExecutionContext): JwtPayload =>
    ctx.switchToHttp().getRequest().user,
);

Why forbidNonWhitelisted over whitelist alone? Silent stripping makes the API behavior confusing and harder to audit. Rejecting with a 400 makes the contract explicit and logs the attempted overpost.

4. Fix 2: Scope Every PostgreSQL Query With User Context

This is the most important fix. If your repository methods are built correctly, IDOR at the query level becomes structurally impossible: a wrong userId returns nothing.

// invoices.repository.ts
@Injectable()
export class InvoicesRepository {
  constructor(private readonly prisma: PrismaService) {}

  findOneByOwner(id: string, userId: string) {
    return this.prisma.invoice.findFirst({
      where: { id, userId },   // wrong userId → null, never another user's row
    });
  }

  findAllByOwner(userId: string, page: PaginationDto) {
    return this.prisma.invoice.findMany({
      where:   { userId },     // scoped — never returns other users' records
      take:    page.limit,
      skip:    page.offset,
      orderBy: { createdAt: 'desc' },
    });
  }

  async updateByOwner(id: string, userId: string, data: Partial<Invoice>) {
    const { count } = await this.prisma.invoice.updateMany({
      where: { id, userId },   // wrong userId → count 0, no mutation
      data,
    });
    return count > 0 ? this.findOneByOwner(id, userId) : null;
  }

  async deleteByOwner(id: string, userId: string): Promise<boolean> {
    const { count } = await this.prisma.invoice.deleteMany({
      where: { id, userId },   // wrong userId → count 0, no deletion
    });
    return count > 0;
  }
}

// invoices.service.ts
async findOne(id: string, userId: string): Promise<Invoice> {
  const invoice = await this.repo.findOneByOwner(id, userId);
  if (!invoice) throw new NotFoundException('Invoice not found');
  // ↑ 404 not 403 — do not reveal that the resource exists for a different user
  return invoice;
}

// invoices.controller.ts
@Controller('invoices')
@UseGuards(JwtAuthGuard)
export class InvoicesController {
  @Get()
  list(@CurrentUser() user: JwtPayload, @Query() page: PaginationDto) {
    return this.service.findAll(user.id, page);
  }

  @Get(':id')
  get(@Param('id') id: string, @CurrentUser() user: JwtPayload) {
    return this.service.findOne(id, user.id);
  }

  @Patch(':id')
  update(@Param('id') id: string, @Body() dto: UpdateInvoiceDto, @CurrentUser() user: JwtPayload) {
    return this.service.update(id, user.id, dto);
  }

  @Delete(':id') @HttpCode(204)
  delete(@Param('id') id: string, @CurrentUser() user: JwtPayload) {
    return this.service.delete(id, user.id);
  }
}

The rule is simple: userId flows down from the JWT payload. It is never accepted from a route parameter, query string, or request body.

5. Fix 3: The Ownership Guard Pattern

Repository-scoped queries protect against IDOR at the database layer. But service methods get called from background jobs, admin scripts, and other services that bypass controllers. A guard enforces ownership at the HTTP boundary independently — catching what the service layer might miss if called from a non-HTTP context.

// guards/ownership.guard.ts
import {
  CanActivate,
  ExecutionContext,
  Injectable,
  NotFoundException,
  SetMetadata,
} from '@nestjs/common';
import { ModuleRef, Reflector } from '@nestjs/core';

export interface OwnershipConfig {
  service:    string;   // Injectable token (e.g. InvoicesService.name)
  method:     string;   // Method that fetches the resource (unscoped)
  paramName:  string;   // Route param key holding the resource ID
  ownerField: string;   // Field on the resource that holds the owner's user ID
}

export const OWNERSHIP_KEY = 'ownership';
export const CheckOwnership = (config: OwnershipConfig) =>
  SetMetadata(OWNERSHIP_KEY, config);

@Injectable()
export class OwnershipGuard implements CanActivate {
  constructor(
    private readonly reflector: Reflector,
    private readonly moduleRef:  ModuleRef,
  ) {}

  async canActivate(context: ExecutionContext): Promise<boolean> {
    const config = this.reflector.getAllAndOverride<OwnershipConfig>(
      OWNERSHIP_KEY,
      [context.getHandler(), context.getClass()],
    );

    if (!config) return true;

    const request  = context.switchToHttp().getRequest();
    const user     = request.user as JwtPayload;
    const id       = request.params[config.paramName];

    const service  = this.moduleRef.get(config.service, { strict: false });
    const resource = await service[config.method](id);

    // No resource, or resource belongs to a different user → same response: 404
    // Never return 403 here — it confirms the resource exists, aiding enumeration
    if (!resource || String(resource[config.ownerField]) !== String(user.id)) {
      throw new NotFoundException();
    }

    request.resource = resource;  // pre-fetched — avoids a second DB call in the handler
    return true;
  }
}

Apply it per route:

@Controller('invoices')
@UseGuards(JwtAuthGuard)
export class InvoicesController {

  @Get(':id')
  @UseGuards(OwnershipGuard)
  @CheckOwnership({
    service:    InvoicesService.name,
    method:     'findOneUnsafe',   // unscoped fetch — guard handles ownership
    paramName:  'id',
    ownerField: 'userId',
  })
  getInvoice(@Request() req) {
    return req.resource;   // already fetched and validated — no second query
  }
}

The two layers do different jobs:

The guard catches IDOR at the HTTP boundary. The scoped query is the backstop for service methods called outside the HTTP lifecycle.

Complex authorization needs? For team memberships, org hierarchies, or status-conditional rules (e.g., "only update if not finalized"), CASL integrates cleanly with NestJS and Prisma. Build the scoped query layer first — CASL adds expressiveness on top, not a replacement for it.

6. Fix 4: PostgreSQL Row-Level Security as a Backstop

All of the above is application-layer defense. RLS enforces ownership at the database itself — even a completely unguarded endpoint returns an empty result set for rows that don't belong to the session's current user.

-- Enable RLS on the table
ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;

-- Policy: app_user_role can only see rows where user_id matches the session variable
CREATE POLICY invoices_owner_policy ON invoices
  AS PERMISSIVE FOR ALL
  TO app_user_role
  USING (user_id = current_setting('app.current_user_id', true)::uuid);
-- The 'true' flag means missing setting returns NULL (row hidden) rather than an error

Set the session variable per request, inside a transaction so SET LOCAL scopes correctly:

// rls.interceptor.ts — runs after JwtAuthGuard populates req.user
@Injectable()
export class RlsInterceptor implements NestInterceptor {
  constructor(@InjectDataSource() private ds: DataSource) {}

  async intercept(ctx: ExecutionContext, next: CallHandler) {
    const user = ctx.switchToHttp().getRequest().user as JwtPayload | undefined;
    if (user?.id) {
      await this.ds.query('SET LOCAL app.current_user_id = $1', [user.id]);
      // SET LOCAL is transaction-scoped; register this interceptor after
      // your transaction middleware if you use one
    }
    return next.handle();
  }
}

RLS is the deepest layer of defense — not a replacement for the application layer, but a guarantee that a missed guard cannot silently leak data.

7. UUIDs vs Sequential IDs

Use UUID primary keys. Sequential IDs (1, 2, 3) make enumeration trivial and expose record counts and creation velocity. UUIDs (v4) make blind guessing statistically impossible.

This is defense-in-depth, not a primary fix. An attacker who obtains a UUID through a shared link, referrer header, or log exposure can still exploit IDOR if ownership checks are absent. Obfuscating the identifier is not a substitute for verifying ownership.

// Prisma
model Invoice {
  id     String @id @default(uuid())
  userId String
}

// TypeORM
@Entity()
export class Invoice {
  @PrimaryGeneratedColumn('uuid') id: string;
  @Column()                       userId: string;
}

Use UUIDs. Add ownership checks. Both.

8. Testing IDOR: The Two-User Pattern

Catching IDOR requires at least two authenticated sessions: one to own the resource, one to attempt unauthorized access. Most automated scanners run a single session and miss this entirely.

// test/security/invoices.idor.spec.ts
describe('IDOR — Invoices', () => {
  let app:        INestApplication;
  let aliceToken: string;
  let bobToken:   string;
  let aliceInvoiceId: string;

  beforeAll(async () => {
    const module = await Test.createTestingModule({ imports: [AppModule] }).compile();
    app = module.createNestApplication();
    app.useGlobalPipes(new ValidationPipe({
      whitelist: true, forbidNonWhitelisted: true, transform: true,
    }));
    await app.init();

    // Create two independent users
    await request(app.getHttpServer()).post('/auth/register')
      .send({ email: 'alice@test.com', password: 'AlicePass1!' });
    await request(app.getHttpServer()).post('/auth/register')
      .send({ email: 'bob@test.com', password: 'BobPass1!' });

    aliceToken = (await request(app.getHttpServer()).post('/auth/login')
      .send({ email: 'alice@test.com', password: 'AlicePass1!' })).body.access_token;
    bobToken = (await request(app.getHttpServer()).post('/auth/login')
      .send({ email: 'bob@test.com', password: 'BobPass1!' })).body.access_token;

    // Alice creates a resource
    aliceInvoiceId = (await request(app.getHttpServer())
      .post('/invoices')
      .set('Authorization', `Bearer ${aliceToken}`)
      .send({ description: 'Alice Invoice', amount: 499, currency: 'USD' })).body.id;
  });

  afterAll(() => app.close());

  it('Alice can read her own invoice', () =>
    request(app.getHttpServer())
      .get(`/invoices/${aliceInvoiceId}`)
      .set('Authorization', `Bearer ${aliceToken}`)
      .expect(200));

  it('Bob CANNOT read Alice\'s invoice', () =>
    request(app.getHttpServer())
      .get(`/invoices/${aliceInvoiceId}`)
      .set('Authorization', `Bearer ${bobToken}`)
      .expect(404));   // Must be 404 — never 200 or 403

  it('Bob CANNOT update Alice\'s invoice', async () => {
    await request(app.getHttpServer())
      .patch(`/invoices/${aliceInvoiceId}`)
      .set('Authorization', `Bearer ${bobToken}`)
      .send({ description: 'Tampered' })
      .expect(404);

    // Verify the invoice was not modified
    const res = await request(app.getHttpServer())
      .get(`/invoices/${aliceInvoiceId}`)
      .set('Authorization', `Bearer ${aliceToken}`)
      .expect(200);
    expect(res.body.description).toBe('Alice Invoice');
  });

  it('Bob CANNOT delete Alice\'s invoice', async () => {
    await request(app.getHttpServer())
      .delete(`/invoices/${aliceInvoiceId}`)
      .set('Authorization', `Bearer ${bobToken}`)
      .expect(404);

    // Alice's invoice must still exist
    await request(app.getHttpServer())
      .get(`/invoices/${aliceInvoiceId}`)
      .set('Authorization', `Bearer ${aliceToken}`)
      .expect(200);
  });

  it('Bulk delete cannot include IDs from other users', async () => {
    await request(app.getHttpServer())
      .delete('/invoices/bulk')
      .set('Authorization', `Bearer ${bobToken}`)
      .send({ ids: [aliceInvoiceId] });

    // Alice's invoice must be untouched regardless of status code
    await request(app.getHttpServer())
      .get(`/invoices/${aliceInvoiceId}`)
      .set('Authorization', `Bearer ${aliceToken}`)
      .expect(200);
  });

  it('Bob\'s invoice list does not contain Alice\'s invoices', async () => {
    const res = await request(app.getHttpServer())
      .get('/invoices')
      .set('Authorization', `Bearer ${bobToken}`)
      .expect(200);

    const ids = res.body.data?.map((i: { id: string }) => i.id) ?? [];
    expect(ids).not.toContain(aliceInvoiceId);
  });

  it('Mass assignment attempt is rejected', () =>
    request(app.getHttpServer())
      .patch(`/invoices/${aliceInvoiceId}`)
      .set('Authorization', `Bearer ${bobToken}`)
      .send({ userId: 'new-owner-id' })
      .expect(400));   // forbidNonWhitelisted rejects unrecognised fields

  it('Unauthenticated request is rejected', () =>
    request(app.getHttpServer())
      .get(`/invoices/${aliceInvoiceId}`)
      .expect(401));
});

Add this test file to every resource module. The pattern is always: User A creates → User B attempts every verb → assert 404 → verify resource is unchanged.

9. Detecting Active Enumeration

Even with all fixes in place, monitor for active probing. The signature is distinctive: a single authenticated user generating a high 404 rate on resource endpoints.

// middleware/idor-detection.middleware.ts
@Injectable()
export class IdorDetectionMiddleware implements NestMiddleware {
  private readonly logger = new Logger('ThreatDetection');
  private readonly RESOURCE_PATH = /^\/(invoices|orders|reports|files)\//;

  constructor(@InjectRedis() private readonly redis: Redis) {}

  use(req: any, res: any, next: Function): void {
    next();
    res.on('finish', async () => {
      try {
        if (res.statusCode !== 404 || !req.user?.id || !this.RESOURCE_PATH.test(req.path)) return;

        const key = `idor:404:${req.user.id}`;

        // Pipeline makes incr + expire atomic — avoids the race condition
        // of a separate incr() call followed by a separate expire() call
        const [[, count]] = await this.redis.pipeline()
          .incr(key)
          .expire(key, 60)
          .exec() as [[null, number]];

        if (count >= 10) {
          this.logger.warn({
            event:    'POTENTIAL_IDOR_ENUMERATION',
            userId:   req.user.id,
            ip:       req.ip,
            path:     req.path,
            count404: count,
            window:   '60s',
          });
        }
      } catch { /* never let monitoring break request handling */ }
    });
  }
}

Wire this up globally and route the structured log to your observability platform. A single user generating 10+ 404s per minute on resource endpoints warrants investigation.

10. Checklist

Before shipping any endpoint that touches user-owned data:

DTO Layer
☐ ValidationPipe: whitelist: true, forbidNonWhitelisted: true, globally
☐ No write DTO (Create/Update) contains userId, orgId, or role fields
☐ Response DTOs use @Exclude() on internal fields (userId, deletedAt)

Controller Layer
☐ Every resource endpoint has @UseGuards(JwtAuthGuard)
☐ userId always comes from @CurrentUser() — never from @Body() or @Param()
☐ Resource-specific routes (GET/PATCH/DELETE /:id) have @UseGuards(OwnershipGuard)

Repository / Service Layer
☐ Every findOne / update / delete takes userId as a required parameter
☐ userId is in every WHERE clause — never optional
☐ List/pagination endpoints scoped by userId — never unfiltered
☐ Bulk operations filter input IDs against the requesting user's owned IDs
☐ Soft-delete queries include userId even when withDeleted: true

Testing
☐ Every resource module has an IDOR test: User A creates → User B attempts all verbs
☐ Bulk endpoint test: User B's request cannot affect User A's records
☐ List endpoint test: User B's response never contains User A's resource IDs
☐ IDOR tests run in CI on every PR

11. FAQ

Q: Should I return 403 or 404 when ownership fails? Return 404. A 403 tells the attacker the resource exists, confirming the ID is valid and encouraging further enumeration. A 404 reveals nothing — the resource either doesn't exist or doesn't belong to the requesting user. Apply this consistently: an inconsistent response pattern across endpoints becomes an oracle.

Q: Is IDOR only a reading problem? No — writes are often more damaging and harder to detect. Blind IDOR writes don't return the victim's data, so they look less obviously wrong, but they can silently overwrite records, cancel orders, or delete content. Test every HTTP verb (GET, PATCH, PUT, DELETE) in your two-user suite, not just GET.

Q: Does @Roles() or any role guard solve IDOR? No. Role guards answer "does this user have the admin role?" They don't answer "does this user own this specific row?" IDOR is a horizontal authorization problem — two users with the same role accessing each other's data. Role guards are vertical access control. You need both.

Q: How do I handle resources that belong to a team, not a single user? Extend both layers. At the repository layer: WHERE user_id = \(1 OR team_id = ANY(\)2). At the guard layer: check user.teamIds.includes(resource.teamId) alongside the userId check. CASL with @casl/prisma can generate these compound WHERE clauses automatically from your defined ability rules.

12. Further Reading

OWASP

• 📖 OWASP Top 10 A01:2025 — Broken Access Control

• 📖 OWASP API Security — API1:2023 BOLA

• 📖 OWASP Authorization Cheat Sheet

• 📖 CWE-639: Authorization Bypass Through User-Controlled Key

Deep Dives

• 📰 IDOR Vulnerabilities: The Complete Technical Guide (2026) — CodeAnt

• 📰 DTOs That Don't Spill: Secure NestJS Validation Patterns — Modexa

• 📰 Why DAST Tools Miss Real IDOR Vulnerabilities — Apiiro

Reference

• 📖 NestJS Guards

• 📖 NestJS Authorization

• 📖 CASL + NestJS

• 📖 PostgreSQL Row-Level Security

Run the two-user test against every existing endpoint before the next release — not just new ones. IDOR is most commonly found in code that was already live.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

Now drop it into SWE-bench Pro — the version built from private codebases the model has never seen. The best standardized score on the public leaderboard is 59.1%. On the truly private commercial set, where nobody gets to tune in advance, no model clears 47%.

That's not a rounding error. That's a 20-to-40-point cliff, and it shows up everywhere. Gemini 3.1 Pro falls from 80.6% to somewhere between 32% and 46%, depending on which cut you trust. Every lab's flagship shows the same shape of fall.

Here's a wrinkle nobody saw coming: the two highest-scoring models that exist right now — Claude Mythos 5 (95.5%) and Claude Fable 5 (95.0%) — aren't something you can actually use. Anthropic suspended public access to both under an export control directive. The "best" model on paper this week is one you can't buy.

That gap, between the benchmark you can quote and the codebase you actually own, is the entire "AI replaces developers" argument in miniature. Models are extraordinary at problems that resemble what they've already seen. They get shaky fast the moment the code is genuinely yours — your weird conventions, your one engineer's undocumented caching hack from three years ago.

Cost was never the obstacle. Reliability was.

But cost is the question people actually ask. So let's answer it — before getting into why the answer doesn't settle anything.

The subscription math, updated for June 2026

Pricing in this category moves fast enough that anything written before May is already stale. And the biggest shift just happened: GitHub Copilot blew up its entire pricing model on June 1, 2026, moving from flat-rate to usage-based credits.

The fallout from the Copilot switch has been loud. Developers who used to burn 3% of their monthly allowance on a normal day are now burning that much in under an hour. One person reported a single file review — no code written — eating a fifth of their monthly cap.

The flat $10–20 sticker price that defined this market for two years is functionally gone the moment you're doing real agentic work.

A solo developer paying for tokens directly, rather than a subscription, can still land around \(5–\)30 a month for light use. But push an agent into a hard debugging session and a single sitting can burn 500,000+ tokens and \(15 or more. Real heavy-user spend across every tool here now sits at \)100–$200/month — not the number on the landing page.

Scale that to a ten-person team running agents seriously, and you're at \(1,000–\)2,000+/month in tooling alone. That's before anyone counts the human time spent reviewing what the agent produced.

The other side of the ledger

Here's the part most "AI is basically free" posts skip: the loaded cost of a developer isn't the number on the offer letter.

Add 25–30% for payroll tax and benefits. Add 15–25% more for equity at the senior tier. A \(150,000 salary becomes \)220,000–$280,000 in year one once recruiting and onboarding get folded in.

Set against that, even Claude Code Max or Cursor Ultra at $2,400/year is rounding error. On subscription-cost-versus-salary-cost alone, AI tooling wins by two orders of magnitude. Every time. Nobody serious disputes that.

The actual argument is whether the AI is doing the job — or just doing the typing while a human still does the job around it.

The productivity number nobody wanted to publish — and the sequel nobody expected

In July 2025, METR ran the most rigorous test of this question anyone had attempted. Sixteen experienced open-source developers. 246 real tasks. Randomized into AI-allowed and AI-disallowed groups, working in codebases they'd maintained for years.

The expected result was a speed-up. What they found instead: developers using AI took 19% longer.

Stranger still — those same developers believed they'd been 20% faster. They were wrong about the direction of their own productivity.

METR re-ran it. Bigger cohort, less self-selected: 57 developers, 143 repos, 800+ tasks. Published February 2026. The topline slowdown shrank to roughly -4%. And among the original developers who did both rounds, the number flipped entirely — an 18% speedup, though the confidence interval was wide enough that METR called it weak evidence, not proof.

Then the story took a turn nobody saw coming.

By April 2026, METR scrapped the whole experimental design. The reason: AI use had become so universal that 30–50% of invited developers simply refused to participate without AI access. You can't run a control group when the population won't be controlled.

So METR pivoted. Their May 2026 replacement is a self-report survey of 349 technical workers. Headline number: a median 1.4–2x self-reported increase in the value of their work, with respondents forecasting 2.5x by 2027.

Here's the twist. The one subgroup best positioned to judge this rigorously — METR's own researchers — reported the lowest gains of anyone surveyed. The agency that proved developers overestimate their own speedup just published a 2026 headline number built entirely on self-report, and told readers to be skeptical of it themselves.

What never changed, across every version of this research: the most experienced developers, on the most mature codebases, got the least benefit — sometimes a negative one. One Chrome engineer put it simply: it wasn't about being unfamiliar with the tools. It was about working in a codebase he already knew cold, versus a small greenfield project where a less experienced developer would benefit far more.

Same shape as the SWE-bench Pro cliff. Just measured with a stopwatch instead of a leaderboard.

A cautionary tale from outside engineering — now with a sequel of its own

Software isn't the only function that ran this experiment.

In 2023, Klarna replaced roughly 700 customer service staff with an OpenAI-built assistant. For a while, the numbers looked unambiguous: the bot handled two-thirds of all queries. By 2025, the company was walking it back. CEO Sebastian Siemiatkowski's own words: cost became too dominant a factor, and quality dropped in a way that wasn't sustainable.

The 2026 version of that story isn't a clean "AI failed, humans win" ending. Klarna's current setup is a hybrid: AI still handles roughly two-thirds of inquiries, but instead of rehiring full-time staff for the rest, the company routes the remainder to gig-style contractors — brought on flexibly, without the overhead of full employment.

It's a third option nobody was describing in 2023 or 2025: AI for volume, on-demand humans for what AI can't close, and full-time headcount for neither end.

Customer service and software engineering aren't the same job. But the failure mode is identical: optimize a replace-the-headcount decision purely on subscription cost versus salary cost, and you forget to price in what happens when the automated version meets a case it wasn't built for. For Klarna, that was an angry customer with a nuanced refund dispute. In engineering, it's a production incident the agent introduced and didn't flag — three weeks before anyone notices.

Where the cost argument actually wins outright

None of this means the cost case is wrong everywhere. It's right in one specific place: greenfield work, low stakes per mistake, human still firmly in the loop.

Solo-founder software businesses clearing seven and eight figures with single-digit or zero employees are a real, growing category in 2026. A few hundred dollars a month in agent subscriptions against an \(80,000–\)120,000-a-month human team isn't an exaggeration for that segment. It's just the math.

The same crowd is also the first to flag the real risk — and it's not the AI's coding ability. It's concentration of failure. Replace fifty employees with five hundred agents, and you don't become the CEO of a lean company. You become the single point of failure for every lawsuit, every hallucination, every 3am incident. That's a cost too. It just doesn't show up on the subscription invoice.

What the job market is doing, not saying

Surveys are cheap to answer dishonestly. Payroll data isn't.

Stanford HAI's 2026 AI Index, built on ADP payroll records rather than self-reported surveys, confirms it: employment for software developers aged 22–25 has fallen nearly 20% since late 2022 — the same window AI coding tools went mainstream. Developers over 30 in the same exposed roles grew employment 6–12% over that period.

But 2026 made the picture messier, not cleaner.

Salesforce CEO Marc Benioff confirmed the company isn't hiring additional software developers this fiscal year, crediting AI agents directly. In the same breath, he announced a push to hire 1,000 new college graduates — redirected toward sales and customer-facing roles, where AI hasn't displaced headcount. Read narrowly, that's not "AI replaced the juniors." It's "AI replaced one kind of junior work, and hiring moved to wherever that work doesn't exist yet."

IBM has kept its public commitment to triple US entry-level hiring, betting junior developers shift toward judgment-heavy, customer-facing work now that AI absorbs the repetitive part. Meanwhile, some labor economists are pushing back on the AI narrative entirely — pointing at interest-rate-driven hiring freezes and a brutal grad recruiting cycle as the more plausible cause, with AI serving as a convenient, simultaneously-timed scapegoat.

A year in, the more careful read: companies aren't cleanly eliminating junior developers. They're raising what "junior" has to mean on day one. AI-fluent juniors now command a premium north of 40% over generalists doing the job the old way.

So, is it cheap enough?

Cheap enough to replace a developer's typing? Yes. Has been for a while. Not a close call. A \(20–\)200/month tool against a quarter-million-dollar loaded engineer wins on cost every single time — even after this year's pricing shake-up made the entry-level numbers less honest than they used to be.

Cheap enough to replace a developer's judgment — the part that knows which benchmark cliff is about to bite you, catches the bug the agent confidently introduced, decides what shouldn't be automated at all? Not yet. The clearest evidence anyone built to answer that question collapsed under the weight of how fast adoption moved. The agency that ran it is now relying on self-reported numbers it has publicly told you to doubt.

The gap between "passes the public benchmark" and "survives contact with your actual codebase" is still tens of points wide, no matter which model you pick. And the models posting the highest scores right now aren't even available to use.

The teams getting real value out of this in 2026 aren't asking "can AI replace developers." They're asking which slice of the week is mechanical enough to hand off, and which slice is the part the salary was actually paying for.

Get that split right, and the cost question mostly answers itself.

AI is cheap enough to replace a developer's typing, but not yet cheap enough to replace their judgment — and the evidence built to settle that question kept dissolving the moment researchers tried to pin it down.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

A cloud-routed AI agent doing real work can cost \(10 to \)20 a day in API credits. That's not a one-time fee. It's a meter that runs every day the agent stays useful.

The alternative gaining traction in 2026: stop renting the model. Own the box it runs on instead. A mini PC, on 24/7, hosting both the language model and the agent layer that drives it.

AMD's "Strix Halo" platform makes this practical. These chips pair a strong CPU with the largest integrated GPU AMD has shipped for a small form factor, fed by enough unified memory to hold genuinely large models.

Minisforum's MS-S1 MAX is one of several systems built on Strix Halo, alongside the Framework Desktop and boxes from Beelink and HP. Paired with Hermes Agent — an open-source autonomous agent maintained by Nous Research, now backed by NVIDIA's own DGX Spark integration — the combination answers a real question: can a business run its AI workflows without a recurring cloud bill?

Based on independent benchmarks and reviews of this hardware, the answer is mostly yes. There are real caveats worth knowing before you buy.

The hardware: what Strix Halo actually offers

The MS-S1 MAX runs on AMD's Ryzen AI Max+ 395: a 16-core/32-thread Zen 5 chip with a Radeon 8060S integrated GPU. That GPU has 40 RDNA 3.5 compute units — roughly the performance class of a discrete RX 7600 XT, built into the SoC instead of sitting on its own card.

Minisforum pairs it with a six-heatpipe, dual-fan cooler and a built-in 320W power supply. Continuous output sits around 110–130W, with peaks up to 160W depending on performance mode.

The detail that matters most for LLMs isn't GPU speed. It's memory architecture.

Strix Halo's GPU shares the system's unified LPDDR5X pool instead of using dedicated VRAM. AMD's driver can allocate a large chunk of that pool as GPU-addressable memory via GTT (Graphics Translation Table). On a 128GB unit, that typically means up to 96GB usable by the GPU.

A reviewer at AkitaOnRails bought the Minisforum specifically to run models too big for any consumer GPU. Their framing: an RTX 5090 is several times faster per token, but it caps out at 32GB. Models that don't fit simply don't run there. Strix Halo's pitch is capacity, not speed, at a fraction of the cost and power draw of a professional GPU.

It's worth knowing the field. The same chip ships in Framework's Desktop (modular, repairable) and Strix Halo boxes from Beelink and HP — all in a similar performance bracket. One step up, the realistic alternative is Apple's Mac Studio with M3 Ultra: up to 512GB of unified memory at roughly 3x the bandwidth. That's for models too large for any Strix Halo box, at a correspondingly higher price.

AMD itself joined that lineup directly this month, launching its own first-party Ryzen AI Halo developer box — same Ryzen AI Max+ 395 chip, same 128GB ceiling, but built and sold by AMD as an explicit answer to NVIDIA's DGX Spark, in Windows 11 Pro or Linux SKUs. At $3,999 it's a real premium over the Minisforum/Framework/Beelink crowd, and the premium buys AMD's own validated software stack rather than any hardware advantage — it's the same silicon discussed throughout this piece. On the NVIDIA side, the headline Computex 2026 announcement, RTX Spark, isn't a DGX Spark successor — it's a Windows-on-ARM platform aimed at laptops and creator desktops, shipping this fall. For the specific always-on, headless-Linux use case here, DGX Spark remains NVIDIA's current answer, and nothing has displaced Strix Halo as AMD's.

Setting up the model layer

The software runs on llama.cpp. The GPU is driven through one of two backend paths: Vulkan (Mesa RADV or AMD's AMDVLK) or ROCm/HIP.

Independent 2026 testing across Strix Halo builds generally finds Vulkan/RADV the most stable path. ROCm sometimes wins on prompt processing for long contexts, but it takes more tuning to get there — driver pinning, environment overrides — and it's less reliable out of the box.

Getting the GPU to claim its full memory allocation takes one BIOS change: set minimum dedicated VRAM as low as the board allows. Then add a few AMDGPU kernel parameters so the driver doesn't cap the GTT allocation.

Running models full-GPU (-ngl 999) keeps the CPU free for everything else sharing the box — the agent process, the VPN daemon, a dashboard.

For anyone running more than one model size, llama-swap helps. It's a small Go binary, maintained by mostlygeek, that sits in front of llama-server and hot-swaps the loaded model based on the incoming request. A lighter model can serve quick replies while a larger one loads on demand for harder tasks, all behind one stable API endpoint.

The performance reality check

Worth triangulating across sources here, not just trusting one build log — the numbers below are pulled from several independently-run Strix Halo benchmark logs (linked below), not just the two reviews cited elsewhere in this piece.

Multiple independent 2026 benchmark logs converge on GPT-OSS-120B (~59GB at Q4) generating at 53–56 tokens/second over Vulkan/RADV. That's consistent enough across reviewers to plan around, and close to what NVIDIA's own DGX Spark posts on comparable models.

That last row matters more than the headline number. Dense (non-MoE) 70B-class models run dramatically slower than MoE models of similar size on this hardware. Token generation here is bandwidth-bound, and dense models activate far more parameters per token.

The AkitaOnRails review, on the same chip, reported high single-digit speeds for a 70B dense model. It also flagged ROCm bugs that, at the time, blocked some dense 70B+ models from running at all on that backend. Usable for batch jobs. Not for live chat.

Use MoE models — GPT-OSS-120B, Qwen 3.6's 35B-A3B — as the local default. Treat large dense models as a slower, occasional-use tool, not the daily driver.

One model family conspicuously missing from that table: Qwen 3.6. Alibaba released it in April 2026 — a 27B dense model and a 35B-A3B MoE variant — and it's specifically what NVIDIA's own Hermes/DGX Spark writeup recommends pairing with this class of hardware, not GPT-OSS. Early Strix Halo numbers back that recommendation: the dense Qwen3.6-27B at Q4 runs around 12 tokens/second on its own, but llama.cpp's new multi-token-prediction (MTP) support — merged in May 2026 — nearly doubles that to roughly 21 t/s on the same hardware. The 35B-A3B MoE variant lands in GPT-OSS-120B's speed class while using a fraction of the memory. If you're setting this up today, start with Qwen 3.6 as the default and treat GPT-OSS as the fallback, not the other way around.

Thermally, the platform handles 24/7 use well. Sustained loads around 110W with edge temperatures in the high 60s°C are typical on better-cooled Strix Halo boxes. That's the actual precondition for "leave it running in a closet" being safe rather than a fire risk.

The workflow layer: Hermes Agent

A fast local model isn't enough on its own. Something still needs to turn tokens into actions.

Hermes Agent is an open-source autonomous agent maintained by Nous Research. It connects to any OpenAI-compatible model endpoint and handles what a raw model can't: persistent memory, scheduled jobs, tool calls, sub-agents, browser automation, and messaging integrations.

A request moving through the stack looks like this:

User
  ↓
Hermes Agent  (memory, tool calls, scheduling)
  ↓
OpenAI-compatible API  (llama-server / llama-swap)
  ↓
llama.cpp
  ↓
Local model

NVIDIA's RTX AI Garage team has since written about pairing Hermes with local hardware directly. That's a reasonable signal: "agent client talking to a local OpenAI-compatible server" is becoming a standard pattern, not a one-off hack — though notably, NVIDIA's own writeup pairs Hermes with Qwen 3.6, not GPT-OSS (see above).

Setup follows a consistent shape across documented builds:

1. Point Hermes at the local server. Choose a self-hosted/OpenAI-compatible provider during setup. Give it localhost:<port> where llama-server (or llama-swap) is listening. No API key needed.

2. Make local the default, not the only option. Pair it with a fallback to a hosted provider — OpenRouter or a frontier-model API — for tasks needing more reasoning, speed, or context. Quick lookups don't need a 120B model. Heavy tool chains sometimes do benefit from a hosted model's larger context window.

3. Run it as a service. Enable Hermes (and any VPN daemon) as a systemd service. It survives reboots without anyone babysitting a terminal.

4. Reach it remotely. Tailscale's free Personal plan now supports up to six users with unlimited devices per tailnet. It's the common choice for reaching a closet PC from a laptop or phone without exposing the dashboard to the open internet.

What about the NPU?

Strix Halo's 50-TOPS XDNA2 NPU mostly sits unused in these setups. That's still accurate as of mid-2026.

FastFlowLM, the project building NPU-native inference for Ryzen AI chips, reports around 19 tokens/second running GPT-OSS-20B entirely on the NPU, at roughly 10x better power efficiency than GPU inference. Genuinely useful on a battery-constrained laptop. Less compelling on a plugged-in desktop, where the iGPU is already faster and power isn't the constraint.

FastFlowLM shipped Windows-first, but added official Linux support in March 2026 via Debian packages and AMD's Lemonade SDK — it's a supported apt install on Ubuntu now, not a community workaround. Pairing the NPU as a fast draft model alongside a larger GPU-served model is still mostly a DIY exercise, though. For a 24/7 desktop agent box, the NPU is currently a "nice to have," not a missing piece holding the setup back.

The actual cost math

This isn't free. Power, hardware amortization, and time spent maintaining drivers all cost something.

What changes is the shape of the cost. A fixed, predictable infrastructure cost replaces a variable per-token bill, one that scales with exactly the workloads you're trying to encourage the agent to take on.

At roughly 110W sustained, running a Strix Halo box continuously costs a few dollars a month in power at typical residential rates. That's a rounding error next to even modest cloud-agent credit spend. The bigger variable is upfront hardware cost. Pricing across the Strix Halo lineup is genuinely volatile — smaller-memory boxes start around \(1,500, while a fully loaded 128GB unit like the MS-S1 Max has listed anywhere from roughly \)2,300 to $3,000+ depending on retailer and promo timing (it ships in a single 128GB configuration, not a range of RAM tiers) — weighed against how much cloud spend it actually displaces.

Privacy doesn't depend on which box or model size you pick. Whatever model is loaded, API keys, customer data, and business workflows never leave the local network. That's true on a cheap machine running a tiny model, and true on a fully loaded MS-S1 MAX running a 120B one. The hardware just decides how much work stays local before something has to go to the cloud anyway.

The verdict

Good fit: steady, repeatable automation where privacy or cost predictability matters more than raw speed. Research summaries, content drafts, routine tool-calling chains — anything that can tolerate the agent thinking for tens of seconds instead of two.

Weaker fit: anything customer-facing and latency-sensitive, or workloads leaning on dense 70B+ models where driver maturity still has rough edges.

Treat the local model as the default path, and a hosted fallback as the exception-handling lane, not the other way around. The economics, and the realistic performance, both work out in the local box's favor.

The bill, this time, doesn't arrive.

Further reading

• ServeTheHome — Minisforum MS-S1 Max Review

• AkitaOnRails — Review: Minisforum MS-S1 Max

• NVIDIA RTX AI Garage — Hermes Agent on RTX PCs and DGX Spark

• mostlygeek/llama-swap on GitHub

• strix-halo-guide — community Strix Halo benchmark log (GitHub)

• Caleb Coffie — Benchmarking llama.cpp's MTP support on Strix Halo

• Qwen3.6-35B-A3B release blog (Alibaba)

• FastFlowLM — Linux installation guide

• TechPowerUp — AMD Ryzen AI Halo dev kit pre-orders go live

• NVIDIA/Microsoft — Introducing RTX Spark

• Tailscale — Free pricing plans

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

This post builds a background job pipeline in NestJS using BullMQ: a request that enqueues work and returns immediately, a worker that processes it with automatic retries, and a durable record a client can poll for the outcome. It's demonstrated on an async draft-generation endpoint — the same shape as a real LLM-backed content pipeline — with a couple of details about BullMQ's actual runtime behavior that are easy to get wrong if you're going off the docs alone. The full project, tested end-to-end against real Postgres and Redis, is linked at the end.

The shape of the problem

A synchronous version of this endpoint looks simple:

@Post('drafts')
async create(@Body() dto: CreateDraftDto) {
  const content = await this.llmProvider.generate(dto.topic); // could take 5-30s, could fail
  return this.draftsRepository.save({ topic: dto.topic, content });
}

The request now blocks for as long as the LLM call takes, and a single transient failure (a timeout, a rate limit, a brief provider outage) becomes a failed request with nothing to show for it — no record that it was attempted, nothing to retry, nothing to inspect afterward.

The fix is to decouple "accept the request" from "do the work":

@Post('drafts')
create(@Body() dto: CreateDraftDto) {
  return this.draftsService.enqueue(dto); // returns in milliseconds
}

enqueue() writes a pending row and hands the job to BullMQ. A separate worker process picks it up, retries it automatically on failure, and updates that row when it's done — successfully or not.

Two sources of truth, on purpose

This implementation keeps two records of a job's state, deliberately:

• BullMQ's own state, in Redis — which attempt it's on, when it'll retry next, its position in the queue. This is operational state, and it's normal for it to get cleaned up after a job finishes (removeOnComplete/removeOnFail).

• A Postgres row, written by the application — pending → processing → completed/failed, plus the result or failure reason. This is what survives a Redis flush, what a client actually polls, and what you'd query for "show me every failed draft from last week."

export enum DraftJobStatus {
  PENDING = 'pending',
  PROCESSING = 'processing',
  COMPLETED = 'completed',
  FAILED = 'failed',
}

@Entity()
export class DraftJob {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  @Column()
  topic: string;

  @Column({ type: 'varchar', default: DraftJobStatus.PENDING })
  status: DraftJobStatus;

  @Column({ type: 'text', nullable: true })
  result: string | null;

  @Column({ type: 'text', nullable: true })
  failureReason: string | null;

  @Column({ default: 0 })
  attemptsMade: number;

  @CreateDateColumn()
  createdAt: Date;

  @Column({ type: 'timestamp', nullable: true })
  completedAt: Date | null;
}

Conflating these two — treating BullMQ's Redis-backed job as the only record — means losing history the moment a job gets cleaned up, and gives clients nothing stable to poll against.

Wiring BullMQ into the Nest app

Before any of that, BullMQ needs a Redis connection — configured once, at the root module:

// app.module.ts
BullModule.forRootAsync({
  inject: [ConfigService],
  useFactory: (configService: ConfigService) => ({
    connection: {
      host: configService.get<string>('REDIS_HOST'),
      port: configService.get<number>('REDIS_PORT'),
    },
  }),
}),

Then, inside whichever feature module actually uses a queue, that queue gets declared by name:

// drafts.module.ts
BullModule.registerQueue({ name: 'draft-generation' }),

That string, 'draft-generation', is the thread tying three separate places together: it's what registerQueue declares here, what @InjectQueue('draft-generation') asks for in the service below, and what @Processor('draft-generation') listens on in the worker. All three have to use the exact same name — there's no compiler check enforcing that, since it's just a string. The two ways this can go wrong behave very differently, and it's worth knowing which is which rather than assuming:

• If @InjectQueue doesn't match anything registerQueue declared, Nest's dependency injection fails loudly at startup with an UnknownDependenciesException — I confirmed this directly, and the error message names the exact missing provider token and suggests the fix. The app won't boot at all, so this mistake gets caught immediately.

• If @Processor doesn't match — while @InjectQueue/registerQueue still agree with each other — the app boots without any error, jobs enqueue successfully, and then just sit in the queue forever. I verified this too: enqueued a job against a correctly-wired producer, waited, and checked the queue directly — the job's state was still waiting, indefinitely, with nothing logged anywhere to indicate why. No worker was ever listening on that name. This is the genuinely silent failure mode worth watching for, since nothing about it looks broken until someone notices a backlog that never drains.

Enqueueing a job

The job's payload type is defined right alongside the service that creates it — enqueue() and the worker (below) both depend on this shape:

export interface DraftJobData {
  draftJobId: string;
  topic: string;
  simulateFailures: number;
}

async enqueue(dto: CreateDraftDto) {
  const draftJob = await this.draftJobsRepository.save(
    this.draftJobsRepository.create({ topic: dto.topic, status: DraftJobStatus.PENDING }),
  );

  await this.draftQueue.add(
    'generate',
    { draftJobId: draftJob.id, topic: dto.topic, simulateFailures: dto.simulateFailures ?? 0 },
    {
      jobId: draftJob.id,
      attempts: 3,
      backoff: { type: 'exponential', delay: 2000 },
      removeOnComplete: { age: 3600 },
      removeOnFail: { age: 86400 },
    },
  );

  return { id: draftJob.id, status: draftJob.status };
}

A few specific choices worth calling out:

jobId: draftJob.id — using the Postgres row's own id as the BullMQ job id, rather than letting BullMQ generate one, is what makes re-enqueueing idempotent (more on this below).

backoff: { type: 'exponential', delay: 2000 } — each retry waits roughly double the previous gap, rather than hammering a struggling upstream API at a fixed interval. With attempts: 3 there are only ever two such gaps to observe, not three — I measured them directly against this exact config rather than trusting the formula from memory: ~2035ms after the first failure, ~4009ms after the second, and no fourth attempt ever fires once those three tries are exhausted. The delay keeps doubling if you raise attempts higher, but at attempts: 3 specifically, a third gap is never reachable — that ceiling comes from attempts, not from backoff alone.

removeOnComplete/removeOnFail — without these, BullMQ keeps every job's data in Redis indefinitely. These ages keep Redis from growing unbounded while still leaving failed jobs around longer (a day, vs. an hour for successes) since they're more likely to need debugging.

The worker

@Processor('draft-generation', { concurrency: 5 })
export class DraftGenerationProcessor extends WorkerHost {
  constructor(private readonly draftsService: DraftsService) {
    super();
  }

  async process(job: Job<DraftJobData>): Promise<string> {
    const { draftJobId, topic, simulateFailures } = job.data;
    await this.draftsService.markProcessing(draftJobId);

    // Throwing here is what tells BullMQ to retry, subject to `attempts`/`backoff`.
    return generateDraftContent(topic, job.attemptsMade, simulateFailures);
  }

  @OnWorkerEvent('completed')
  async onCompleted(job: Job<DraftJobData>, result: string) {
    await this.draftsService.markCompleted(job.data.draftJobId, result, job.attemptsMade);
  }

  @OnWorkerEvent('failed')
  async onFailed(job: Job<DraftJobData> | undefined, error: Error) {
    if (!job) return;
    const maxAttempts = job.opts.attempts ?? 1;
    if (job.attemptsMade >= maxAttempts) {
      await this.draftsService.markFailed(job.data.draftJobId, job.attemptsMade, error.message);
    }
    // else: still has retries left, BullMQ will reschedule it automatically
  }
}

@Processor(...) plus extending WorkerHost is the @nestjs/bullmq pattern for defining a worker — process() is the actual job handler, and @OnWorkerEvent hooks into BullMQ's lifecycle events. (The actual source file also logs each attempt via Nest's Logger, trimmed from the snippet above for readability — the logic shown is otherwise unchanged from what's in the repo.)

The detail that's easy to get backwards: attemptsMade

BullMQ's 'failed' event fires after every failed attempt, not just the last one. If you write the failure-handling logic without checking attempt count, a job that's about to succeed on its third try gets incorrectly marked failed after its first.

The fix is the job.attemptsMade >= maxAttempts check above — but getting that comparison right depends on knowing exactly what attemptsMade contains at each point, which isn't obvious from the type signature alone. I checked this directly against a running BullMQ instance rather than assuming:

attemptLog (job.attemptsMade as seen INSIDE the processor on each run):
[ { attemptsMade: 0, opts: 3 },
  { attemptsMade: 1, opts: 3 },
  { attemptsMade: 2, opts: 3 } ]
final 'completed' event attemptsMade: 3

attemptsMade is 0 on the very first execution, not 1 — it counts completed prior attempts, not the current attempt number. Inside process(), a job configured with attempts: 3 sees 0, 1, 2 across its three tries; the 'completed'/'failed' event handlers see it afterward, already incremented to 3. Getting this backwards (checking attemptsMade <= maxAttempts instead of >=, or assuming attempt 1 reads as 1 inside the processor) produces a retry condition that's off by exactly one — either giving up one attempt early, or never giving up at all.

What "idempotent" actually means here

Using the Postgres row's id as the BullMQ jobId means re-enqueueing the same id is a no-op — but it's worth being precise about what that actually does, rather than taking it on faith. I tested this directly:

await queue.add('task', { n: 1 }, { jobId: 'fixed-id' }); // creates the job
await queue.add('task', { n: 2 }, { jobId: 'fixed-id' }); // returns a Job object, but...

const stored = await queue.getJob('fixed-id');
console.log(stored.data); // { n: 1 } — the SECOND add() never took effect

The second add() call doesn't throw, and it doesn't error — it just silently does nothing. The job already in Redis under that id keeps its original data. This holds whether the original job is still waiting, actively processing, or has already completed (as long as it hasn't been cleaned up by removeOnComplete).

That's exactly the property you want for a reconciliation or retry path elsewhere in your own backend: if something upstream of enqueue() ever calls it twice for the same logical request — a retried HTTP call, a duplicate webhook delivery, a race in a distributed system — the second call doesn't create a second job, doesn't reprocess, and doesn't overwrite the first job's data with whatever the second call happened to pass. It's a much cheaper idempotency guarantee than building your own deduplication table, but it only works because the jobId is something stable and meaningful (the Postgres row's id) rather than an auto-generated one.

Setting up and running the project

Clone the repo, install dependencies, and copy the environment template:

git clone <your-repo-url>
cd bullmq-nestjs-demo
npm install
cp .env.example .env

This project needs both Postgres and Redis running locally — Postgres for the durable job records, Redis for BullMQ's own queue state. The fastest path for both is Docker:

docker run --name jobs-postgres \
  -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=jobs_demo \
  -p 5432:5432 -d postgres:16

docker run --name jobs-redis -p 6379:6379 -d redis:7

The defaults in .env.example already line up with those two containers, so you shouldn't need to edit .env if you used them as-is. Then start the API:

npm run start:dev

synchronize: true is enabled in app.module.ts for this demo, so the draft_job table is created automatically on first boot — no manual migration step needed to follow along. (Turn that off and switch to real migrations before deploying anywhere real.) The worker runs inside this same process for simplicity here; see the production notes below on why you'd typically split it out.

With the server up on http://localhost:3000, you're ready to run through the flow below.

Testing it end-to-end

This is the actual sequence run against a live server, real Postgres, and real Redis before publishing — happy path, retry-then-succeed, permanent failure, and the validation/not-found edge cases:

# Happy path — no simulated failures
curl -X POST http://localhost:3000/drafts \
  -H "Content-Type: application/json" \
  -d '{"topic":"NestJS background jobs"}'
# -> { "id": "...", "status": "pending" }

curl http://localhost:3000/drafts/<id>
# -> { "status": "completed", "attemptsMade": 1, "result": "..." }

# Forces 2 failures before success — watch attemptsMade end at 3,
# with ~2s then ~4s of exponential backoff between attempts
curl -X POST http://localhost:3000/drafts \
  -H "Content-Type: application/json" \
  -d '{"topic":"Retry demo","simulateFailures":2}'

curl http://localhost:3000/drafts/<id>
# (poll a few times over ~6-8s)
# -> { "status": "completed", "attemptsMade": 3, "result": "..." }

# More failures than the configured 3 attempts allow — exercises the
# permanent-failure path instead
curl -X POST http://localhost:3000/drafts \
  -H "Content-Type: application/json" \
  -d '{"topic":"Permanent failure demo","simulateFailures":5}'

curl http://localhost:3000/drafts/<id>
# -> { "status": "failed", "attemptsMade": 3, "failureReason": "Simulated upstream failure..." }

# Validation and not-found paths, checked the same way
curl -X POST http://localhost:3000/drafts \
  -H "Content-Type: application/json" \
  -d '{"topic":"a"}'
# -> 400 Bad Request — topic is shorter than the DTO's @MinLength(3)

curl http://localhost:3000/drafts/00000000-0000-0000-0000-000000000000
# -> 404 { "message": "Draft job not found" }

Every path above — immediate success, eventual success after retries, permanent failure after exhausting retries, a rejected validation error, and an unknown id — was checked against the actual status, attemptsMade, and HTTP status code returned by a live server, not just "the request didn't error." The duplicate-jobId idempotency behavior from the previous section was verified the same way, against this same running instance, using a small script that called queue.add() directly rather than going through the HTTP API (since triggering a raw duplicate enqueue isn't something a normal client request can do on its own).

Production notes

A few things were deliberately simplified for this demo and are worth tightening before shipping:

• Run the worker as a separate process from the API. Here, the processor lives inside the same NestJS app as the controller — fine for a demo, but in production a traffic spike on the HTTP side will compete for CPU with job processing unless they're split into independent, independently-scalable deployments.

• Distinguish retryable from non-retryable errors. A timeout or a 429 should retry. A 401 from a bad API key will fail identically on every attempt — throw new UnrecoverableError(...) (exported by BullMQ) skips the remaining retries instead of wasting them on a guaranteed failure.

• Size concurrency to what the upstream and your database can actually sustain. concurrency: 5 here is a demo default, not a number derived from real capacity.

• Monitor queue depth, not just individual job outcomes. A queue that's silently backing up faster than it's draining is a different failure mode than any single job failing, and won't show up by looking at one job's status at a time.

• There's no authentication on these endpoints. This demo is scoped to the queueing mechanics, not access control — anyone who can reach POST /drafts can enqueue work, and anyone who knows (or guesses) a job id can read its result. If this sits behind a real API, put it behind the same kind of auth boundary as anything else you wouldn't want publicly writable (a prior post on this blog covers building TOTP-based 2FA in NestJS, if that's useful context for the kind of guard logic involved).

Source code

The complete, tested implementation — NestJS module, the worker, entity, DTOs, and a README with the full setup and curl walkthrough — is available as a standalone repository: bullmq-nestjs-demo. Clone it, point it at your own Postgres and Redis, and the enqueue → retry → status-poll flow above works out of the box.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

Below is both halves of a real defense: the IP throttle for volume, and a small Redis-backed service that locks by email for everything the throttle misses. Every response shape, status code, and timing claim here came from actually running the requests, not from the docs — including one interaction between two timers that will quietly undo a lockout's own "fresh start" if you're not paying attention to how they relate.

The throttle, and what it actually returns

A baseline policy on every route, plus a stricter override on login specifically:

// app.module.ts — global baseline: 20 requests/60s per IP, everywhere
ThrottlerModule.forRoot([{ name: 'default', ttl: 60000, limit: 20 }]),
providers: [{ provide: APP_GUARD, useClass: ThrottlerGuard }],

// auth.controller.ts — login gets a stricter limit than the rest of the API
@Throttle({ default: { limit: 10, ttl: 60000 } })
@Post('login')
async login(@Body() dto: LoginDto) { /* ... */ }

Whether that override replaces the global 20/60s or stacks on top of it isn't obvious from reading the decorator, so it's worth confirming rather than guessing: it replaces. A route-level @Throttle with the same policy name (default) fully takes over for that route. Eleven rapid requests to /auth/register (no override, rides the global policy) all came back 201; the same eleven against /auth/login would have tripped its 10-request limit well before the last one.

The response headers on a request that's still under the limit:

X-RateLimit-Limit: 10
X-RateLimit-Remaining: 9
X-RateLimit-Reset: 60

And once it's crossed:

// HTTP 429
{ "statusCode": 429, "message": "ThrottlerException: Too Many Requests" }

That's the library's real, unmodified default — not a placeholder for something cleaner. It reads like a stringified exception because that's more or less what it is, and it's worth routing through @nestjs/throttler's exception factory if this needs to match the rest of an API's error shape.

Where the throttle's job ends

None of that knows or cares who's logging in. Ten requests a minute from one IP against alice@example.com and ten requests a minute from ten different IPs, one attempt each, all against alice@example.com, look completely different to a per-IP counter — the second pattern never trips it, no matter how long it continues. That's the gap a second, IP-independent mechanism has to cover.

Locking the account, not the address

async recordFailure(email: string): Promise<FailureResult> {
  const key = this.attemptsKey(email);
  const attempts = await this.redis.incr(key);
  if (attempts === 1) {
    await this.redis.expire(key, this.windowSeconds);
  }

  if (attempts >= this.maxAttempts) {
    await this.redis.set(this.lockKey(email), '1', 'EX', this.lockoutSeconds);
    return { attempts, locked: true, retryAfterSeconds: this.lockoutSeconds };
  }

  return { attempts, locked: false };
}

async recordSuccess(email: string): Promise<void> {
  await this.redis.del(this.attemptsKey(email), this.lockKey(email));
}

INCR against a key that doesn't exist yet starts it at 1, so there's no separate setup step for a first failure. The counter's expiry is set once — only when attempts === 1 — so a streak of failures within one window doesn't keep pushing its own deadline out; it's one streak, with one expiry, not a self-renewing one.

Both key-building helpers lowercase the email first:

private attemptsKey(email: string): string {
  return `login-attempts:${email.toLowerCase()}`;
}

Skipping that normalization would open a real gap, not just a style nitpick — so rather than assume it works, this was tested directly: three failures as dana@example.com, then two more as Dana@Example.COM, a different casing entirely. The Redis counter read 5 afterward, under a single key, and a sixth attempt in yet another casing (DANA@EXAMPLE.COM) — this time with the correct password — still came back locked. Without the .toLowerCase(), those three casings would have been three separate five-strike budgets instead of one.

Two timers that look independent and aren't

The service has two separate durations: how long the failure counter itself lives (windowSeconds), and how long an actual lock lasts once triggered (lockoutSeconds). Treating them as unrelated knobs is the natural first instinct, and it's wrong — tested directly, with a 10-second window and a 6-second lock:

lock status right after the 6s lock expires: { locked: false }
attempts count at that same moment:          5

The lock is gone, but the counter — on its own, longer, 10-second clock — hasn't caught up yet. It's still sitting at 5. One more failed attempt right then doesn't start over at 1; it pushes the existing counter to 6, still over the 5-attempt threshold, and the account locks again immediately:

result of one failure right after unlock: { attempts: 6, locked: true }

So "unlocked" didn't mean "clean slate" here — it meant "one more mistake and you're back in." That might be exactly the behavior you want (harsher consequences for someone who fails again right after a lockout), but it should be a choice, not a side effect of two numbers that happened to get picked independently. Defaulting LOGIN_LOCKOUT_WINDOW_SECONDS and LOGIN_LOCKOUT_DURATION_SECONDS to the same value is what makes the out-of-the-box behavior a genuine reset: by the time the lock is gone, so is the counter it was based on. Tested with matched timers instead of mismatched ones, the positive case is exactly as boring as it should be — wait out the lock, and a correct password just works:

curl -X POST http://localhost:3000/auth/login -H "Content-Type: application/json" \
  -d '{"email":"alice@example.com","password":"correct-horse-battery"}'
# -> 201 { "accessToken": "..." }   (no failures logged anywhere afterward)

The failure that triggers a lock doesn't announce it

try {
  const user = await this.authService.validatePassword(dto.email, dto.password);
  await this.loginAttemptService.recordSuccess(dto.email);
  return { accessToken: this.authService.issueToken(user) };
} catch (err) {
  await this.loginAttemptService.recordFailure(dto.email);
  throw err; // same error whether or not this failure just triggered a lock
}

Four wrong-password attempts return four identical 401s. The fifth — the one that actually crosses the threshold — returns that same 401, not a warning. The lock only becomes visible on whatever comes next, correct password included. Confirmed end to end: four 401s, a fifth 401 that locked the account behind the scenes, and only the sixth request got the 429. Nothing in that fifth response tells anyone it was the last try, which is one less signal an attacker gets to calibrate around.

Both mechanisms, same request

# alice fails 5 times, then even the right password is rejected
curl -X POST http://localhost:3000/auth/login -H "Content-Type: application/json" \
  -d '{"email":"alice@example.com","password":"wrong-password"}'
# -> 401  (x5)

curl -X POST http://localhost:3000/auth/login -H "Content-Type: application/json" \
  -d '{"email":"alice@example.com","password":"correct-horse-battery"}'
# -> 429 {"statusCode":429,"message":"Account temporarily locked...","reason":"ACCOUNT_LOCKED"}

# bob, meanwhile, is entirely unaffected
curl -X POST http://localhost:3000/auth/login -H "Content-Type: application/json" \
  -d '{"email":"bob@example.com","password":"correct-horse-battery"}'
# -> 201 { "accessToken": "..." }

The lockout 429 and the throttler's 429 share a status code but not a body — "reason":"ACCOUNT_LOCKED" versus the generic ThrottlerException message — so nothing downstream has to guess which mechanism fired.

One sequencing detail if you're replicating this yourself: the IP throttle counts every call to /login in its window, not just the ones in whatever you'd mentally group as "the throttle test." Running the lockout sequence above first, then immediately sending a batch of requests to check the throttle, tripped the 429 on the third request of that batch rather than somewhere near the tenth — because the eight requests just spent on the lockout scenario were already sitting in the same 60-second window. Not a bug, just a shared counter that doesn't know your test plan has phases.

Running it

Needs Postgres and Redis; Docker is the fastest path to both:

git clone <your-repo-url> && cd rate-limiting-nestjs-demo
npm install && cp .env.example .env
docker run --name rl-postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=ratelimit_demo -p 5432:5432 -d postgres:16
docker run --name rl-redis -p 6379:6379 -d redis:7
npm run start:dev

The shipped defaults (5 attempts, a 15-minute window and lock) are real production numbers, not demo placeholders, but they're still a starting point rather than a universal answer — how forgiving to be with a legitimate user who fumbles their password a few times is a judgment call specific to what's being protected. Lower LOGIN_LOCKOUT_WINDOW_SECONDS/LOGIN_LOCKOUT_DURATION_SECONDS in your own .env if you want to watch a lock expire without an actual 15-minute wait.

What's deliberately not here: the in-memory throttle storage this demo uses doesn't share counters across multiple app instances, so a horizontally-scaled deployment needs a shared store (@nestjs/throttler supports pluggable backends, Redis included) or the limit quietly stops meaning what it says. There's also no lockout notification to the account owner, and no CAPTCHA as a third layer for public-facing forms — both reasonable additions, both left out here to keep the two mechanisms that are built easy to see clearly.

The full project — this module, the lockout service, and a README with the complete setup and curl walkthrough — is in the rate-limiting-nestjs-demo repository alongside this post.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

User A clicks the link in Gmail. The prefetch scanner in their email client hits it half a second earlier. Server issues two sessions. Neither user knows, but one of them is now authenticated in a context they didn't control.

This post builds it right. By the end you'll have a working Node.js implementation with proper atomic token verification, email enumeration protection, an Ethereal fallback so you can see the full flow without any SMTP configuration, and 19 tests that run without Redis or a real mail server.

Source code: http://github.com/zyvop27-cmyk/zyvop-blogs/tree/main/magic-link-auth

The one decision that matters most

Before touching any code: the difference between a safe magic link implementation and a broken one is a single Redis command.

The broken pattern:

const email = await redis.get(key);
if (email) await redis.del(key);
return email;

Between the get and the del, another request can read the same key. Both get a valid email back, both get sessions issued.

getDel collapses that into one atomic operation:

const email = await redis.getDel(key);
return email ?? null;

Redis executes this as a single command. No window — one request gets the email, any concurrent request gets null.

That's why Redis fits this problem better than a Postgres row. You'd need a transaction and an advisory lock to get the same guarantee from a relational database.

The full verify function adds a length check before the Redis call. A token that's too short or too long gets rejected immediately, no round-trip needed:

// src/lib/token.js
import crypto from "node:crypto";

const TOKEN_PREFIX = "magic:";
const TOKEN_TTL_SECONDS = 15 * 60;
const TOKEN_BYTES = 32;

export async function createToken(redis, email) {
  const token = crypto.randomBytes(TOKEN_BYTES).toString("hex");
  await redis.set(TOKEN_PREFIX + token, email.toLowerCase(), { EX: TOKEN_TTL_SECONDS });
  return token;
}

export async function verifyToken(redis, token) {
  if (!token || typeof token !== "string" || token.length !== TOKEN_BYTES * 2) {
    return null;
  }
  return (await redis.getDel(TOKEN_PREFIX + token)) ?? null;
}

crypto.randomBytes(32) gives 256 bits of entropy — 64 hex characters. Brute-forcing that against a 15-minute window isn't happening. The email gets lowercased on creation so User@Example.COM and user@example.com don't end up as separate keys that never match.

Redis also handles expiry natively via the EX option. No cron job to clean up stale tokens, no WHERE expires_at < NOW() queries — they disappear on their own.

What the user actually sees

The sign-in page is minimal HTML with one real piece of JavaScript: it handles the ?error=link_invalid query param that the server redirects to when a token has expired or already been used.

<!-- public/index.html (abbreviated) -->
<form id="form">
  <input type="email" id="email" placeholder="you@example.com" required>
  <button type="submit">Send sign-in link</button>
</form>
<div id="msg" class="message"></div>

<script>
  const ERRORS = {
    link_invalid: "That link has expired or already been used. Please request a new one.",
    default: "Something went wrong. Please try again."
  };

  // Show error from redirect after a failed verify attempt
  const errKey = new URLSearchParams(location.search).get("error");
  if (errKey) showMsg(ERRORS[errKey] ?? ERRORS.default, "error");

  document.getElementById("form").addEventListener("submit", async (e) => {
    e.preventDefault();
    const email = document.getElementById("email").value.trim();
    const btn = e.target.querySelector("button");

    btn.disabled = true;
    btn.textContent = "Sending…";

    const res = await fetch("/auth/request", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ email })
    });

    const data = await res.json();

    if (res.ok) {
      showMsg("Check your inbox — a sign-in link is on its way.", "success");
      document.getElementById("form").style.display = "none";
    } else {
      showMsg(data.error ?? ERRORS.default, "error");
      btn.disabled = false;
      btn.textContent = "Send sign-in link";
    }
  });
</script>

The ?error=link_invalid flow matters more than it looks. When verification fails — expired token, already used, someone typed a URL wrong — it redirects to /?error=link_invalid instead of returning a JSON 400.

Users arrive at /auth/verify by clicking a link in their email client, not via a fetch call. A JSON error on a blank page is a dead end; a redirect back to the sign-in form with a clear message is something a person can act on.

Sending the email without configuring SMTP

Production email uses SMTP_HOST, SMTP_PORT, SMTP_USER, and SMTP_PASS. Without those, the mailer silently creates an Ethereal test account and logs a preview URL. Ethereal is a real catch-all SMTP service — emails don't deliver anywhere, but you can open the URL and see the full rendered email, including the magic link button, right in your browser.

// src/lib/mailer.js
import nodemailer from "nodemailer";

let transporter = null;

async function getTransporter() {
  if (transporter) return transporter;

  if (process.env.SMTP_HOST) {
    transporter = nodemailer.createTransport({
      host: process.env.SMTP_HOST,
      port: Number(process.env.SMTP_PORT) || 587,
      secure: process.env.SMTP_SECURE === "true",
      auth: { user: process.env.SMTP_USER, pass: process.env.SMTP_PASS }
    });
  } else {
    const testAccount = await nodemailer.createTestAccount();
    transporter = nodemailer.createTransport({
      host: "smtp.ethereal.email",
      port: 587,
      auth: { user: testAccount.user, pass: testAccount.pass }
    });
    console.log(`[mailer] Ethereal test account: ${testAccount.user}`);
  }

  return transporter;
}

export async function sendMagicLink(to, magicUrl) {
  const transport = await getTransporter();

  const info = await transport.sendMail({
    from: process.env.EMAIL_FROM || '"Magic Link Auth" <no-reply@example.com>',
    to,
    subject: "Your sign-in link",
    text: `Sign-in link (expires in 15 minutes, single use):\n\n${magicUrl}`,
    html: `
      <p>Click below to sign in. Expires in <strong>15 minutes</strong>, single use.</p>
      <p style="margin:24px 0">
        <a href="${magicUrl}" style="background:#111;color:#fff;padding:12px 24px;border-radius:6px;text-decoration:none">
          Sign in
        </a>
      </p>
      <p style="color:#999;font-size:12px">Didn't request this? Safe to ignore.</p>
    `
  });

  if (!process.env.SMTP_HOST) {
    console.log(`[mailer] Preview URL: ${nodemailer.getTestMessageUrl(info)}`);
  }
}

The transporter is cached in a module-level variable. nodemailer.createTestAccount() makes a real HTTP call to Ethereal — you don't want that per-request.

For production, Resend and Postmark are the cleaner options over raw SMTP. They handle deliverability, bounce handling, and SPF/DKIM automatically. Hook them in via the SMTP_HOST and friends in .env.

Sessions, cookies, and why SameSite: lax is correct here

After verification, the token is gone and the user needs something that persists across requests. A JWT in an httpOnly cookie is the right shape: stateless (no server-side session table), survives page refreshes, inaccessible to JavaScript running on the page.

// src/lib/session.js
import jwt from "jsonwebtoken";

const COOKIE_NAME = "session";
const SESSION_TTL_SECONDS = 7 * 24 * 60 * 60;

function getSecret() {
  const secret = process.env.JWT_SECRET;
  if (!secret || secret.length < 32) {
    throw new Error("JWT_SECRET must be set and at least 32 characters long");
  }
  return secret;
}

export function issueSession(res, email) {
  const token = jwt.sign({ email }, getSecret(), { expiresIn: SESSION_TTL_SECONDS });
  res.cookie(COOKIE_NAME, token, {
    httpOnly: true,
    secure: process.env.NODE_ENV === "production",
    sameSite: "lax",
    maxAge: SESSION_TTL_SECONDS * 1000
  });
}

export function requireAuth(req, res, next) {
  const token = req.cookies?.[COOKIE_NAME];
  if (!token) return res.status(401).json({ error: "Not authenticated." });

  try {
    const payload = jwt.verify(token, getSecret());
    req.user = { email: payload.email };
    next();
  } catch {
    res.clearCookie(COOKIE_NAME);
    return res.status(401).json({ error: "Session expired. Please sign in again." });
  }
}

sameSite: "lax" needs explaining here because it interacts directly with how magic links work. When a user clicks a link in their email client, that's a top-level navigation — the browser follows it like a normal page load. Lax allows the cookie to be sent on those top-level navigations from external origins.

Strict would block that, requiring another sign-in if a user arrives from any external link. None requires HTTPS everywhere and opens CSRF exposure. Lax is the right call for this pattern.

secure only goes on in production. Without that conditional, local development over HTTP would silently fail to set the cookie and you'd spend an hour wondering why sessions don't persist.

The two routes that do the work

// src/routes/auth.js
import { Router } from "express";
import { createToken, verifyToken } from "../lib/token.js";
import { sendMagicLink as defaultSendMagicLink } from "../lib/mailer.js";
import { issueSession } from "../lib/session.js";

const EMAIL_RE = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;

export function createAuthRouter(redis, { sendMagicLink = defaultSendMagicLink } = {}) {
  const router = new Router();

  router.post("/request", async (req, res) => {
    const email = (req.body?.email ?? "").trim().toLowerCase();

    if (!EMAIL_RE.test(email)) {
      return res.status(400).json({ error: "A valid email address is required." });
    }

    try {
      const token = await createToken(redis, email);
      const appUrl = process.env.APP_URL || `http://localhost:${process.env.PORT || 3000}`;
      await sendMagicLink(email, `\({appUrl}/auth/verify?token=\){token}`);
      res.json({ ok: true, message: "Check your inbox for a sign-in link." });
    } catch (err) {
      console.error("[auth] request failed:", err);
      res.status(500).json({ error: "Failed to send the sign-in link. Please try again." });
    }
  });

  router.get("/verify", async (req, res) => {
    const email = await verifyToken(redis, req.query.token);
    if (!email) return res.redirect("/?error=link_invalid");
    issueSession(res, email);
    res.redirect("/dashboard");
  });

  router.post("/logout", (req, res) => {
    res.clearCookie("session");
    res.redirect("/");
  });

  return router;
}

/request returns 200 whether the email exists in your system or not. A 404 for unknown emails would tell anyone who tries that an address isn't registered — a user enumeration leak. The response is always "check your inbox," whether you're a real user or someone probing your database.

The sendMagicLink function is passed in as a default parameter, not imported at the top. That's the dependency injection hook tests use — swap it for a no-op, no SMTP connection required.

Running the whole thing

# Start Redis (Docker is fastest)
docker run -d -p 6379:6379 redis:alpine

# Install deps and start
npm install && cp .env.example .env && npm start

The server needs at minimum a JWT_SECRET in .env. Generate one:

node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

Request a link:

curl -X POST http://localhost:3000/auth/request \
  -H "Content-Type: application/json" \
  -d '{"email": "you@example.com"}'

{ "ok": true, "message": "Check your inbox for a sign-in link." }

Your server console will print something like:

[mailer] Preview URL: https://ethereal.email/message/WaQKMgKddxQDoou

Open that URL, click the "Sign in" button in the rendered email. You'll land on the dashboard. The token is gone — clicking the link again redirects to /?error=link_invalid.

Check your session after clicking (saves cookie to cookies.txt):

curl -c cookies.txt -b cookies.txt http://localhost:3000/api/me

{ "email": "you@example.com" }

Invalid email format:

curl -s -X POST http://localhost:3000/auth/request \
  -H "Content-Type: application/json" \
  -d '{"email": "notanemail"}'

{ "error": "A valid email address is required." }

Rate limiter — 5 requests per 15 minutes per IP, sixth gets a 429:

for i in $(seq 1 6); do
  curl -s -o /dev/null -w "request $i -> %{http_code}\n" \
    -X POST http://localhost:3000/auth/request \
    -H "Content-Type: application/json" \
    -d '{"email":"test@example.com"}'
done

request 1 -> 200
request 2 -> 200
request 3 -> 200
request 4 -> 200
request 5 -> 200
request 6 -> 429

Five is tight enough to stop inbox flooding and loose enough that a real user who typo'd their address gets a few retries.

Expired or already-used token:

curl -v "http://localhost:3000/auth/verify?token=$( python3 -c 'print("a"*64)')" 2>&1 | grep "Location:"

< Location: /?error=link_invalid

Tests

19 tests, no Redis or SMTP connection needed. The token suite runs against a plain in-memory Map mimicking the Redis interface; the session suite sets JWT_SECRET in before() and cleans up after; the route suite injects the no-op mailer:

npm test

# tests 19
# pass  19
# fail   0

If you want to verify the atomic single-use behavior beyond the unit test, run the server with a real Redis instance and hit /auth/verify with the same token from two curl commands fired in parallel:

TOKEN="paste-a-real-token-from-console-here"
curl "http://localhost:3000/auth/verify?token=$TOKEN" &
curl "http://localhost:3000/auth/verify?token=$TOKEN" &
wait

One will redirect to /dashboard. The other will redirect to /?error=link_invalid. That's getDel doing its job.

Before going live

Set NODE_ENV=production — the Secure cookie flag only activates in production, and without HTTPS the cookie won't be sent by browsers at all. Point REDIS_URL at a managed instance (Upstash has a free tier and works well with this setup). Swap Ethereal for a transactional email provider; Resend has a generous free tier and a clean Node.js SDK. Put the whole thing behind nginx or Caddy for TLS termination.

One thing the repo doesn't include but you'll want eventually: a users table or equivalent. Right now any email can request a link and get a session — there's no concept of "registered users." Adding a check in /request that verifies the email exists in your database before sending the link is one line, but the shape of that check depends on your stack.

Get the code: http://github.com/zyvop27-cmyk/zyvop-blogs/tree/main/magic-link-auth

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

Most Python developers write simple scrapers—requests, BeautifulSoup, a loop, CSV writer—just to get data once or twice. When scaling or running them over months, the real challenge is building a robust, scheduled system, not the parsing logic. The extraction code is tiny; the critical components are the queue, cache, storage, block detection, recovery loop, and exporters that keep the scraper alive against real‑world internet conditions.

At small scale, a single async Python process can handle thousands of pages per hour. But when you need to crawl millions of pages per day — competitor catalogues, job markets, news archives, e-commerce databases — a single machine hits hard limits: one IP address, one CPU, one point of failure.

Distributed scraping involves spreading tasks across multiple machines to increase speed and volume. Throttling and introducing random delays between requests can help to prevent IP bans, while rotating proxies help distribute requests and avoid detection. Managing sessions and leveraging parallel processing can further enhance efficiency.

This guide shows you exactly how to build a distributed scraping system that runs across multiple machines — using scrapy-redis for shared request queues, Docker for containerisation, and a production proxy management layer that survives real-world conditions.

The Architecture: One Queue, Many Workers

The core insight behind distributed scraping is simple: replace Scrapy's in-memory request queue with a shared Redis queue that every worker machine can read from.

┌─────────────────────────────────────────────────────────────┐
│                    DISTRIBUTED SCRAPER                      │
│                                                             │
│  Master                                                     │
│  ┌──────────┐    Seeds requests    ┌─────────┐             │
│  │  Spider  │──────────────────────▶│  Redis  │             │
│  │ (seed)   │                      │  Queue  │             │
│  └──────────┘                      └────┬────┘             │
│                                         │                   │
│  Workers (any number of machines)       │                   │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                  │
│  │ Worker 1 │  │ Worker 2 │  │ Worker N │  ← pull jobs     │
│  │ (spider) │  │ (spider) │  │ (spider) │                  │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘                  │
│       │              │              │                        │
│       └──────────────┴──────────────┘                       │
│                          │                                   │
│                    ┌─────▼──────┐                           │
│                    │  MongoDB   │  ← all workers write here │
│                    │ (results)  │                           │
│                    └────────────┘                           │
└─────────────────────────────────────────────────────────────┘

scrapy-redis lets multiple spider instances across machines pull from the same request queue with deduplication and job scheduling. This is the standard approach for large-scale web scraping that Python teams use in production.

Part 1: Setting Up scrapy-redis

pip install scrapy scrapy-redis redis pymongo

The distributed spider

# spiders/distributed_product_spider.py
import scrapy
from scrapy_redis.spiders import RedisSpider
from urllib.parse import urljoin
from datetime import datetime, timezone

class DistributedProductSpider(RedisSpider):
    """
    A Scrapy spider that pulls start URLs from a Redis list
    instead of a hardcoded start_urls list.

    To start crawling, push seed URLs to Redis:
        redis-cli lpush products:start_urls "https://example-store.com/products/"

    Any number of workers running this spider will cooperatively
    process the shared queue — each URL is processed exactly once.
    """

    name         = "distributed_products"
    redis_key    = "products:start_urls"   # Redis list to pop URLs from

    # How many URLs to pop from Redis at once per worker
    redis_batch_size = 16

    # Spider-level settings — override settings.py per spider
    custom_settings = {
        "CONCURRENT_REQUESTS":            32,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 8,
        "DOWNLOAD_DELAY":                 0.5,
        "RANDOMIZE_DOWNLOAD_DELAY":       True,
        "AUTOTHROTTLE_ENABLED":           True,
        "AUTOTHROTTLE_TARGET_CONCURRENCY": 16.0,
        "AUTOTHROTTLE_MAX_DELAY":         5.0,
        "RETRY_TIMES":                    3,
        "RETRY_HTTP_CODES":              [429, 500, 502, 503, 504],
    }

    def parse(self, response):
        """
        Parse a product listing page.
        Yields product detail requests AND discovers pagination links.
        """
        # Follow product links to detail pages
        for href in response.css("a.product-link::attr(href)").getall():
            yield response.follow(href, callback=self.parse_product)

        # Auto-discover pagination — push next page back to Redis queue
        next_page = response.css("a[rel='next']::attr(href)").get()
        if next_page:
            # Use follow() to handle relative URLs
            yield response.follow(next_page, callback=self.parse)

    def parse_product(self, response):
        """Extract product data from a detail page."""
        yield {
            "url":        response.url,
            "title":      response.css("h1::text").get("").strip(),
            "price":      response.css(".price::text").get("").strip(),
            "sku":        response.css("[data-sku]::attr(data-sku)").get(),
            "in_stock":   bool(response.css(".in-stock")),
            "description":response.css(".product-description::text").get("").strip()[:500],
            "scraped_at": datetime.now(timezone.utc).isoformat(),
            "worker_id":  self.settings.get("WORKER_ID", "unknown"),
        }

settings.py for distributed mode

# settings.py

BOT_NAME = "distributed_scraper"
SPIDER_MODULES = ["spiders"]

# ── scrapy-redis settings ─────────────────────────────────────
SCHEDULER            = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS     = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_URL            = "redis://redis-host:6379"

# Keep crawl state across restarts — resumable crawls
SCHEDULER_PERSIST    = True

# How long to wait for new URLs before worker shuts down
SCHEDULER_IDLE_BEFORE_CLOSE = 30

# ── MongoDB pipeline ──────────────────────────────────────────
ITEM_PIPELINES = {
    "pipelines.MongoPipeline":        100,
    "pipelines.DuplicateFilterPipeline": 50,
}
MONGO_URI      = "mongodb://mongo-host:27017/"
MONGO_DATABASE = "distributed_scrape"

# ── Concurrency ───────────────────────────────────────────────
CONCURRENT_REQUESTS              = 32
CONCURRENT_REQUESTS_PER_DOMAIN   = 8
DOWNLOAD_DELAY                   = 0.5
RANDOMIZE_DOWNLOAD_DELAY         = True

# ── Retry ─────────────────────────────────────────────────────
RETRY_ENABLED    = True
RETRY_TIMES      = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524]

# ── Logging ───────────────────────────────────────────────────
LOG_LEVEL = "INFO"

# ── Downloader middlewares ────────────────────────────────────
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
    "middlewares.RotatingProxyMiddleware": 350,
    "middlewares.UserAgentRotationMiddleware": 400,
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
}

# ── User agent pool ───────────────────────────────────────────
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/119.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) Chrome/118.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
]

Part 2: Production Middlewares

# middlewares.py
import random
import logging
from scrapy import signals
from scrapy.exceptions import NotConfigured

logger = logging.getLogger(__name__)

class UserAgentRotationMiddleware:
    """Rotate User-Agent on every request."""

    def __init__(self, user_agents):
        self.user_agents = user_agents

    @classmethod
    def from_crawler(cls, crawler):
        agents = crawler.settings.getlist("USER_AGENTS")
        if not agents:
            raise NotConfigured("USER_AGENTS not set")
        return cls(agents)

    def process_request(self, request, spider):
        request.headers["User-Agent"] = random.choice(self.user_agents)

class RotatingProxyMiddleware:
    """
    Rotate through a proxy pool.
    Tracks failures per proxy and removes bad proxies from the pool.
    """

    def __init__(self, proxies, max_failures=5):
        self.proxies      = list(proxies)
        self.failures     = {}
        self.max_failures = max_failures

    @classmethod
    def from_crawler(cls, crawler):
        proxies = crawler.settings.getlist("PROXY_LIST", [])
        if not proxies:
            raise NotConfigured("PROXY_LIST is empty")
        return cls(proxies)

    def process_request(self, request, spider):
        if not self.proxies:
            return  # No proxies left — run without
        proxy = random.choice(self.proxies)
        request.meta["proxy"] = proxy

    def process_response(self, request, response, spider):
        if response.status in (403, 407, 429):
            proxy = request.meta.get("proxy")
            self._mark_failure(proxy)
        return response

    def process_exception(self, request, exception, spider):
        proxy = request.meta.get("proxy")
        self._mark_failure(proxy)

    def _mark_failure(self, proxy):
        if not proxy:
            return
        self.failures[proxy] = self.failures.get(proxy, 0) + 1
        if self.failures[proxy] >= self.max_failures:
            if proxy in self.proxies:
                self.proxies.remove(proxy)
                logger.warning(f"Removed bad proxy: {proxy} ({self.max_failures} failures)")

class BlockDetectionMiddleware:
    """
    Detect common block patterns and trigger retries with a different proxy.
    """

    BLOCK_PATTERNS = [
        "access denied", "captcha", "blocked", "forbidden",
        "unusual traffic", "robot", "automated queries",
        "cf-browser-verification", "ddos-guard",
    ]

    def process_response(self, request, response, spider):
        body_lower = response.text[:2000].lower()

        is_blocked = (
            response.status in (403, 429, 503) or
            any(p in body_lower for p in self.BLOCK_PATTERNS) or
            len(response.text) < 300
        )

        if is_blocked:
            logger.warning(f"Block detected on {request.url} — retrying")
            request.meta["proxy"]    = None  # Force new proxy on retry
            request.dont_filter      = True
            return request           # Re-schedule the request

        return response

Part 3: MongoDB Pipeline with Bulk Writes

# pipelines.py
import logging
from datetime import datetime, timezone
from pymongo import MongoClient, UpdateOne
from pymongo.errors import BulkWriteError
from itemadapter import ItemAdapter

logger = logging.getLogger(__name__)

class DuplicateFilterPipeline:
    """Track seen URLs in memory to drop duplicates before DB write."""

    def open_spider(self, spider):
        self.seen_urls = set()

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        url = adapter.get("url", "")
        if url in self.seen_urls:
            from scrapy.exceptions import DropItem
            raise DropItem(f"Duplicate URL: {url}")
        self.seen_urls.add(url)
        return item

class MongoPipeline:
    """
    Write scraped items to MongoDB using bulk operations.
    Upserts on URL — safe to re-run without creating duplicates.
    """

    BULK_SIZE  = 200   # Flush every 200 items
    COLLECTION = "products"

    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db  = mongo_db
        self._buffer   = []

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get("MONGO_URI", "mongodb://localhost:27017/"),
            mongo_db =crawler.settings.get("MONGO_DATABASE", "scrapy_data"),
        )

    def open_spider(self, spider):
        self.client     = MongoClient(self.mongo_uri)
        self.col        = self.client[self.mongo_db][self.COLLECTION]
        self.col.create_index("url", unique=True)
        self.items_written = 0
        logger.info(f"MongoDB connected: {self.mongo_db}.{self.COLLECTION}")

    def close_spider(self, spider):
        if self._buffer:
            self._flush()
        self.client.close()
        logger.info(f"MongoDB closed. Total items written: {self.items_written}")

    def process_item(self, item, spider):
        self._buffer.append(dict(item))
        if len(self._buffer) >= self.BULK_SIZE:
            self._flush()
        return item

    def _flush(self):
        ops = [
            UpdateOne({"url": doc["url"]}, {"$set": doc}, upsert=True)
            for doc in self._buffer
        ]
        try:
            result = self.col.bulk_write(ops, ordered=False)
            count  = result.upserted_count + result.modified_count
            self.items_written += count
            logger.info(f"Flushed {len(self._buffer)} items → MongoDB")
        except BulkWriteError as e:
            logger.error(f"Bulk write error: {e.details.get('writeErrors', [])[:2]}")
        finally:
            self._buffer.clear()

Part 4: Docker Compose — Multi-Worker Setup

# docker-compose.yml
version: "3.9"

x-worker-base: &worker-base
  build: .
  volumes:
    - .:/app
  environment:
    - REDIS_URL=redis://redis:6379
    - MONGO_URI=mongodb://mongo:27017/
    - PROXY_LIST=${PROXY_LIST}
  depends_on:
    - redis
    - mongo
  restart: unless-stopped

services:

  # ── Infrastructure ──────────────────────────────────────────
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
    command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru
    volumes: ["redis_data:/data"]

  mongo:
    image: mongo:7
    ports: ["27017:27017"]
    volumes: ["mongo_data:/data/db"]

  # ── Scrapy Workers ──────────────────────────────────────────
  # Run as many of these as you have cores / IPs
  worker-1:
    <<: *worker-base
    command: >
      scrapy crawl distributed_products
      -s WORKER_ID=worker-1
      -s CONCURRENT_REQUESTS=16
    environment:
      - REDIS_URL=redis://redis:6379
      - MONGO_URI=mongodb://mongo:27017/
      - WORKER_ID=worker-1

  worker-2:
    <<: *worker-base
    command: >
      scrapy crawl distributed_products
      -s WORKER_ID=worker-2
      -s CONCURRENT_REQUESTS=16
    environment:
      - REDIS_URL=redis://redis:6379
      - MONGO_URI=mongodb://mongo:27017/
      - WORKER_ID=worker-2

  worker-3:
    <<: *worker-base
    command: >
      scrapy crawl distributed_products
      -s WORKER_ID=worker-3
      -s CONCURRENT_REQUESTS=16
    environment:
      - REDIS_URL=redis://redis:6379
      - MONGO_URI=mongodb://mongo:27017/
      - WORKER_ID=worker-3

  # ── Seed Service — pushes start URLs into Redis ─────────────
  seeder:
    <<: *worker-base
    command: python seeder.py
    restart: "no"   # Run once then exit

volumes:
  redis_data:
  mongo_data:

Part 5: The Seeder — Feeding URLs Into the Queue

# seeder.py
import redis
import time
import sys
from urllib.parse import urlencode

REDIS_URL   = "redis://localhost:6379"
REDIS_KEY   = "products:start_urls"

def seed_from_list(urls: list[str], batch_size: int = 500):
    """Push a list of start URLs into the Redis queue."""
    r = redis.from_url(REDIS_URL)

    # Clear existing queue if resuming fresh
    existing = r.llen(REDIS_KEY)
    if existing > 0:
        print(f"Queue already has {existing:,} URLs. Adding to it.")

    pushed = 0
    for i in range(0, len(urls), batch_size):
        batch = urls[i:i + batch_size]
        r.rpush(REDIS_KEY, *batch)
        pushed += len(batch)
        print(f"Seeded {pushed:,}/{len(urls):,} URLs")

    print(f"\nDone. Redis queue '{REDIS_KEY}' has {r.llen(REDIS_KEY):,} URLs.")

def seed_paginated_site(
    base_url: str,
    start_page: int = 1,
    end_page: int = 500,
    page_param: str = "page"
):
    """Generate paginated URLs and push to Redis."""
    urls = []
    for page in range(start_page, end_page + 1):
        params = {page_param: page}
        urls.append(f"{base_url}?{urlencode(params)}")

    seed_from_list(urls)

def monitor_queue():
    """Monitor queue depth and worker progress in real time."""
    r = redis.from_url(REDIS_URL)
    print("Monitoring queue depth (Ctrl+C to stop)...")
    try:
        while True:
            depth   = r.llen(REDIS_KEY)
            seen    = r.scard(f"{REDIS_KEY}:dupefilter") or 0
            print(f"  Queue: {depth:,} pending | Seen: {seen:,} processed", end="\r")
            time.sleep(2)
    except KeyboardInterrupt:
        print("\nMonitor stopped.")

if __name__ == "__main__":
    mode = sys.argv[1] if len(sys.argv) > 1 else "seed"

    if mode == "seed":
        seed_paginated_site(
            base_url  = "https://example-store.com/products",
            start_page = 1,
            end_page   = 1000,
        )
    elif mode == "monitor":
        monitor_queue()
    elif mode == "clear":
        r = redis.from_url(REDIS_URL)
        r.delete(REDIS_KEY)
        print(f"Queue '{REDIS_KEY}' cleared.")

Part 6: Scaling to Kubernetes

For truly large-scale crawling across dozens of machines, Kubernetes is the standard deployment target. The key insight: each Scrapy worker is a stateless pod that reads from the shared Redis queue.

# k8s/scrapy-worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: scrapy-workers
  labels:
    app: scrapy-worker
spec:
  replicas: 10   # Start with 10 workers; scale up/down with kubectl
  selector:
    matchLabels:
      app: scrapy-worker
  template:
    metadata:
      labels:
        app: scrapy-worker
    spec:
      containers:
        - name: scrapy-worker
          image: your-registry/scrapy-worker:latest
          command:
            - scrapy
            - crawl
            - distributed_products
          env:
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: scraper-secrets
                  key: redis-url
            - name: MONGO_URI
              valueFrom:
                secretKeyRef:
                  name: scraper-secrets
                  key: mongo-uri
            - name: WORKER_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name  # Pod name as worker ID
          resources:
            requests:
              memory: "256Mi"
              cpu:    "250m"
            limits:
              memory: "512Mi"
              cpu:    "500m"

Scale workers up or down instantly:

# Scale to 20 workers
kubectl scale deployment scrapy-workers --replicas=20

# Check worker status
kubectl get pods -l app=scrapy-worker

# View logs from all workers
kubectl logs -l app=scrapy-worker --tail=50

# Auto-scale based on Redis queue depth (requires custom metrics)
kubectl autoscale deployment scrapy-workers --min=2 --max=50

Part 7: Proxy Management at Scale

For serious scraping in 2026, residential proxies are almost always the safer option. Proxy quality matters far more than proxy quantity. A smaller pool of clean residential IPs usually performs much better than massive low-quality networks.

Here's a production proxy manager with health checking:

# proxy_manager.py
import asyncio
import httpx
import random
import time
import json
from dataclasses import dataclass, field
from typing import Optional

@dataclass
class ProxyHealth:
    url:              str
    success_count:    int   = 0
    failure_count:    int   = 0
    last_used:        float = 0.0
    last_success:     float = 0.0
    avg_response_ms:  float = 0.0
    is_banned:        bool  = False

    @property
    def success_rate(self) -> float:
        total = self.success_count + self.failure_count
        return self.success_count / total if total > 0 else 0.0

    @property
    def score(self) -> float:
        """Composite score: higher = better proxy to use."""
        if self.is_banned:
            return 0.0
        recency_bonus = max(0, 1 - (time.time() - self.last_success) / 3600)
        speed_score   = max(0, 1 - self.avg_response_ms / 5000)
        return self.success_rate * 0.6 + recency_bonus * 0.2 + speed_score * 0.2

class ProxyPool:
    """
    Intelligent proxy pool with health tracking and weighted selection.
    Workers register success/failure, pool learns which proxies are best.
    """

    BAN_THRESHOLD = 0.2   # Mark as banned if success rate drops below 20%

    def __init__(self, proxy_urls: list[str]):
        self.proxies = {url: ProxyHealth(url=url) for url in proxy_urls}

    def get_proxy(self, strategy: str = "weighted") -> Optional[str]:
        """Select a proxy using the specified strategy."""
        available = [
            p for p in self.proxies.values()
            if not p.is_banned
        ]

        if not available:
            return None

        if strategy == "random":
            return random.choice(available).url

        elif strategy == "weighted":
            # Weight by score — best proxies get used more often
            scores = [max(p.score, 0.01) for p in available]
            total  = sum(scores)
            weights = [s / total for s in scores]
            return random.choices(available, weights=weights)[0].url

        elif strategy == "round_robin":
            # Sort by last_used timestamp — use least recently used
            available.sort(key=lambda p: p.last_used)
            return available[0].url

        return available[0].url

    def report_success(self, proxy_url: str, response_ms: float):
        if proxy_url in self.proxies:
            p = self.proxies[proxy_url]
            p.success_count  += 1
            p.last_used       = time.time()
            p.last_success    = time.time()
            # Rolling average response time
            p.avg_response_ms = (p.avg_response_ms * 0.8 + response_ms * 0.2)

    def report_failure(self, proxy_url: str):
        if proxy_url in self.proxies:
            p = self.proxies[proxy_url]
            p.failure_count += 1
            p.last_used      = time.time()
            # Auto-ban proxies with very low success rate
            if p.failure_count > 10 and p.success_rate < self.BAN_THRESHOLD:
                p.is_banned = True
                print(f"  Proxy banned (success rate {p.success_rate:.0%}): {proxy_url}")

    def get_stats(self) -> dict:
        active  = [p for p in self.proxies.values() if not p.is_banned]
        banned  = [p for p in self.proxies.values() if p.is_banned]
        avg_sr  = sum(p.success_rate for p in active) / len(active) if active else 0

        return {
            "total":    len(self.proxies),
            "active":   len(active),
            "banned":   len(banned),
            "avg_success_rate": f"{avg_sr:.1%}",
            "best_proxy": max(active, key=lambda p: p.score).url if active else None,
        }

async def health_check_proxies(pool: ProxyPool, test_url: str = "https://httpbin.org/ip"):
    """
    Periodically check all proxies and un-ban those that have recovered.
    Run this as a background task.
    """
    async with httpx.AsyncClient(timeout=10) as client:
        for url, proxy in list(pool.proxies.items()):
            if not proxy.is_banned:
                continue
            try:
                start = time.time()
                r = await client.get(test_url, proxies={"https": url})
                if r.status_code == 200:
                    elapsed_ms    = (time.time() - start) * 1000
                    proxy.is_banned = False
                    pool.report_success(url, elapsed_ms)
                    print(f"  Proxy recovered: {url}")
            except Exception:
                pass   # Still banned

    stats = pool.get_stats()
    print(f"Proxy pool: {stats['active']} active, {stats['banned']} banned, "
          f"avg success rate: {stats['avg_success_rate']}")

Part 8: Monitoring Your Distributed Crawl

# monitor.py — real-time crawl progress dashboard
import redis
import time
import json
from pymongo import MongoClient
from datetime import datetime

def live_dashboard(redis_url: str, mongo_uri: str, refresh_seconds: int = 5):
    """Print a live crawl progress dashboard to the terminal."""
    r   = redis.from_url(redis_url)
    db  = MongoClient(mongo_uri)["distributed_scrape"]

    start_time   = time.time()
    prev_count   = 0

    try:
        while True:
            # Queue stats
            queue_depth  = r.llen("products:start_urls")
            seen_count   = r.scard("products:start_urls:dupefilter") or 0

            # DB stats
            items_stored = db["products"].count_documents({})
            items_delta  = items_stored - prev_count
            rate_per_min = items_delta * (60 / refresh_seconds)
            prev_count   = items_stored

            # Worker stats (scrapy-redis stores worker heartbeats)
            workers = r.smembers("scrapy:workers") or set()

            # Elapsed
            elapsed = time.time() - start_time
            h, m    = divmod(int(elapsed), 3600)
            m, s    = divmod(m, 60)

            print(f"\033[2J\033[H")   # Clear screen
            print(f"{'═'*55}")
            print(f"  DISTRIBUTED SCRAPE MONITOR — {datetime.now().strftime('%H:%M:%S')}")
            print(f"{'═'*55}")
            print(f"  Runtime:       {h:02d}h {m:02d}m {s:02d}s")
            print(f"  Active workers:{len(workers)}")
            print(f"{'─'*55}")
            print(f"  Queue depth:   {queue_depth:>10,}  (URLs remaining)")
            print(f"  URLs seen:     {seen_count:>10,}  (deduplicated total)")
            print(f"  Items stored:  {items_stored:>10,}  (in MongoDB)")
            print(f"  Rate:          {rate_per_min:>10.0f}  items/minute")
            print(f"{'─'*55}")

            if rate_per_min > 0 and queue_depth > 0:
                eta_mins = queue_depth / rate_per_min
                h2, m2   = divmod(int(eta_mins * 60), 3600)
                m2, s2   = divmod(m2, 60)
                print(f"  ETA:           {h2:02d}h {m2:02d}m  (estimated)")

            print(f"{'═'*55}")
            time.sleep(refresh_seconds)

    except KeyboardInterrupt:
        print("\nMonitor stopped.")

if __name__ == "__main__":
    live_dashboard(
        redis_url="redis://localhost:6379",
        mongo_uri="mongodb://localhost:27017/",
    )

Part 9: Resumable Crawls

One of the biggest advantages of scrapy-redis is that crawls are inherently resumable. If a worker crashes or you need to add more workers mid-crawl, simply restart:

# Start a fresh crawl
python seeder.py seed

# Launch workers (they'll pick up from where they left off if SCHEDULER_PERSIST=True)
docker-compose up --scale worker=5

# Pause all workers (Ctrl+C in docker-compose)
# Resume later — queue state is preserved in Redis
docker-compose up --scale worker=10   # Can add more workers

To completely reset a crawl:

# Clear the queue and deduplication filter
redis-cli del products:start_urls
redis-cli del products:start_urls:dupefilter
python seeder.py seed   # Re-seed with fresh URLs

Performance Numbers: What to Expect

One machine first, then scale out. Tune Scrapy performance optimization settings before throwing hardware at the problem — raise CONCURRENT_REQUESTS, turn on AUTOTHROTTLE, and enable HTTP caching. If one server can't handle scraping millions of pages, set up a shared queue for distributed crawling across multiple workers.

Real-world throughput benchmarks:

Summary

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

One row per token, one family per login

@Entity()
export class RefreshToken {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  @Column()
  userId: string;

  @Index()
  @Column()
  familyId: string;

  @Index({ unique: true })
  @Column()
  tokenHash: string;

  @Column({ default: false })
  used: boolean;

  @Column({ default: false })
  revoked: boolean;

  @Column({ type: 'timestamp' })
  expiresAt: Date;

  @CreateDateColumn()
  createdAt: Date;
}

familyId is what makes "family" more than a metaphor — it's set once, at login, and then carried forward unchanged into every row that session's rotation produces afterward:

async login(@Body() dto: LoginDto) {
  const user = await this.authService.validatePassword(dto.email, dto.password);
  const accessToken = this.authService.issueAccessToken(user.id, user.email);
  const refresh = await this.refreshTokenService.issue(user.id); // no familyId = fresh session
  return { accessToken, refreshToken: refresh.rawToken };
}

Omitting familyId here is the signal that this is a brand new session rather than a rotation — issue() generates a fresh one. Every later call to issue() for this same chain, from inside rotate(), passes that same value back in instead. Two logins from the same user, at the same time, produce two completely independent families — there's nothing linking them beyond sharing a userId, which matters later.

Three ordinary rejections

A token that was never real. Someone sends a string that doesn't hash to anything in the table.

if (!stored) {
  throw new UnauthorizedException('Invalid refresh token');
}

curl -X POST http://localhost:3000/auth/refresh -H "Content-Type: application/json" \
  -d '{"refreshToken":"not-a-real-token-at-all"}'
# -> 401 {"message":"Invalid refresh token", ...}

A token that aged out. Refresh tokens in this implementation live 7 days by default. Past that, they're just gone.

if (stored.expiresAt < new Date()) {
  throw new UnauthorizedException('Refresh token has expired');
}

Confirmed directly rather than waiting a week: backdate a token's expiresAt in Postgres by a day, then try to use it —

{"message":"Refresh token has expired","error":"Unauthorized","statusCode":401}

— exactly the branch that's supposed to fire.

A token whose session already ended. Logout, or (as covered below) a reuse event elsewhere in the same family, flips a revoked flag.

if (stored.revoked) {
  throw new UnauthorizedException('Refresh token has been revoked');
}

None of these three are interesting on their own — they're the normal outcomes of a token being wrong, old, or deliberately ended. The fourth rejection reason looks identical from the outside (same 401, same shape) but means something completely different underneath.

The fourth: a token that's already been spent

if (stored.used) {
  await this.revokeFamily(stored.familyId);
  throw new UnauthorizedException('Refresh token reuse detected; session revoked');
}

stored.used = true;
await this.refreshTokenRepository.save(stored);

return this.issue(stored.userId, stored.familyId);

Every successful refresh marks the token it consumed as used before issuing its replacement. A legitimate client only ever moves forward through that chain — it gets a new token and uses that one next time, never the old one again.

So if an already-used token shows up in a request, the client presenting it isn't following the chain. Either it's a genuine client retrying a request whose response got lost on the way back (a real possibility, not a hypothetical), or it's a second party holding a copy of a token the real owner has already moved past.

There's no way to tell those apart from inside this one request — both look exactly like "this exact token, again" — so both get treated as the same signal: something's wrong with this session, not just this token.

That's why the response isn't "reject this token and move on." It's revokeFamily, which doesn't touch just the token that got reused — it kills every unrevoked row sharing that familyId, including whichever token the legitimate client is currently holding as its "real" one.

Proving that actually happens, not just reading the code and assuming: rotate once (RT1 → RT2), then try RT1 again.

curl -X POST http://localhost:3000/auth/refresh -H "Content-Type: application/json" -d '{"refreshToken":"<RT1>"}'
# -> 401 {"message":"Refresh token reuse detected; session revoked", ...}

RT1 failing is expected — it's already used. The real test is what happens to RT2, which was never itself reused, was legitimately issued, and — if the family weren't revoked — should still work fine:

curl -X POST http://localhost:3000/auth/refresh -H "Content-Type: application/json" -d '{"refreshToken":"<RT2>"}'
# -> 401 {"message":"Refresh token has been revoked", ...}

RT2 dies too, and the error message even confirms why — not "reuse detected" (that already happened, on RT1) but plain "revoked," which is exactly what a downstream consequence of someone else's reuse should look like from RT2's own perspective. This is the actual security property, not the headline description of it: a compromise anywhere in the chain takes down the entire chain, including the part that was never touched.

The blast radius runs the other direction too, and it's worth confirming it actually stops where it should: a second user, bob@example.com, with his own completely separate family, refreshes without incident while all of the above is happening to alice's account. Nothing about bob's session is touched — revokeFamily only ever operates on rows sharing the one familyId it was given, and bob's family was never that one. The failure is total within a compromised family and entirely absent outside it.

Logging out is the same mechanism, aimed on purpose

Every revocation above happened as a side effect of something going wrong. Logout is the identical underlying operation — revokeFamily — called directly, when nothing is wrong at all:

async revokeFamilyByToken(rawToken: string): Promise<void> {
  const stored = await this.refreshTokenRepository.findOne({
    where: { tokenHash: this.hash(rawToken) },
  });
  if (stored) {
    await this.revokeFamily(stored.familyId);
  }
}

curl -X POST http://localhost:3000/auth/logout -H "Content-Type: application/json" \
  -d '{"refreshToken":"<RT>"}'
# -> { "message": "Logged out" }

curl -X POST http://localhost:3000/auth/refresh -H "Content-Type: application/json" \
  -d '{"refreshToken":"<RT>"}'
# -> 401 {"message":"Refresh token has been revoked", ...}

One question this raises immediately: does logging out on one device end every session, or just the one that called it? Worth an actual answer instead of a guess — log in twice for the same user (call them a phone session and a laptop session, two independent families), log out using only the phone's token, then try refreshing each:

phone session, after logout: {"message":"Refresh token has been revoked", ...}
laptop session, untouched:   { "accessToken": "...", "refreshToken": "..." }   (succeeds normally)

Logging out ends the one family attached to the token that was actually presented. It has no way to reach a different family for the same user, because nothing about revokeFamily looks at userId at all — only familyId. Two logins are two unrelated chains that happen to belong to the same person, and ending one says nothing about the other.

What doesn't die with the family

Everything above is about the refresh token. The access token issued alongside RT2 — call it AT2 — is a separate, stateless JWT with its own 15-minute clock, and revoking a refresh family doesn't reach into that clock:

curl http://localhost:3000/auth/me -H "Authorization: Bearer <AT2>"
# -> 200 { "id": "...", "email": "alice@example.com" }

Issued before the reuse event above, and it still works after the whole family got revoked. That's not a gap in the implementation — it's the actual reason access tokens are kept short-lived in the first place. There's no database row for an access token to flip a revoked bit on; the only thing bounding how long a leaked one stays useful is its own expiry. Fifteen minutes is the real ceiling on "how bad is it if this specific token leaks," independent of anything the refresh layer detects or reacts to.

Why the token hash is SHA-256, not bcrypt

The password column in this same project uses bcrypt, and it should — passwords are short, human-chosen, and drawn from a relatively small effective space, so a hash that's deliberately slow is what makes brute-forcing a stolen hash impractical.

A refresh token is a different kind of secret: 40 bytes of crypto.randomBytes, 320 bits of pure randomness, with no human-guessable structure to exploit. Brute-forcing a hash of that is infeasible no matter how fast the hash function is, so bcrypt's deliberate slowness buys nothing here — it just adds cost to every single refresh call for no security benefit. A plain, fast sha256 is the correct tool once the secret's strength comes from entropy rather than from making guessing expensive.

Trying this out, and what it doesn't cover

npm install && cp .env.example .env
docker run --name refresh-postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=refresh_demo -p 5432:5432 -d postgres:16
npm run start:dev

synchronize: true handles table creation for this demo; swap it for real migrations before this touches production. Register and log in to get real values for the <RT1>/<RT2>/<AT2> placeholders used throughout above:

curl -X POST http://localhost:3000/auth/register -H "Content-Type: application/json" \
  -d '{"email":"alice@example.com","password":"correct-horse-battery"}'

curl -X POST http://localhost:3000/auth/login -H "Content-Type: application/json" \
  -d '{"email":"alice@example.com","password":"correct-horse-battery"}'
# -> { "accessToken": "<AT1>", "refreshToken": "<RT1>" }

Left out on purpose:

• A cleanup job for expired and revoked rows. They'll otherwise just accumulate — the BullMQ patterns from elsewhere in this series are a natural fit for sweeping them out.

• Rate limiting on /auth/login and /auth/refresh. Both are obvious targets, and the earlier rate-limiting post in this series covers them directly.

• A grace period for the "legitimate retry" half of the reuse ambiguity. Some production systems let a just-rotated token keep working for a few seconds, specifically to absorb a lost-response retry before treating reuse as a hard signal. This implementation takes the strict stance instead — simpler to reason about, simpler to verify — which is a deliberate tradeoff, not an oversight.

The full project, including the parts of this walkthrough not reproduced above, is in the refresh-token-rotation-demo repository next to this post.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

This post builds a complete URL shortener with analytics — short code generation, SQLite storage, IP geolocation, referrer tracking, and a dashboard showing daily click trends, browser breakdown, and country distribution.

No external analytics service, no database server to spin up. The whole thing runs on SQLite and ships as a single Node.js process.

Source code: https://github.com/zyvop27-cmyk/zyvop-blogs/tree/main/url-shortener

Stack and setup

Express 5 · SQLite (better-sqlite3) · geoip-lite · ua-parser-js · nanoid

git clone [YOUR_GITHUB_REPO_URL]
cd url-shortener
npm install
npm start

Open http://localhost:3000. The database creates itself in data/links.db on first run.

One environment variable is required before clicks will be tracked — HASH_SALT, used to hash IP addresses before storing them. Generate one:

node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

Add it to .env. Without it, the server starts but logs a warning and skips click recording on every redirect.

Short code generation

The first decision: how to make the codes. Random is the right choice over sequential integers — sequential IDs are trivially enumerable, someone can increment from 1 and hit every link ever created.

Random codes with a good alphabet make that infeasible.

The alphabet deliberately excludes characters that look alike:

// src/lib/shortcode.js
import { customAlphabet } from "nanoid";

const ALPHABET = "ABCDEFGHJKLMNPQRSTUVWXYZabcdefghjkmnpqrstuvwxyz23456789";
const CODE_LENGTH = 7;
const generate = customAlphabet(ALPHABET, CODE_LENGTH);

export function generateCode(db, existsFn, maxAttempts = 5) {
  for (let i = 0; i < maxAttempts; i++) {
    const code = generate();
    if (!existsFn(db, code)) return code;
  }
  throw new Error("Failed to generate a unique code after multiple attempts");
}

No 0, O, I, i, l, o, or 1 — seven characters that look like each other on printed paper or low-res screens. With 55 characters and 7 positions that's 55^7 ≈ 1.52 trillion possible codes. Collision handling is implemented correctly anyway (retry up to 5 times), but with that address space and any realistic link volume, it will never be needed.

The database

SQLite handles everything: URL storage, click records, and all the analytics queries. No Postgres, no Redis, no infrastructure to run. better-sqlite3 is synchronous, which means no connection pool to manage and no async/await ceremony around queries.

Two tables:

CREATE TABLE links (
  id         INTEGER PRIMARY KEY AUTOINCREMENT,
  code       TEXT NOT NULL UNIQUE,
  url        TEXT NOT NULL,
  created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
);

CREATE TABLE clicks (
  id         INTEGER PRIMARY KEY AUTOINCREMENT,
  link_id    INTEGER NOT NULL REFERENCES links(id),
  clicked_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
  country    TEXT,
  referrer   TEXT,
  browser    TEXT,
  os         TEXT,
  ip_hash    TEXT
);

The analytics queries (top countries, referrers, daily trend) all run against the clicks table with a JOIN on links. The most complex one is the daily trend:

db.prepare(`
  SELECT
    strftime('%Y-%m-%d', clicked_at) AS date,
    COUNT(*) AS clicks
  FROM clicks WHERE link_id = ?
  GROUP BY date ORDER BY date DESC LIMIT 30
`).all(link.id);

SQLite's strftime handles the date bucketing natively. No date library needed.

One gotcha with the database module: better-sqlite3 is synchronous and stateful, so the module caches the connection after the first call. Tests need isolated in-memory databases, not the cached production one. The fix is simple — never cache :memory: connections:

export function getDb(dbPath = DB_PATH) {
  if (dbPath !== ":memory:" && _db) return _db;
  const db = new Database(dbPath);
  // ... schema setup ...
  if (dbPath !== ":memory:") _db = db;
  return db;
}

Each test gets its own fresh in-memory database. Nothing bleeds between test cases.

Recording clicks without slowing down redirects

The redirect handler records the click and then sends the 301. better-sqlite3 is synchronous — it blocks the Node.js event loop during the write — but WAL mode means it's appending to the write-ahead log file rather than syncing the main database.

A single INSERT completes in microseconds in practice. The try/catch ensures analytics failures never break a redirect:

// src/routes/redirect.js
const CODE_RE = /^[A-Za-z0-9]{4,10}$/;

router.get("/:code", (req, res) => {
  const { code } = req.params;
  if (!CODE_RE.test(code)) return res.status(404).send("Link not found.");

  const link = getLinkByCode(db, code);
  if (!link) return res.status(404).send("Link not found.");

  try {
    const clickData = parseClickData(req);
    insertClick(db, link.id, clickData);
  } catch (err) {
    console.error("[redirect] analytics error:", err.message);
  }

  res.redirect(301, link.url);
});

The /:code route must be registered last in server.js. It matches any single-segment path (including /healthz), so any named routes registered after it will never be reached. Express matches in registration order, not specificity order.

One other decision worth noting: 301 vs 302. A 301 (permanent redirect) tells browsers and search engines to cache the destination — follow the link once and the browser may never hit the shortener again.

That means if you later change the destination URL, existing users won't see the change. For a personal shortener that's usually fine; use 302 if you need updatable destinations.

The POST /api/shorten endpoint is rate-limited to 20 requests per hour per IP. Redirect and stats endpoints are not limited — a popular link shouldn't throttle its own clicks.

What gets tracked and how

Three pieces of data come in on every click: the IP address, the User-Agent header, and the Referer header.

// src/lib/analytics.js
export function parseClickData(req) {
  const ip = getClientIp(req);
  const ua = req.headers["user-agent"] ?? "";
  const rawReferrer = req.headers["referer"] || req.headers["referrer"] || null;

  const salt = process.env.HASH_SALT;
  if (!salt) throw new Error("HASH_SALT is not set. Add it to your .env file.");
  const ipHash = crypto.createHash("sha256").update(ip + salt).digest("hex").slice(0, 16);
  const geo = ip ? geoip.lookup(ip) : null;
  const country = geo?.country ?? null;

  const parsed = UAParser(ua);
  const browser = parsed.browser?.name ?? null;
  const os = parsed.os?.name ?? null;

  let referrer = null;
  if (rawReferrer) {
    try {
      referrer = new URL(rawReferrer).hostname;
    } catch {
      referrer = rawReferrer.slice(0, 100);
    }
  }

  return { country, referrer, browser, os, ipHash };
}

The IP gets hashed before storage. Raw IPs are PII — storing them without justification is a compliance headache in most jurisdictions.

A SHA-256 hash of the IP (keyed with a secret from your environment) lets you count unique visitors reliably without keeping the raw address. geoip-lite does the country lookup from a bundled local database, so there's no external API call.

Referrers get normalized to hostname only. https://google.com/search?q=example becomes google.com. The full URL isn't useful for the dashboard, and storing full search queries you didn't ask for is the kind of thing that causes problems later.

The dashboard

The dashboard is a single HTML file that calls the API. No framework:

• GET /api/links — all shortened links with total click counts

• GET /api/stats/:code — full breakdown for one link (by country, referrer, browser, daily trend)

The daily trend renders as a bar chart built from raw div elements. Bar height is a percentage of the maximum day's count, so no chart library dependency.

Clicking "Stats" on any row loads that link's analytics inline below the table. The stat panel shows total clicks, unique visitors (by hashed IP), and three bar charts. Country shows the top 10 countries, referrer shows where traffic came from, browser breaks down the client distribution.

Trying it with curl

Make sure HASH_SALT is set in .env before running these — without it the server starts and redirects work, but clicks won't be recorded and stats will show zero.

Shorten a URL:

curl -X POST http://localhost:3000/api/shorten \
  -H "Content-Type: application/json" \
  -d '{"url": "https://zyvop.com/building-a-production-ai-agent-in-node-js-tool-calling-the-react-loop-and-error-handling-zzftm}'

{
  "code": "P8WbGd7",
  "shortUrl": "http://localhost:3000/P8WbGd7",
  "url": "https://zyvop.com/building-a-production-ai-agent-in-nodejs"
}

Follow the redirect:

curl -I http://localhost:3000/P8WbGd7

HTTP/1.1 301 Moved Permanently
Location: https://zyvop.com/building-a-production-ai-agent-in-nodejs

Pull the stats:

curl http://localhost:3000/api/stats/P8WbGd7

{
  "link": { "code": "P8WbGd7", "url": "https://zyvop.com/...", "created_at": "2026-07-14T06:00:00Z" },
  "totalClicks": 3,
  "uniqueVisitors": 1,
  "byCountry": [{ "country": "US", "clicks": 3 }],
  "byReferrer": [{ "referrer": "google.com", "clicks": 2 }, { "referrer": "Direct", "clicks": 1 }],
  "byBrowser": [{ "browser": "Chrome", "clicks": 3 }],
  "dailyTrend": [{ "date": "2026-07-14", "clicks": 3 }]
}

"Direct" in byReferrer is what COALESCE(referrer, 'Direct') returns for clicks that arrived without a Referer header — typed directly into the browser bar, opened from a native app, or clicked from an email client that strips referrers.

Invalid URL:

curl -s -w " [%{http_code}]" -X POST http://localhost:3000/api/shorten \
  -H "Content-Type: application/json" \
  -d '{"url": "ftp://notallowed"}'

{"error":"url must be a valid http or https URL."} [400]

Unknown code:

curl -I http://localhost:3000/ZZZZZZZ

HTTP/1.1 404 Not Found

Tests

npm test

# tests 24
# pass  24
# fail   0

24 tests across three suites. The shortcode suite verifies the alphabet, the collision retry logic, and that it throws correctly when retries are exhausted.

The analytics suite tests IP hashing, country lookup, referrer normalization, and user-agent parsing — including the privacy property that raw IPs don't appear in the stored hash. The database suite runs every query against an in-memory SQLite instance: inserts, retrieval, unique visitor counting, daily trend grouping, and the unique constraint on codes.

Taking it further

The setup above works for personal use or a small team. A few things to add before opening it to the public:

• Auth on the shorten endpoint — right now anyone who can reach the server can create links. A simple API key check is enough for most cases.

• Custom slugs — let users specify POST /api/shorten with an optional customCode field; validate it against the same alphabet and check for conflicts before inserting.

• Click buffering — if redirect volume gets high, the synchronous SQLite write on every click becomes the bottleneck. Buffer clicks in memory and flush in batches of 100 or every 5 seconds.

• Link expiry — add an expires_at column to links and check it in the redirect handler.

Get the code: https://github.com/zyvop27-cmyk/zyvop-blogs/tree/main/url-shortener

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

Social media data is the closest thing the internet has to real-time public opinion. It powers:

• Sentiment analysis — What do people think about your product, brand, or competitor right now?

• Trend detection — What topics are accelerating before they go mainstream?

• Academic research — Studying misinformation, political discourse, crisis communication

• Market intelligence — Tracking competitor mentions, industry conversations

• AI training datasets — Social text is gold for fine-tuning conversational models

The challenge in 2026: platforms have made scraping harder than ever. The official X API now starts at $100/month and the free tier is almost useless for research. Instagram aggressively blocks headless browsers. Reddit has rate-limited its API severely after the 2023 controversy.

This guide gives you working approaches for all three platforms — from DIY Python scrapers to managed tools — with honest assessments of what each method can and cannot do.

The 2026 Reality Check: What's Still Scrapable

Before writing a single line of code, here's the honest state of social media scraping in 2026:

Key principles:

• Never scrape private data, DMs, or anything behind a login you don't own

• Always respect robots.txt and Terms of Service for your use case

• For academic or commercial research, official APIs or licensed providers are the safer path

Part 1: Scraping X (Twitter) in 2026

The Landscape

X's public guest API was effectively removed in 2023. Almost every endpoint now requires a logged-in session and a CSRF token. The internal GraphQL API that underpins X's own web interface is the main DIY scraping target — but its endpoint identifiers change regularly.

Three working approaches exist in 2026:

Method 1: X API v2 (Official)

For authorised projects, the official API is always the right choice. The free tier gives 500k tweets/month read access:

pip install tweepy

import tweepy
import pandas as pd
from datetime import datetime, timezone, timedelta

# Get credentials at developer.twitter.com
client = tweepy.Client(bearer_token="YOUR_BEARER_TOKEN")

def search_recent_tweets(
    query: str,
    max_results: int = 100,
    days_back: int = 7
) -> pd.DataFrame:
    """
    Search tweets from the last 7 days using X API v2.

    Args:
        query: Search query. Supports operators like:
               - AND/OR: "python scraping OR web crawling"
               - Exact: '"machine learning" tutorial'
               - Exclude: "python -is:retweet"
               - Language: "python lang:en"
               - Has media: "python has:images"
    """
    start_time = datetime.now(timezone.utc) - timedelta(days=days_back)

    # Append filters to reduce noise
    full_query = f"{query} -is:retweet lang:en"

    tweets_data = []
    paginator = tweepy.Paginator(
        client.search_recent_tweets,
        query=full_query,
        tweet_fields=[
            "created_at", "public_metrics", "author_id",
            "lang", "context_annotations", "entities"
        ],
        user_fields=["name", "username", "public_metrics", "verified"],
        expansions=["author_id"],
        start_time=start_time,
        max_results=min(100, max_results),
    )

    users_map = {}

    for response in paginator:
        if not response.data:
            break

        # Build user lookup from includes
        if response.includes and "users" in response.includes:
            for user in response.includes["users"]:
                users_map[user.id] = user

        for tweet in response.data:
            user = users_map.get(tweet.author_id)
            metrics = tweet.public_metrics or {}

            tweets_data.append({
                "tweet_id":       str(tweet.id),
                "text":           tweet.text,
                "created_at":     tweet.created_at,
                "author_id":      str(tweet.author_id),
                "username":       user.username if user else None,
                "name":           user.name if user else None,
                "followers":      user.public_metrics.get("followers_count") if user else None,
                "retweets":       metrics.get("retweet_count", 0),
                "likes":          metrics.get("like_count", 0),
                "replies":        metrics.get("reply_count", 0),
                "quotes":         metrics.get("quote_count", 0),
                "impressions":    metrics.get("impression_count", 0),
                "url":            f"https://x.com/{user.username if user else 'i'}/status/{tweet.id}",
            })

        if len(tweets_data) >= max_results:
            break

    df = pd.DataFrame(tweets_data)
    print(f"Collected {len(df)} tweets for query: '{query}'")
    return df

# Example: track brand mentions
df = search_recent_tweets(
    query="python web scraping 2026",
    max_results=200,
    days_back=7
)
df.to_csv("tweets.csv", index=False)
print(df[["username", "text", "likes", "retweets"]].head(10))

Method 2: Playwright-Based X Scraper (No API Key)

For personal research without API access, Playwright can scrape public search results and profiles using a saved session — exactly as described for LinkedIn in Blog 02:

import asyncio
import random
import json
from playwright.async_api import async_playwright
from playwright_stealth import stealth_async

async def save_x_session():
    """Log in to X manually once and save cookies. Run this first."""
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120"
        )
        page = await context.new_page()
        await page.goto("https://x.com/login")

        print("Log in manually, then press Enter...")
        input()

        cookies = await context.cookies()
        with open("x_cookies.json", "w") as f:
            json.dump(cookies, f)
        print(f"Saved {len(cookies)} cookies.")
        await browser.close()

async def scrape_x_search(query: str, scroll_times: int = 10) -> list[dict]:
    """
    Scrape X search results using a saved session.
    Returns a list of tweet dicts.
    """
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            args=["--disable-blink-features=AutomationControlled"]
        )
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120",
            viewport={"width": 1280, "height": 900}
        )

        # Load saved session
        with open("x_cookies.json") as f:
            await context.add_cookies(json.load(f))

        page = await context.new_page()
        await stealth_async(page)

        # Navigate to search
        encoded = query.replace(" ", "%20")
        await page.goto(
            f"https://x.com/search?q={encoded}&src=typed_query&f=top",
            wait_until="domcontentloaded"
        )
        await asyncio.sleep(random.uniform(2, 4))

        tweets = []
        seen_ids = set()

        for scroll in range(scroll_times):
            # Extract tweet data from the current viewport
            tweet_articles = await page.query_selector_all("article[data-testid='tweet']")

            for article in tweet_articles:
                try:
                    # Get tweet text
                    text_el = await article.query_selector("[data-testid='tweetText']")
                    text = await text_el.inner_text() if text_el else None

                    # Username
                    user_el = await article.query_selector("[data-testid='User-Name'] a")
                    href = await user_el.get_attribute("href") if user_el else ""
                    username = href.strip("/").split("/")[-1] if href else None

                    # Display name
                    name_els = await article.query_selector_all(
                        "[data-testid='User-Name'] span"
                    )
                    display_name = None
                    for el in name_els:
                        txt = await el.inner_text()
                        if txt and not txt.startswith("@"):
                            display_name = txt
                            break

                    # Engagement stats
                    async def get_stat(testid):
                        el = await article.query_selector(f"[data-testid='{testid}']")
                        if el:
                            txt = await el.inner_text()
                            return txt.strip() or "0"
                        return "0"

                    likes    = await get_stat("like")
                    replies  = await get_stat("reply")
                    retweets = await get_stat("retweet")

                    # Time
                    time_el = await article.query_selector("time")
                    posted_at = await time_el.get_attribute("datetime") if time_el else None

                    # Unique ID to deduplicate
                    tweet_id = f"{username}_{posted_at}"
                    if tweet_id in seen_ids or not text:
                        continue
                    seen_ids.add(tweet_id)

                    tweets.append({
                        "username":     username,
                        "display_name": display_name,
                        "text":         text,
                        "likes":        likes,
                        "replies":      replies,
                        "retweets":     retweets,
                        "posted_at":    posted_at,
                    })
                except Exception:
                    continue

            # Scroll down for more tweets
            await page.evaluate("window.scrollBy(0, window.innerHeight * 1.5)")
            await asyncio.sleep(random.uniform(1.5, 3.0))
            print(f"  Scroll {scroll+1}/{scroll_times} — {len(tweets)} tweets collected")

        await browser.close()

    print(f"\nTotal unique tweets: {len(tweets)}")
    return tweets

# Run
tweets = asyncio.run(scrape_x_search("python scraping 2026", scroll_times=8))
df = pd.DataFrame(tweets)
df.to_csv("x_search_results.csv", index=False)

Part 2: Reddit Scraping with PRAW

Reddit is the most developer-friendly major social platform for data collection. Its official Python wrapper PRAW (Python Reddit API Wrapper) is free, well-documented, and gives access to posts, comments, user profiles, and subreddit data at no cost.

pip install praw pandas

Register a Reddit app at reddit.com/prefs/apps (takes 2 minutes, free) to get your client ID and secret.

Scraping subreddit posts

import praw
import pandas as pd
from datetime import datetime, timezone

reddit = praw.Reddit(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    user_agent="PythonResearchBot/1.0 by u/your_username",
)

def scrape_subreddit(
    subreddit_name: str,
    sort: str = "hot",         # "hot", "new", "top", "rising"
    time_filter: str = "week", # "hour", "day", "week", "month", "year", "all"
    limit: int = 500
) -> pd.DataFrame:
    """
    Scrape posts from a subreddit.

    Best for: topic research, trend monitoring, content analysis.
    """
    sub = reddit.subreddit(subreddit_name)

    # Choose sort method
    if sort == "hot":
        posts_gen = sub.hot(limit=limit)
    elif sort == "new":
        posts_gen = sub.new(limit=limit)
    elif sort == "top":
        posts_gen = sub.top(time_filter=time_filter, limit=limit)
    elif sort == "rising":
        posts_gen = sub.rising(limit=limit)
    else:
        posts_gen = sub.hot(limit=limit)

    records = []
    for post in posts_gen:
        records.append({
            "post_id":       post.id,
            "title":         post.title,
            "text":          post.selftext[:500] if post.selftext else None,
            "author":        str(post.author) if post.author else "[deleted]",
            "score":         post.score,
            "upvote_ratio":  post.upvote_ratio,
            "num_comments":  post.num_comments,
            "url":           post.url,
            "permalink":     f"https://reddit.com{post.permalink}",
            "flair":         post.link_flair_text,
            "is_self":       post.is_self,
            "created_utc":   datetime.fromtimestamp(post.created_utc, tz=timezone.utc).isoformat(),
            "subreddit":     subreddit_name,
        })

    df = pd.DataFrame(records)
    print(f"Scraped {len(df)} posts from r/{subreddit_name}")
    return df

# Example: research Python discussions
df = scrape_subreddit("learnpython", sort="top", time_filter="month", limit=500)
df.to_csv("reddit_learnpython.csv", index=False)
print(df[["title", "score", "num_comments"]].head(10))

Scraping post comments (deep dive)

def scrape_post_comments(
    post_url: str,
    max_comments: int = 200,
    include_replies: bool = False
) -> pd.DataFrame:
    """
    Scrape all comments from a Reddit post.
    Useful for sentiment analysis, topic deep-dives, building datasets.
    """
    submission = reddit.submission(url=post_url)

    # Replace "MoreComments" objects to get all comments
    submission.comments.replace_more(limit=0)

    all_comments = submission.comments.list() if include_replies else submission.comments

    records = []
    for comment in list(all_comments)[:max_comments]:
        if not hasattr(comment, "body"):
            continue
        records.append({
            "comment_id":    comment.id,
            "author":        str(comment.author) if comment.author else "[deleted]",
            "body":          comment.body,
            "score":         comment.score,
            "depth":         comment.depth,
            "created_utc":   datetime.fromtimestamp(
                                 comment.created_utc, tz=timezone.utc
                             ).isoformat(),
            "is_op":         comment.is_submitter,
            "awards":        comment.total_awards_received,
        })

    return pd.DataFrame(records)

# Example: deep dive on a specific post
comments_df = scrape_post_comments(
    "https://www.reddit.com/r/MachineLearning/comments/example/",
    max_comments=300
)
print(f"Top comment: {comments_df.sort_values('score', ascending=False).iloc[0]['body'][:200]}")

Multi-subreddit keyword monitoring

def monitor_keyword_across_subreddits(
    keyword: str,
    subreddits: list[str],
    limit_per_sub: int = 100
) -> pd.DataFrame:
    """
    Search for a keyword across multiple subreddits.
    Great for competitive intelligence and brand monitoring.
    """
    all_posts = []

    for sub_name in subreddits:
        print(f"Searching r/{sub_name} for '{keyword}'...")
        try:
            sub = reddit.subreddit(sub_name)
            for post in sub.search(keyword, sort="new", limit=limit_per_sub):
                all_posts.append({
                    "subreddit":    sub_name,
                    "title":        post.title,
                    "text":         post.selftext[:300] if post.selftext else None,
                    "author":       str(post.author) if post.author else "[deleted]",
                    "score":        post.score,
                    "comments":     post.num_comments,
                    "permalink":    f"https://reddit.com{post.permalink}",
                    "created_utc":  datetime.fromtimestamp(
                                        post.created_utc, tz=timezone.utc
                                    ).isoformat(),
                })
        except Exception as e:
            print(f"  Error on r/{sub_name}: {e}")

    df = pd.DataFrame(all_posts)

    # Summary by subreddit
    summary = df.groupby("subreddit").agg(
        posts=("title", "count"),
        avg_score=("score", "mean"),
        total_comments=("comments", "sum")
    ).sort_values("posts", ascending=False)

    print(f"\n── Results for '{keyword}' ──")
    print(summary.to_string())

    return df

# Track "Python scraping" across tech subreddits
df = monitor_keyword_across_subreddits(
    keyword="python scraping",
    subreddits=["Python", "learnpython", "webdev", "datascience", "MachineLearning"],
    limit_per_sub=50
)
df.to_csv("reddit_keyword_monitor.csv", index=False)

Part 3: Instagram Scraping (Public Profiles Only)

Instagram is the most aggressively protected major platform in 2026. The public guest API was removed years ago. Almost all data requires a logged-in session. The GraphQL API underlying Instagram's web interface changes identifiers constantly.

This section covers only public profile and hashtag data — the bare minimum available without login — and the Playwright approach for logged-in access to your own account's data.

Scraping public profile metadata

import httpx
import json
import asyncio

async def get_instagram_profile(username: str) -> dict | None:
    """
    Fetch public profile data for an Instagram username.
    Uses the semi-public shared data endpoint.
    Only works for public accounts.
    """
    url = f"https://www.instagram.com/{username}/?__a=1&__d=dis"

    headers = {
        "User-Agent": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
            "AppleWebKit/537.36 Chrome/120 Safari/537.36"
        ),
        "Accept": "*/*",
        "Referer": "https://www.instagram.com/",
        "X-IG-App-ID": "936619743392459",  # Instagram Web app ID
    }

    async with httpx.AsyncClient(headers=headers, follow_redirects=True) as client:
        try:
            r = await client.get(url, timeout=15)
            if r.status_code == 200:
                data = r.json()
                user = data.get("graphql", {}).get("user", {})
                return {
                    "username":       user.get("username"),
                    "full_name":      user.get("full_name"),
                    "biography":      user.get("biography"),
                    "followers":      user.get("edge_followed_by", {}).get("count"),
                    "following":      user.get("edge_follow", {}).get("count"),
                    "posts":          user.get("edge_owner_to_timeline_media", {}).get("count"),
                    "is_verified":    user.get("is_verified"),
                    "is_business":    user.get("is_business_account"),
                    "profile_url":    f"https://www.instagram.com/{username}/",
                }
        except Exception as e:
            print(f"Error fetching @{username}: {e}")
    return None

# Batch scrape public profiles
async def batch_profile_scrape(usernames: list[str]) -> pd.DataFrame:
    results = []
    for username in usernames:
        print(f"Fetching @{username}...")
        profile = await get_instagram_profile(username)
        if profile:
            results.append(profile)
        await asyncio.sleep(random.uniform(2, 5))
    return pd.DataFrame(results)

profiles_df = asyncio.run(batch_profile_scrape([
    "natgeo", "nasa", "python.learning"
]))
print(profiles_df[["username", "followers", "posts", "is_verified"]])

Playwright-based Instagram scraper (logged-in session)

For your own account's data or influencer research on public accounts:

import asyncio, json, random
from playwright.async_api import async_playwright
from playwright_stealth import stealth_async

async def save_instagram_session():
    """Log in manually and save session cookies. Run once."""
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) "
                       "AppleWebKit/605.1.15 Mobile/15E148 Safari/604.1",
            viewport={"width": 390, "height": 844},
            is_mobile=True,
        )
        page = await context.new_page()
        await page.goto("https://www.instagram.com/accounts/login/")
        print("Log in manually, then press Enter...")
        input()
        cookies = await context.cookies()
        with open("ig_cookies.json", "w") as f:
            json.dump(cookies, f)
        print("Session saved.")
        await browser.close()

async def scrape_instagram_posts(username: str, max_posts: int = 30) -> list[dict]:
    """
    Scrape recent posts from a public Instagram profile.
    Requires a saved session from save_instagram_session().
    """
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            args=["--no-sandbox", "--disable-blink-features=AutomationControlled"]
        )
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120",
            viewport={"width": 1280, "height": 900}
        )
        with open("ig_cookies.json") as f:
            await context.add_cookies(json.load(f))

        page = await context.new_page()
        await stealth_async(page)

        await page.goto(
            f"https://www.instagram.com/{username}/",
            wait_until="domcontentloaded"
        )
        await asyncio.sleep(random.uniform(2, 4))

        posts = []
        # Scroll to load posts
        for _ in range(max_posts // 12 + 1):
            post_links = await page.query_selector_all("article a[href*='/p/']")
            for link in post_links:
                href = await link.get_attribute("href")
                if href and href not in [p.get("href") for p in posts]:
                    posts.append({"href": href, "username": username})
            await page.evaluate("window.scrollBy(0, window.innerHeight)")
            await asyncio.sleep(random.uniform(1, 2.5))

        # Deduplicate and limit
        seen = set()
        unique_posts = []
        for p in posts:
            if p["href"] not in seen:
                seen.add(p["href"])
                unique_posts.append(p)
        posts = unique_posts[:max_posts]

        await browser.close()

    print(f"Found {len(posts)} posts for @{username}")
    return posts

Part 4: Sentiment Analysis on Collected Data

Once you've collected social media data, analysing sentiment turns raw text into actionable signals. The transformers library from HuggingFace provides excellent pre-trained sentiment models:

pip install transformers torch

from transformers import pipeline
import pandas as pd

# Load a pre-trained sentiment model (downloads ~67MB on first run)
# "cardiffnlp/twitter-roberta-base-sentiment-latest" is trained on tweets
sentiment_pipeline = pipeline(
    "sentiment-analysis",
    model="cardiffnlp/twitter-roberta-base-sentiment-latest",
    tokenizer="cardiffnlp/twitter-roberta-base-sentiment-latest",
    max_length=512,
    truncation=True,
)

LABEL_MAP = {
    "LABEL_0": "negative",
    "LABEL_1": "neutral",
    "LABEL_2": "positive",
}

def analyse_sentiment_batch(texts: list[str], batch_size: int = 32) -> list[dict]:
    """
    Run sentiment analysis on a list of social media texts.
    Returns list of {label, score} dicts.
    """
    results = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        # Clean texts — remove URLs and excessive whitespace
        cleaned = [
            " ".join(word for word in t.split() if not word.startswith("http"))[:512]
            for t in batch
        ]
        preds = sentiment_pipeline(cleaned)
        for pred in preds:
            results.append({
                "sentiment": LABEL_MAP.get(pred["label"], pred["label"]),
                "confidence": round(pred["score"], 3),
            })
        print(f"  Processed {min(i + batch_size, len(texts))}/{len(texts)}")

    return results

def sentiment_report(df: pd.DataFrame, text_col: str = "text") -> pd.DataFrame:
    """
    Enrich a DataFrame with sentiment scores and produce a summary report.
    """
    texts = df[text_col].fillna("").tolist()
    print(f"Analysing sentiment for {len(texts)} posts...")
    sentiments = analyse_sentiment_batch(texts)

    df["sentiment"]   = [s["sentiment"]  for s in sentiments]
    df["confidence"]  = [s["confidence"] for s in sentiments]

    # Summary
    summary = df["sentiment"].value_counts(normalize=True).mul(100).round(1)
    print("\n── Sentiment Distribution ──")
    for label, pct in summary.items():
        bar = "█" * int(pct / 2)
        print(f"  {label:10s}: {bar} {pct}%")

    return df

# Apply to Reddit data
reddit_df = pd.read_csv("reddit_learnpython.csv")
reddit_df = sentiment_report(reddit_df, text_col="title")
reddit_df.to_csv("reddit_with_sentiment.csv", index=False)

# Apply to X data
tweets_df = pd.read_csv("tweets.csv")
tweets_df = sentiment_report(tweets_df, text_col="text")
tweets_df.to_csv("tweets_with_sentiment.csv", index=False)

Part 5: Building a Brand Monitoring Dashboard

Combine all sources into a unified brand monitoring pipeline:

import asyncio
import pandas as pd
from datetime import datetime, timezone

async def run_brand_monitor(brand_name: str, competitor: str = None) -> dict:
    """
    Collect brand mentions from Reddit and X, analyse sentiment,
    and produce a unified brand health report.
    """
    report = {
        "brand":       brand_name,
        "run_at":      datetime.now(timezone.utc).isoformat(),
        "reddit":      {},
        "twitter":     {},
        "sentiment":   {},
    }

    # ── Reddit ────────────────────────────────────────────────
    print(f"\n[1/3] Scraping Reddit for '{brand_name}'...")
    reddit_df = monitor_keyword_across_subreddits(
        keyword=brand_name,
        subreddits=["technology", "Python", "webdev", "datascience", "programming"],
        limit_per_sub=50
    )
    reddit_df = sentiment_report(reddit_df, text_col="title")

    report["reddit"] = {
        "total_posts":     len(reddit_df),
        "avg_score":       round(reddit_df["score"].mean(), 1),
        "total_comments":  int(reddit_df["comments"].sum()),
        "top_post":        reddit_df.sort_values("score", ascending=False).iloc[0]["title"],
        "top_subreddit":   reddit_df["subreddit"].value_counts().index[0],
    }

    # ── Sentiment ─────────────────────────────────────────────
    all_sentiments = pd.concat([
        reddit_df[["sentiment", "confidence"]],
    ])
    sentiment_counts = all_sentiments["sentiment"].value_counts(normalize=True).mul(100)
    report["sentiment"] = {
        "positive_pct": round(sentiment_counts.get("positive", 0), 1),
        "neutral_pct":  round(sentiment_counts.get("neutral", 0), 1),
        "negative_pct": round(sentiment_counts.get("negative", 0), 1),
        "overall":      sentiment_counts.idxmax(),
    }

    # ── Print Report ──────────────────────────────────────────
    print(f"\n{'═'*50}")
    print(f"  BRAND MONITOR: {brand_name.upper()}")
    print(f"  Run at: {report['run_at']}")
    print(f"{'═'*50}")
    print(f"\n  Reddit mentions:   {report['reddit']['total_posts']}")
    print(f"  Avg post score:    {report['reddit']['avg_score']}")
    print(f"  Most active sub:   r/{report['reddit']['top_subreddit']}")
    print(f"\n  Sentiment:")
    print(f"    ✅ Positive:  {report['sentiment']['positive_pct']}%")
    print(f"    😐 Neutral:   {report['sentiment']['neutral_pct']}%")
    print(f"    ❌ Negative:  {report['sentiment']['negative_pct']}%")
    print(f"    Overall:      {report['sentiment']['overall'].upper()}")
    print(f"\n  Top post: \"{report['reddit']['top_post'][:80]}...\"")

    return report

# Run
report = asyncio.run(run_brand_monitor("python scraping"))

Rate Limits and Platform Rules Reference

FAQ

Q: snscrape is broken in 2026 — what should I use instead? snscrape's X scraper broke in 2023 when X removed the guest API and has not been reliably fixed. Use Tweepy with X API v2 for authorised projects. For DIY scraping, use the Playwright approach above.

Q: Can I scrape private Instagram posts? No — and you shouldn't. Scraping private account data without consent violates Instagram's Terms of Service, GDPR, and potentially criminal hacking laws in many jurisdictions. Only collect publicly available data.

Q: What's the best model for social media sentiment analysis in 2026?cardiffnlp/twitter-roberta-base-sentiment-latest is trained specifically on tweets and outperforms generic models. For Reddit, which is longer-form, distilbert-base-uncased-finetuned-sst-2-english works well. For multilingual content, use nlptown/bert-base-multilingual-uncased-sentiment.

Q: Reddit changed its API — is PRAW still free? Yes. PRAW's free tier (500 requests per 10 minutes) was not affected by the 2023 API changes. Only third-party apps accessing certain endpoints at very high volume were impacted. Standard research use via PRAW remains free.

Q: How do I handle deleted posts in my Reddit dataset? Posts with [deleted] author or [removed] body are common. Filter them before analysis: df = df[df['author'] != '[deleted]'] and df = df[df['text'] != '[removed]'].

Summary

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

This post builds the whole flow from scratch: PKCE code generation, the authorization redirect, the callback handler that validates everything before touching the database, token exchange, and refresh token handling. No Passport, no auth library. Just four endpoints and the Google API.

This is the third post in Zyvop's auth series. If you haven't read the 2FA implementation or the magic link post yet, the session module here is the same one from those — a JWT in an httpOnly cookie. The new parts are everything that happens before you issue that cookie.

Source: https://github.com/zyvop27-cmyk/zyvop-blogs/tree/master/oauth-google

Setup

git clone [YOUR_GITHUB_REPO_URL]
cd oauth-google
npm install
cp .env.example .env

Then in Google Cloud Console:

1. APIs & Services → Credentials → Create Credentials → OAuth 2.0 Client ID

2. Application type: Web application

3. Authorized Redirect URIs: http://localhost:3000/auth/google/callback

4. Copy the Client ID and Secret into .env

Generate a JWT_SECRET:

node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

npm start
# open http://localhost:3000

Why PKCE, and why does it matter for server-side apps

PKCE (Proof Key for Code Exchange) was designed for mobile and single-page apps that can't keep a client secret truly secret. For server-side apps you already have a secret, so adding PKCE is belt-and-suspenders. Google now requires it for all new OAuth clients regardless of app type, so it's no longer optional — it's just how OAuth with Google works in 2026.

The mechanism: before sending the user to Google, you generate a random code_verifier, hash it to produce a code_challenge, and send the challenge to Google.

When the user comes back with an authorization code, you exchange the code by also sending the original verifier. Google hashes it and checks it matches the challenge it stored. Anyone who intercepts the authorization code without the original verifier can't use it.

// src/lib/pkce.js
import { randomBytes, createHash } from "node:crypto";

export function generateVerifier() {
  // 32 bytes → 43-char URL-safe base64, no padding (RFC 7636)
  return randomBytes(32).toString("base64url");
}

export function deriveChallenge(verifier) {
  return createHash("sha256").update(verifier).digest("base64url");
}

export function verifyChallenge(verifier, expectedChallenge) {
  if (typeof verifier !== "string" || verifier.length < 43) return false;
  return deriveChallenge(verifier) === expectedChallenge;
}

base64url is the base64 alphabet with + → -, / → _, and all = padding stripped. Node has it built in since v14 — no library needed.

CSRF protection: the state parameter

The state parameter is how you prevent cross-site request forgery on the callback. You generate a random value, send it to Google, and Google sends it back in the callback. You verify it matches what you sent before touching anything else.

The state also carries the PKCE verifier. When the user lands on the callback, you need the verifier to complete the token exchange — but you can't put it in the URL, since that exposes it to server logs and referrer headers.

You could store it in a signed httpOnly cookie, but that means setting an additional cookie before the redirect and cleaning it up on callback. The state parameter is a neater single-point solution: one random value carries both the CSRF token and the verifier retrieval key.

// src/lib/state.js
import { randomBytes } from "node:crypto";

const _store = new Map();
const STATE_TTL_MS = 10 * 60 * 1000;

export function createState(verifier) {
  const state = randomBytes(24).toString("hex");
  _store.set(state, { verifier, expiresAt: Date.now() + STATE_TTL_MS });
  return state;
}

export function consumeState(state) {
  if (typeof state !== "string" || !state) return null;

  const entry = _store.get(state);
  _store.delete(state); // delete before returning — no replay window

  if (!entry) return null;
  if (Date.now() > entry.expiresAt) return null;
  return entry.verifier;
}

consumeState deletes the entry before checking it — not after. The order matters: if you check first and delete second, two simultaneous requests with the same state both pass the check before either deletes.

This way only one gets the verifier back; the other gets null.

The in-memory Map works fine for a single server process. For multiple instances behind a load balancer, move it to Redis with a 10-minute TTL.

Building the authorization URL

// src/lib/google.js
export function buildAuthUrl() {
  const verifier = generateVerifier();
  const challenge = deriveChallenge(verifier);
  const state = createState(verifier);

  const params = new URLSearchParams({
    client_id: process.env.GOOGLE_CLIENT_ID,
    redirect_uri: process.env.GOOGLE_REDIRECT_URI,
    response_type: "code",
    scope: "openid email profile",
    state,
    code_challenge: challenge,
    code_challenge_method: "S256",
    access_type: "offline",   // ask for a refresh token
    prompt: "consent",        // force consent screen so refresh token is always issued
  });

  return { url: `https://accounts.google.com/o/oauth2/v2/auth?${params}` };
}

access_type: "offline" tells Google you want a refresh token. Without it, you only get an access token that expires in an hour, and the user has to sign in again. prompt: "consent" forces the consent screen on every sign-in, which is the only reliable way to get a refresh token every time — Google silently skips it on returning users unless you force it.

The callback: three validations before anything else

When Google redirects the user back, the callback handler does three things before touching the database:

// src/routes/auth.js
router.get("/google/callback", async (req, res) => {
  const { code, state, error } = req.query;

  if (error) return res.redirect("/?error=access_denied");
  if (!code || !state) return res.redirect("/?error=invalid_callback");

  const verifier = consumeState(state);  // validates + consumes in one step
  if (!verifier) return res.redirect("/?error=invalid_state");

  try {
    const { accessToken, refreshToken } = await exchangeCode(code, verifier);
    const userInfo = await fetchUserInfo(accessToken);
    const user = upsertUser(db, userInfo, refreshToken);

    issueSession(res, { sub: user.google_sub, email: user.email,
                        name: user.name, picture: user.picture });
    res.redirect("/dashboard");
  } catch (err) {
    console.error("[oauth] callback failed:", err.message);
    res.redirect("/?error=auth_failed");
  }
});

Check error first — this is what Google sends when the user clicks "Cancel". Then check code and state are present.

consumeState in one call validates the state is real, not expired, and not already used, and returns the PKCE verifier. Only if all three pass do you make any network requests.

Token exchange and userinfo

The token exchange sends the authorization code plus the PKCE verifier to Google's token endpoint:

export async function exchangeCode(code, verifier, fetchFn = fetch) {
  const res = await fetchFn("https://oauth2.googleapis.com/token", {
    method: "POST",
    headers: { "Content-Type": "application/x-www-form-urlencoded" },
    body: new URLSearchParams({
      code,
      client_id: process.env.GOOGLE_CLIENT_ID,
      client_secret: process.env.GOOGLE_CLIENT_SECRET,
      redirect_uri: process.env.GOOGLE_REDIRECT_URI,
      grant_type: "authorization_code",
      code_verifier: verifier,
    }),
  });

  if (!res.ok) {
    const err = await res.text();
    throw new Error(`Token exchange failed (\({res.status}): \){err}`);
  }
  const data = await res.json();
  return {
    accessToken: data.access_token,
    refreshToken: data.refresh_token ?? null,
    idToken: data.id_token,
    expiresIn: data.expires_in,
  };
}

Google's response includes an id_token — a JWT containing the user's identity claims. You could decode and verify it locally using Google's published JWK keys, which avoids a second network request.

This post uses the /userinfo endpoint instead, because it always returns current data and sidesteps implementing RS256 signature verification. For high-throughput production code, verify the id_token locally.

The userinfo call is straightforward:

export async function fetchUserInfo(accessToken, fetchFn = fetch) {
  const res = await fetchFn("https://www.googleapis.com/oauth2/v3/userinfo", {
    headers: { Authorization: `Bearer ${accessToken}` },
  });
  if (!res.ok) throw new Error(`Userinfo fetch failed (${res.status})`);

  const data = await res.json();
  if (!data.email_verified) throw new Error("Google account email is not verified");

  return { sub: data.sub, email: data.email, name: data.name ?? null,
           picture: data.picture ?? null, emailVerified: data.email_verified };
}

The email_verified check matters. Google allows accounts with unverified email addresses. Silently accepting one would let someone claim ownership of an email they don't control.

Storing users and handling refresh tokens

Users are keyed by Google's sub (subject), not email. Email addresses can change; sub is permanent for a given Google account. The upsert uses COALESCE to preserve the existing refresh token when the new one is null:

db.prepare(`
  INSERT INTO users (google_sub, email, name, picture, refresh_token)
  VALUES (@sub, @email, @name, @picture, @refreshToken)
  ON CONFLICT (google_sub) DO UPDATE SET
    email         = excluded.email,
    name          = excluded.name,
    picture       = excluded.picture,
    refresh_token = COALESCE(excluded.refresh_token, users.refresh_token),
    last_login    = strftime('%Y-%m-%dT%H:%M:%SZ', 'now')
  RETURNING *
`).get({ sub, email, name, picture, refreshToken: refreshToken ?? null });

Google only sends a refresh token on first sign-in (or after the user revokes access). If you overwrite the stored refresh token with null on subsequent logins, you lose the ability to refresh the user's access token without asking them to sign in again. COALESCE keeps the old one when the new one is absent.

When the access token expires and you need a new one:

export async function refreshAccessToken(refreshToken, fetchFn = fetch) {
  const res = await fetchFn("https://oauth2.googleapis.com/token", {
    method: "POST",
    headers: { "Content-Type": "application/x-www-form-urlencoded" },
    body: new URLSearchParams({
      client_id: process.env.GOOGLE_CLIENT_ID,
      client_secret: process.env.GOOGLE_CLIENT_SECRET,
      refresh_token: refreshToken,
      grant_type: "refresh_token",
    }),
  });
  if (!res.ok) {
    const err = await res.text();
    throw new Error(`Token refresh failed (\({res.status}): \){err}`);
  }
  const data = await res.json();
  return { accessToken: data.access_token, expiresIn: data.expires_in };
}

A 400 response from this endpoint usually means the refresh token was revoked — the user removed your app from their Google account. Handle it by clearing the session and prompting a fresh sign-in.

The sign-in page

public/index.html is a single static page with a "Continue with Google" button that links to /auth/google. The interesting part is the error handling — when the OAuth flow fails for any reason, the server redirects back to /?error=<reason>. The page reads that query param and shows a human-readable message:

const ERRORS = {
  access_denied:    "You cancelled the sign-in. Try again when you're ready.",
  invalid_state:    "Something went wrong with the sign-in flow. Please try again.",
  invalid_callback: "The callback from Google was malformed. Please try again.",
  auth_failed:      "Sign-in failed. Please try again or contact support.",
  config:           "The server is misconfigured. Contact the site owner.",
  default:          "Something went wrong. Please try again."
};

const params = new URLSearchParams(location.search);
const errKey = params.get("error");
if (errKey) {
  const box = document.getElementById("error-box");
  box.textContent = ERRORS[errKey] ?? ERRORS.default;
  box.classList.add("visible");
}

Each error key maps to a specific failure point in the callback handler — access_denied when the user clicks Cancel, invalid_state when the state check fails, auth_failed when the token exchange or userinfo call throws. This matters: a generic "something went wrong" on a sign-in page sends users nowhere. Mapping errors to actionable messages costs five lines.

Testing without a Google account

Every external call takes an injectable fetchFn parameter. Tests pass a mock instead:

function mockFetch(status, body) {
  return async () => ({
    ok: status >= 200 && status < 300,
    status,
    json: async () => body,
    text: async () => JSON.stringify(body),
  });
}

test("exchangeCode throws on non-200 response", async () => {
  await assert.rejects(
    () => exchangeCode("code", "verifier", mockFetch(400, { error: "invalid_grant" })),
    /Token exchange failed/
  );
});

test("fetchUserInfo throws if email is not verified", async () => {
  await assert.rejects(
    () => fetchUserInfo("token", mockFetch(200, {
      sub: "123", email: "unverified@example.com", email_verified: false
    })),
    /email is not verified/
  );
});

The state store test exercises expiry by monkey-patching Date.now — no timers, no waiting:

test("expired state entries are not returned", () => {
  const realNow = Date.now;
  const state = createState("verifier");

  Date.now = () => realNow() + 11 * 60 * 1000; // 11 minutes later
  try {
    assert.equal(consumeState(state), null);
  } finally {
    Date.now = realNow;
  }
});

npm test
# tests 31 · pass 31 · fail 0

Trying it with curl

Once the server is running with real Google credentials:

Hit the sign-in page:

curl -I http://localhost:3000/auth/google

HTTP/1.1 302 Found
Location: https://accounts.google.com/o/oauth2/v2/auth?client_id=...&state=...&code_challenge=...

The OAuth flow goes through a browser — there's no way to automate it with curl end-to-end. To check the API after signing in, copy your session cookie from browser DevTools (Application → Cookies) and pass it directly:

curl -H "Cookie: session=<paste-your-jwt-here>" http://localhost:3000/api/me

{
  "sub": "1234567890",
  "email": "you@gmail.com",
  "name": "Your Name",
  "picture": "https://lh3.googleusercontent.com/..."
}

Refresh the access token (stored server-side):

curl -H "Cookie: session=<paste-your-jwt-here>" \
  -X POST http://localhost:3000/auth/refresh

{ "accessToken": "ya29.new-token", "expiresIn": 3599 }

No active session:

curl http://localhost:3000/api/me

{ "error": "Not authenticated." }

Before going to production

Move the state store to Redis — the in-memory Map disappears on restart and won't work across multiple server instances. The interface is the same, just backed by Redis keys with a 10-minute TTL.

Set NODE_ENV=production to enable the Secure flag on the session cookie, which browsers require over HTTPS. Add your production domain to Authorized Redirect URIs in Google Cloud Console — that list is an exact-match allowlist, and the callback will silently fail with redirect_uri_mismatch if your domain isn't on it.

If your threat model requires it, encrypt the refresh_token column at rest before writing to SQLite. A leaked database shouldn't hand an attacker long-lived access to user Google accounts.

Get the code: https://github.com/zyvop27-cmyk/zyvop-blogs/tree/master/oauth-google

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

At some point every scraper outgrows its original script form. Maybe you want to trigger scraping from a dashboard. Maybe a frontend team needs to query scraped data. Maybe you're building a product — a rank tracker, a price monitor, a lead generation tool — and users need to submit URLs and get results back.

The moment your scraper needs to serve multiple callers, run on a schedule, handle concurrent requests, or return results asynchronously, you need to wrap it in a proper API.

In this practical approach, we'll build a scraper wrapped in a web API powered by FastAPI to scrape and deliver data on demand — implementing FastAPI web services with asynchronous request handling, configuring real-time data scraping with caching and webhook support, and handling concurrent requests with rate limiting for scalable scraping API endpoints.

The stack we'll build is a proven pattern:

Client → FastAPI (HTTP layer) → Redis (task queue) → Celery Workers (scraping)
                                      ↓
                              PostgreSQL / SQLite (results storage)

FastAPI handles HTTP requests and returns immediate job IDs. Redis acts as the message broker. Celery workers run the actual scraping in the background. Results are stored in a database and fetched when the client polls.

This architecture means: your API never blocks, workers can scale horizontally, failed jobs retry automatically, and your scraping logic is completely decoupled from your HTTP layer.

Project Structure

scraping_api/
├── app/
│   ├── __init__.py
│   ├── main.py          # FastAPI application
│   ├── models.py        # SQLAlchemy models
│   ├── schemas.py       # Pydantic request/response schemas
│   ├── database.py      # DB connection
│   ├── tasks.py         # Celery scraping tasks
│   ├── scrapers/
│   │   ├── __init__.py
│   │   ├── generic.py   # Generic HTTP scraper
│   │   └── browser.py   # Playwright scraper
│   └── middleware/
│       ├── ratelimit.py  # Rate limiting
│       └── auth.py       # API key auth
├── worker.py            # Celery worker entrypoint
├── requirements.txt
├── docker-compose.yml
└── .env

Step 1: Install Dependencies

pip install fastapi uvicorn celery redis httpx \
            beautifulsoup4 sqlalchemy aiosqlite \
            python-dotenv pydantic slowapi

# requirements.txt
fastapi==0.115.0
uvicorn[standard]==0.30.0
celery[redis]==5.4.0
redis==5.0.1
httpx==0.27.0
beautifulsoup4==4.12.3
lxml==5.2.2
sqlalchemy==2.0.30
aiosqlite==0.20.0
python-dotenv==1.0.1
pydantic==2.7.0
slowapi==0.1.9
flower==2.0.1

Step 2: Database Models

# app/models.py
from sqlalchemy import Column, String, Float, Text, DateTime, Integer, Enum
from sqlalchemy.orm import DeclarativeBase
from datetime import datetime, timezone
import enum, uuid

class Base(DeclarativeBase):
    pass

class JobStatus(str, enum.Enum):
    PENDING   = "pending"
    RUNNING   = "running"
    SUCCESS   = "success"
    FAILED    = "failed"
    RETRYING  = "retrying"

class ScrapeJob(Base):
    __tablename__ = "scrape_jobs"

    id           = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
    url          = Column(String, nullable=False)
    scraper_type = Column(String, default="generic")   # "generic" | "browser"
    status       = Column(String, default=JobStatus.PENDING)
    created_at   = Column(DateTime, default=lambda: datetime.now(timezone.utc))
    started_at   = Column(DateTime, nullable=True)
    completed_at = Column(DateTime, nullable=True)
    api_key      = Column(String, nullable=True)
    error        = Column(Text, nullable=True)
    retry_count  = Column(Integer, default=0)

class ScrapeResult(Base):
    __tablename__ = "scrape_results"

    id         = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
    job_id     = Column(String, nullable=False, index=True)
    url        = Column(String, nullable=False)
    title      = Column(String, nullable=True)
    content    = Column(Text, nullable=True)
    html_len   = Column(Integer, nullable=True)
    status_code= Column(Integer, nullable=True)
    scraped_at = Column(DateTime, default=lambda: datetime.now(timezone.utc))
    metadata_  = Column(Text, nullable=True)   # JSON string for extra fields

Step 3: Pydantic Schemas

# app/schemas.py
from pydantic import BaseModel, HttpUrl, Field
from typing import Optional, List
from datetime import datetime
from app.models import JobStatus

class ScrapeRequest(BaseModel):
    url:          HttpUrl
    scraper_type: str    = Field("generic", pattern="^(generic|browser)$")
    css_selectors: Optional[dict] = None   # {"title": "h1", "price": ".price"}
    wait_for:     Optional[str]  = None    # CSS selector to wait for (browser mode)
    webhook_url:  Optional[HttpUrl] = None # POST results here when done

    model_config = {"json_schema_extra": {
        "example": {
            "url": "https://books.toscrape.com/",
            "scraper_type": "generic",
            "css_selectors": {"title": "h1", "books": ".product_pod h3 a"}
        }
    }}

class BulkScrapeRequest(BaseModel):
    urls:         List[HttpUrl]
    scraper_type: str = "generic"
    css_selectors: Optional[dict] = None

class JobResponse(BaseModel):
    job_id:   str
    status:   JobStatus
    url:      str
    created_at: datetime
    message:  str = "Job queued successfully"

class JobStatusResponse(BaseModel):
    job_id:      str
    status:      JobStatus
    url:         str
    created_at:  datetime
    started_at:  Optional[datetime]
    completed_at: Optional[datetime]
    retry_count:  int
    error:        Optional[str]

class ScrapeResultResponse(BaseModel):
    job_id:     str
    url:        str
    title:      Optional[str]
    content:    Optional[str]
    html_len:   Optional[int]
    status_code: Optional[int]
    scraped_at: datetime
    data:       Optional[dict]   # Parsed CSS selector results

Step 4: The Scraping Logic

# app/scrapers/generic.py
import httpx
import asyncio
import random
from bs4 import BeautifulSoup
from curl_cffi.requests import AsyncSession
from datetime import datetime, timezone
from typing import Optional

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,*/*;q=0.8",
}

async def scrape_generic(
    url: str,
    css_selectors: Optional[dict] = None,
    max_retries: int = 3
) -> dict:
    """
    Generic HTTP scraper using curl_cffi for TLS impersonation.
    Returns title, content, status_code, and any custom selector results.
    """
    for attempt in range(max_retries):
        try:
            async with AsyncSession(impersonate="chrome120") as session:
                r = await session.get(url, headers=HEADERS, timeout=20)

            result = {
                "url":         url,
                "status_code": r.status_code,
                "html_len":    len(r.text),
                "scraped_at":  datetime.now(timezone.utc).isoformat(),
                "data":        {},
            }

            if r.status_code != 200:
                result["error"] = f"HTTP {r.status_code}"
                return result

            soup = BeautifulSoup(r.text, "lxml")

            # Extract title
            title_el = soup.find("title")
            result["title"] = title_el.get_text(strip=True) if title_el else None

            # Extract body text (clean)
            for tag in soup(["script", "style", "nav", "footer", "header"]):
                tag.decompose()
            result["content"] = soup.get_text(separator=" ", strip=True)[:5000]

            # Apply custom CSS selectors if provided
            if css_selectors:
                for key, selector in css_selectors.items():
                    elements = soup.select(selector)
                    result["data"][key] = [
                        el.get_text(strip=True) for el in elements
                    ] if len(elements) > 1 else (
                        elements[0].get_text(strip=True) if elements else None
                    )

            return result

        except Exception as e:
            wait = 2 ** attempt + random.random()
            if attempt < max_retries - 1:
                await asyncio.sleep(wait)
            else:
                return {
                    "url":        url,
                    "status_code": None,
                    "error":      str(e),
                    "scraped_at": datetime.now(timezone.utc).isoformat(),
                    "data":       {},
                }

# app/scrapers/browser.py
import asyncio
import random
from playwright.async_api import async_playwright
from playwright_stealth import stealth_async
from datetime import datetime, timezone
from typing import Optional

async def scrape_browser(
    url: str,
    css_selectors: Optional[dict] = None,
    wait_for: Optional[str] = None,
) -> dict:
    """
    Playwright-based browser scraper for JavaScript-rendered pages.
    """
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            args=["--no-sandbox", "--disable-blink-features=AutomationControlled"]
        )
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120",
            viewport={"width": 1280, "height": 800}
        )

        # Block unnecessary resources
        await context.route(
            "**/*.{png,jpg,gif,woff,woff2}",
            lambda route: route.abort()
        )

        page = await context.new_page()
        await stealth_async(page)

        result = {
            "url": url,
            "scraped_at": datetime.now(timezone.utc).isoformat(),
            "data": {},
        }

        try:
            await page.goto(url, wait_until="domcontentloaded", timeout=30000)

            if wait_for:
                await page.wait_for_selector(wait_for, timeout=10000)
            else:
                await asyncio.sleep(random.uniform(1.5, 3.0))

            result["title"]       = await page.title()
            result["status_code"] = 200
            result["html_len"]    = len(await page.content())

            # Apply CSS selectors
            if css_selectors:
                for key, selector in css_selectors.items():
                    elements = await page.query_selector_all(selector)
                    texts = [await el.inner_text() for el in elements]
                    result["data"][key] = texts if len(texts) > 1 else (
                        texts[0] if texts else None
                    )

        except Exception as e:
            result["error"]       = str(e)
            result["status_code"] = None

        finally:
            await browser.close()

    return result

Step 5: Celery Tasks

# app/tasks.py
import asyncio
import json
from celery import Celery
from datetime import datetime, timezone
from app.scrapers.generic import scrape_generic
from app.scrapers.browser import scrape_browser
import httpx

# Celery app — connects to Redis as broker and result backend
celery_app = Celery(
    "scraping_api",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1",
)

celery_app.conf.update(
    task_serializer       = "json",
    result_serializer     = "json",
    accept_content        = ["json"],
    result_expires        = 3600,        # Results expire after 1 hour
    task_acks_late        = True,        # Only ack after task completes
    worker_prefetch_multiplier = 1,      # One task per worker at a time
    task_track_started    = True,
    task_soft_time_limit  = 60,          # Warn after 60s
    task_time_limit       = 90,          # Kill after 90s
    # Route browser tasks to a dedicated queue
    task_routes           = {
        "app.tasks.scrape_url_task":         {"queue": "generic"},
        "app.tasks.scrape_browser_task":     {"queue": "browser"},
        "app.tasks.scrape_bulk_task":        {"queue": "bulk"},
    },
)

def run_async(coro):
    """Run an async coroutine from a sync Celery task."""
    loop = asyncio.new_event_loop()
    try:
        return loop.run_until_complete(coro)
    finally:
        loop.close()

@celery_app.task(
    bind=True,
    name="app.tasks.scrape_url_task",
    max_retries=3,
    default_retry_delay=30,
    autoretry_for=(Exception,),
    retry_backoff=True,
)
def scrape_url_task(
    self,
    job_id: str,
    url: str,
    css_selectors: dict = None,
    webhook_url: str = None,
):
    """
    Celery task: scrape a single URL using the generic HTTP scraper.
    Automatically retries up to 3 times on failure.
    """
    from app.database import SessionLocal
    from app.models import ScrapeJob, ScrapeResult, JobStatus

    db = SessionLocal()
    try:
        # Mark job as running
        job = db.query(ScrapeJob).filter(ScrapeJob.id == job_id).first()
        if job:
            job.status     = JobStatus.RUNNING
            job.started_at = datetime.now(timezone.utc)
            db.commit()

        # Run the scraper
        result = run_async(scrape_generic(url, css_selectors))

        # Save result
        scrape_result = ScrapeResult(
            job_id      = job_id,
            url         = url,
            title       = result.get("title"),
            content     = result.get("content"),
            html_len    = result.get("html_len"),
            status_code = result.get("status_code"),
            metadata_   = json.dumps(result.get("data", {})),
        )
        db.add(scrape_result)

        # Update job status
        if job:
            job.status       = JobStatus.SUCCESS if result.get("status_code") == 200 \
                               else JobStatus.FAILED
            job.completed_at = datetime.now(timezone.utc)
            job.error        = result.get("error")
        db.commit()

        # Fire webhook if configured
        if webhook_url:
            run_async(_fire_webhook(webhook_url, job_id, result))

        return {"job_id": job_id, "status": "success"}

    except Exception as exc:
        if job:
            job.status      = JobStatus.RETRYING
            job.retry_count = (job.retry_count or 0) + 1
            db.commit()
        db.close()
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

    finally:
        db.close()

@celery_app.task(name="app.tasks.scrape_browser_task", max_retries=2)
def scrape_browser_task(
    job_id: str,
    url: str,
    css_selectors: dict = None,
    wait_for: str = None,
    webhook_url: str = None,
):
    """Celery task: scrape with Playwright browser."""
    from app.database import SessionLocal
    from app.models import ScrapeJob, ScrapeResult, JobStatus

    db = SessionLocal()
    try:
        job = db.query(ScrapeJob).filter(ScrapeJob.id == job_id).first()
        if job:
            job.status     = JobStatus.RUNNING
            job.started_at = datetime.now(timezone.utc)
            db.commit()

        result = run_async(scrape_browser(url, css_selectors, wait_for))

        scrape_result = ScrapeResult(
            job_id      = job_id,
            url         = url,
            title       = result.get("title"),
            content     = None,
            html_len    = result.get("html_len"),
            status_code = result.get("status_code"),
            metadata_   = json.dumps(result.get("data", {})),
        )
        db.add(scrape_result)

        if job:
            job.status       = JobStatus.SUCCESS
            job.completed_at = datetime.now(timezone.utc)
        db.commit()

        if webhook_url:
            run_async(_fire_webhook(webhook_url, job_id, result))

        return {"job_id": job_id, "status": "success"}
    finally:
        db.close()

async def _fire_webhook(webhook_url: str, job_id: str, data: dict):
    """POST results to a webhook URL when a job completes."""
    async with httpx.AsyncClient() as client:
        try:
            await client.post(
                webhook_url,
                json={"job_id": job_id, "result": data},
                timeout=10
            )
        except Exception as e:
            print(f"Webhook delivery failed for {job_id}: {e}")

Step 6: The FastAPI Application

# app/main.py
import json
from typing import List, Optional
from fastapi import FastAPI, Depends, HTTPException, Header, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.util import get_remote_address
from slowapi.errors import RateLimitExceeded
from sqlalchemy.orm import Session

from app.database import get_db, engine
from app.models import Base, ScrapeJob, ScrapeResult, JobStatus
from app.schemas import (
    ScrapeRequest, BulkScrapeRequest,
    JobResponse, JobStatusResponse, ScrapeResultResponse
)
from app.tasks import scrape_url_task, scrape_browser_task

# Create all tables
Base.metadata.create_all(bind=engine)

# Rate limiter — 60 requests per minute per IP
limiter = Limiter(key_func=get_remote_address)
app     = FastAPI(
    title="Python Scraping API",
    description="Production-grade web scraping as a service",
    version="1.0.0",
)
app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

# Simple API key auth
VALID_API_KEYS = {"dev-key-123", "prod-key-456"}   # In prod: load from DB or env

def verify_api_key(x_api_key: str = Header(...)):
    if x_api_key not in VALID_API_KEYS:
        raise HTTPException(status_code=401, detail="Invalid API key")
    return x_api_key

# ── Endpoints ─────────────────────────────────────────────────

@app.get("/health")
async def health():
    return {"status": "ok", "version": "1.0.0"}

@app.post("/scrape", response_model=JobResponse, status_code=202)
@limiter.limit("30/minute")
async def submit_scrape_job(
    request: Request,
    payload: ScrapeRequest,
    db: Session = Depends(get_db),
    api_key: str = Depends(verify_api_key),
):
    """
    Submit a URL for scraping. Returns a job_id immediately.
    Poll /jobs/{job_id} for status, /jobs/{job_id}/result for data.
    """
    job = ScrapeJob(
        url          = str(payload.url),
        scraper_type = payload.scraper_type,
        api_key      = api_key,
    )
    db.add(job)
    db.commit()
    db.refresh(job)

    # Dispatch to appropriate Celery queue
    task_kwargs = {
        "job_id":       job.id,
        "url":          str(payload.url),
        "css_selectors": payload.css_selectors,
        "webhook_url":  str(payload.webhook_url) if payload.webhook_url else None,
    }

    if payload.scraper_type == "browser":
        task_kwargs["wait_for"] = payload.wait_for
        scrape_browser_task.apply_async(
            kwargs=task_kwargs,
            queue="browser",
            task_id=job.id
        )
    else:
        scrape_url_task.apply_async(
            kwargs=task_kwargs,
            queue="generic",
            task_id=job.id
        )

    return JobResponse(
        job_id     = job.id,
        status     = JobStatus.PENDING,
        url        = str(payload.url),
        created_at = job.created_at,
    )

@app.post("/scrape/bulk", status_code=202)
@limiter.limit("5/minute")
async def submit_bulk_scrape(
    request: Request,
    payload: BulkScrapeRequest,
    db: Session = Depends(get_db),
    api_key: str = Depends(verify_api_key),
):
    """Submit multiple URLs at once. Returns list of job_ids."""
    if len(payload.urls) > 50:
        raise HTTPException(400, "Max 50 URLs per bulk request")

    job_ids = []
    for url in payload.urls:
        job = ScrapeJob(url=str(url), scraper_type=payload.scraper_type, api_key=api_key)
        db.add(job)
        db.commit()
        db.refresh(job)

        scrape_url_task.apply_async(
            kwargs={"job_id": job.id, "url": str(url), "css_selectors": payload.css_selectors},
            queue="bulk",
        )
        job_ids.append(job.id)

    return {"submitted": len(job_ids), "job_ids": job_ids}

@app.get("/jobs/{job_id}", response_model=JobStatusResponse)
async def get_job_status(job_id: str, db: Session = Depends(get_db)):
    """Check the status of a scrape job."""
    job = db.query(ScrapeJob).filter(ScrapeJob.id == job_id).first()
    if not job:
        raise HTTPException(404, f"Job {job_id} not found")
    return job

@app.get("/jobs/{job_id}/result", response_model=ScrapeResultResponse)
async def get_job_result(job_id: str, db: Session = Depends(get_db)):
    """Fetch the scraped data for a completed job."""
    job = db.query(ScrapeJob).filter(ScrapeJob.id == job_id).first()
    if not job:
        raise HTTPException(404, f"Job {job_id} not found")

    if job.status not in (JobStatus.SUCCESS, JobStatus.FAILED):
        raise HTTPException(202, f"Job is still {job.status}")

    result = db.query(ScrapeResult).filter(ScrapeResult.job_id == job_id).first()
    if not result:
        raise HTTPException(404, "No result found for this job")

    return ScrapeResultResponse(
        job_id      = job_id,
        url         = result.url,
        title       = result.title,
        content     = result.content,
        html_len    = result.html_len,
        status_code = result.status_code,
        scraped_at  = result.scraped_at,
        data        = json.loads(result.metadata_ or "{}"),
    )

@app.get("/jobs")
async def list_jobs(
    status:  Optional[str] = None,
    limit:   int           = 20,
    offset:  int           = 0,
    db:      Session       = Depends(get_db),
):
    """List all scrape jobs with optional status filter."""
    query = db.query(ScrapeJob)
    if status:
        query = query.filter(ScrapeJob.status == status)
    total = query.count()
    jobs  = query.order_by(ScrapeJob.created_at.desc()).offset(offset).limit(limit).all()
    return {"total": total, "jobs": jobs}

@app.delete("/jobs/{job_id}")
async def cancel_job(job_id: str, db: Session = Depends(get_db)):
    """Cancel a pending job."""
    job = db.query(ScrapeJob).filter(ScrapeJob.id == job_id).first()
    if not job:
        raise HTTPException(404, f"Job {job_id} not found")
    if job.status != JobStatus.PENDING:
        raise HTTPException(400, f"Cannot cancel a {job.status} job")

    from app.tasks import celery_app as celery
    celery.control.revoke(job_id, terminate=True)
    job.status = JobStatus.FAILED
    job.error  = "Cancelled by user"
    db.commit()
    return {"message": f"Job {job_id} cancelled"}

Step 7: Database Setup

# app/database.py
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
import os

DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///./scraping_api.db")

engine = create_engine(
    DATABASE_URL,
    connect_args={"check_same_thread": False} if "sqlite" in DATABASE_URL else {}
)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)

def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

Step 8: Docker Compose

# docker-compose.yml
version: "3.9"

services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  api:
    build: .
    command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=sqlite:///./scraping_api.db
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      redis:
        condition: service_healthy
    volumes:
      - .:/app

  worker_generic:
    build: .
    command: celery -A app.tasks.celery_app worker -Q generic -c 8 --loglevel=info
    environment:
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      - redis
    volumes:
      - .:/app

  worker_browser:
    build: .
    command: celery -A app.tasks.celery_app worker -Q browser -c 2 --loglevel=info
    environment:
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      - redis
    volumes:
      - .:/app

  worker_bulk:
    build: .
    command: celery -A app.tasks.celery_app worker -Q bulk -c 4 --loglevel=info
    environment:
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      - redis
    volumes:
      - .:/app

  flower:
    build: .
    command: celery -A app.tasks.celery_app flower --port=5555
    ports:
      - "5555:5555"
    depends_on:
      - redis

Step 9: Using the API

Start everything:

docker-compose up --build

Submit a scrape job:

# Submit a scrape job
curl -X POST http://localhost:8000/scrape \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dev-key-123" \
  -d '{
    "url": "https://books.toscrape.com/",
    "scraper_type": "generic",
    "css_selectors": {
      "title": "h1",
      "books": ".product_pod h3 a"
    }
  }'

# Response:
# {"job_id": "abc-123", "status": "pending", "url": "https://..."}

# Check status
curl http://localhost:8000/jobs/abc-123

# Get results when complete
curl http://localhost:8000/jobs/abc-123/result

Python client:

import httpx
import time

BASE = "http://localhost:8000"
KEY  = "dev-key-123"

def scrape(url: str, selectors: dict = None) -> dict:
    """Submit a scrape job and poll until complete."""
    r = httpx.post(
        f"{BASE}/scrape",
        json={"url": url, "scraper_type": "generic", "css_selectors": selectors},
        headers={"X-API-Key": KEY}
    )
    job_id = r.json()["job_id"]

    # Poll for completion
    for _ in range(30):
        status_r = httpx.get(f"{BASE}/jobs/{job_id}")
        status   = status_r.json()["status"]
        if status in ("success", "failed"):
            break
        print(f"  Status: {status}...")
        time.sleep(2)

    # Fetch result
    result_r = httpx.get(f"{BASE}/jobs/{job_id}/result")
    return result_r.json()

result = scrape(
    "https://books.toscrape.com/",
    selectors={"books": ".product_pod h3 a"}
)
print(f"Title: {result['title']}")
print(f"Books found: {len(result['data'].get('books', []))}")

Bulk scrape:

# Submit 20 URLs at once
urls = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 21)]

r = httpx.post(
    f"{BASE}/scrape/bulk",
    json={"urls": urls, "scraper_type": "generic"},
    headers={"X-API-Key": KEY}
)
print(f"Submitted {r.json()['submitted']} jobs")
print(f"Job IDs: {r.json()['job_ids'][:3]}...")

Step 10: Monitoring with Flower

Flower dashboard provides real-time monitoring of your task queue system. Open http://localhost:5555 to see:

• Live task execution graph

• Worker status and load

• Task success/failure rates

• Queue depths per queue

• Task history and retry counts

For production alerting, add Prometheus metrics:

# Add to requirements.txt: celery-prometheus-exporter
# Then scrape http://worker:9808/metrics with Prometheus

Production Checklist

• Replace SQLite with PostgreSQL (DATABASE_URL=postgresql://...)

• Store API keys in database with rate limits per key

• Add request logging with structlog

• Set task_time_limit and task_soft_time_limit per task type

• Configure dead-letter queue for permanently failed tasks

• Add Redis Sentinel or Cluster for HA Redis

• Run behind Nginx with TLS termination

• Set CELERY_WORKER_CONCURRENCY per queue based on task type

• Add Sentry for error tracking

• Expose /metrics endpoint for Prometheus

FAQ

Q: Why use Celery instead of FastAPI BackgroundTasks? FastAPI's BackgroundTasks runs in the same process as the web server — if the server restarts, running tasks are lost. Celery with Redis persists tasks in the broker, supports retries, scales across multiple machines, and provides monitoring via Flower. For any serious scraping API, Celery is the right choice.

Q: How many concurrent scrapers can I run? Generic HTTP scrapers (curl_cffi/httpx) are cheap — a single worker can handle 8–16 concurrent tasks. Browser scrapers (Playwright) are expensive — limit to 2 per worker and 1 worker per CPU core. Scale horizontally by adding more worker containers.

Q: How do I handle proxy rotation in the task layer? Pass a proxy parameter through the task kwargs, or implement proxy selection inside the scraper using a ProxyPool class that round-robins or selects randomly on each task invocation.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

I wanted something different.

I wanted to look at what developers are talking about right now — the Hacker News thread with 700 points, the Dev.to post with 150 reactions, the search-query patterns appearing around a topic, the GitHub repository that has quickly attracted thousands of stars — and instantly turn those signals into blog content briefs I can execute on immediately.

So I built it. Here's how.

The Problem: Fresh Websites Have Zero Historical Data

When you launch a new developer blog or technical publication, Google Search Console (GSC) is practically empty. You might have a handful of impressions for your brand name and a few long-tail queries that nobody else searches for.

The typical SEO workflow looks like this:

This is a cold-start loop. You're waiting for data that depends on traffic you don't have yet.

The question became: What if I could bypass the cold-start entirely by pulling real-time demand signals from the places developers actually hang out?

The Architecture: Four Live Streams, One Unified Radar

The system I built aggregates live or recent data from four primary developer platforms into a single, queryable radar feed. The radar uses observed source signals rather than inventing engagement metrics. The separate keyword-discovery engine can also use model-generated estimates, but those are treated as estimates rather than measured search data.

The key architectural decision: live streams are cached in-memory, but content decisions are persisted to PostgreSQL. The radar shows you what's hot right now; once you decide to write about something, it becomes a permanent, trackable content opportunity with a full AI-generated content brief.

Data Source #1: Hacker News Front Page

Hacker News is arguably the most concentrated source of developer attention on the internet. A front-page post can attract substantial developer attention within hours.

I use the Algolia HN Search API to pull the current front page:

private async fetchHackerNewsTrends(): Promise<RealTrendingItem[]> {
  const res = await fetch(
    'https://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=25',
    { headers: { 'User-Agent': 'ZyVopSeoRadar/1.0' } },
  );

  const data = await res.json();

  return data.hits
    .filter((h) => h.title && h.points > 20)
    .map((h) => {
      // Extract real tags from HN Algolia _tags field
      // (filters out generic 'story', 'front_page', 'author_xyz')
      const rawTags = Array.isArray(h._tags)
        ? h._tags.filter((t) => !t.startsWith('author_') && t !== 'story' && t !== 'front_page')
        : [];
      const tags = rawTags.length > 0 ? rawTags : ['Tech', 'Engineering'];

      return {
        title: h.title,
        source: 'HACKER_NEWS',
        url: h.url || `https://news.ycombinator.com/item?id=${h.objectID}`,
        summary: `\({h.points} points · \){h.num_comments || 0} comments`,
        tags,
        score: h.points,
        commentsCount: h.num_comments,
        publishedAt: h.created_at, // Real ISO timestamp from HN
        suggestedAngle: `Write an engineering deep dive addressing "${h.title}"...`,
      };
    });
}

The points > 20 filter is simply a practical noise threshold for the feed. A post with 700 points and 400 comments is a strong signal of visible community attention, although it does not by itself prove search demand.

What this gives you: Real-time awareness of what the developer community is debating right now. Topics like "Htmx 4.0", "GLM-5.3 is now open-weight", or "GUIs should be fully keyboard-driven" — these are the conversations you can join with a well-timed deep-dive article.

Data Source #2: Dev.to Trending Articles

Dev.to's top=7 feed surfaces popular articles from the previous 7 days. Unlike Hacker News (which skews toward links and discussions), Dev.to content is written by developers for developers — tutorials, opinion pieces, and how-to guides.

private async fetchDevToTrends(): Promise<RealTrendingItem[]> {
  const res = await fetch(
    'https://dev.to/api/articles?per_page=30&top=7',
    { headers: { 'User-Agent': 'ZyVopSeoRadar/1.0' } },
  );

  const data = await res.json();

  return data.map((d) => ({
    title: d.title,
    source: 'DEV_TO',
    url: d.url,
    summary: d.description,
    tags: d.tag_list,
    score: d.positive_reactions_count,
    commentsCount: d.comments_count,
    suggestedAngle: `Write a comprehensive, code-rich guide on "${d.title}"...`,
  }));
}

What this gives you: Validated content formats. If "10 Git Commands You'll Wish You Knew Earlier" has 178 reactions, you know that listicle-format developer productivity content resonates. You can write a more comprehensive version, targeting the same search intent with deeper technical substance.

Data Source #3: Google Autocomplete (Search-Intent Signal)

This is the most directly actionable signal for query discovery. Google Autocomplete reflects real searches, but its predictions can also depend on language, location, trending interest, and past searches. It is useful for discovering query patterns and emerging search intent, but it is not a direct search-volume metric. Google documents these factors here.

private async fetchGoogleSearchTrends(): Promise<RealTrendingItem[]> {
  const seedTerms = [
    'Next.js 15', 'AI agents', 'FastAPI', 'TypeScript',
    'PostgreSQL', 'Docker', 'DeepSeek R1', 'Rust programming',
  ];

  const results: RealTrendingItem[] = [];

  for (const term of seedTerms) {
    const url = `https://suggestqueries.google.com/complete/search` +
      `?client=chrome&q=${encodeURIComponent(term)}`;
    const res = await fetch(url, {
      headers: { 'User-Agent': 'Mozilla/5.0' },
    });
    const data = await res.json();

    // data[1] contains the autocomplete suggestions
    for (const query of data[1].slice(1, 4)) {
      results.push({
        title: query.trim(),
        source: 'GOOGLE_SEARCH',
        url: `https://www.google.com/search?q=${encodeURIComponent(query)}`,
        summary: `Google Autocomplete prediction for "${term}" — a search-intent signal, not a volume metric`,
        score: 0,            // No engagement score — autocomplete is a signal, not a post
        publishedAt: null,   // Query signal — there is no "published date"
      });
    }
  }

  return results;
}

An important design decision here: Google Autocomplete items have no engagement score or verified search-volume number. Unlike a Hacker News post (which has real points) or a Dev.to article (which has real reactions), an autocomplete suggestion is a search-intent signal. I deliberately set score: 0 and publishedAt: null instead of faking numbers — the frontend handles these cases with distinct labels ("Live Search Demand" and "Live Query") so the user knows exactly what kind of signal they're looking at.

For example, querying "Next.js 15" might return:

• next.js 15 server actions

• next.js vs react

• next.js latest version

• next.js tutorial

These are query patterns Google is surfacing around the seed topic. Writing a comprehensive article targeting a phrase such as "next.js 15 server actions" may align with emerging search intent, but the autocomplete result itself is not proof of current search volume.

What this gives you: Query discovery based on current autocomplete signals, without pretending those signals are equivalent to measured keyword volume.

Data Source #4: GitHub Breakout Repositories

New open-source projects with large star counts shortly after creation can signal emerging developer interest in a technology, pattern, or tool. The current query surfaces recently created repositories and sorts them by current star count; it does not yet measure star-growth velocity over time.

private async fetchGithubTrending(): Promise<RealTrendingItem[]> {
  const oneMonthAgo = new Date(Date.now() - 30 * 86400000)
    .toISOString().split('T')[0];

  const url = `https://api.github.com/search/repositories` +
    `?q=created:>${oneMonthAgo}&sort=stars&order=desc&per_page=12`;

  const res = await fetch(url, {
    headers: {
      'User-Agent': 'ZyVopSeoRadar/1.0',
      Accept: 'application/vnd.github.v3+json',
    },
  });

  const data = await res.json();

  return data.items.map((repo) => ({
    title: `\({repo.full_name}: \){repo.description}`,
    source: 'GITHUB',
    url: repo.html_url,
    summary: `⭐ \({repo.stargazers_count.toLocaleString()} stars · \){repo.language}`,
    tags: [repo.language, 'Open Source', 'GitHub Trending'],
    score: Math.min(Math.round(repo.stargazers_count / 10), 1000),
    suggestedAngle: `Write a technical review or getting-started walkthrough...`,
  }));
}

What this gives you: An early publishing opportunity. A getting-started guide for a newly popular repository can give you a head start while search results are still relatively sparse, but it does not guarantee rankings.

The Backend: NestJS Service with In-Memory Caching

All four data sources are fetched concurrently using Promise.allSettled(). This means a timeout or failure in one source doesn't block the others — you always get results from whichever sources respond.

The 5-minute in-memory cache (Map<string, CachedRadar>) prevents hammering external APIs on every dashboard refresh while keeping the data fresh enough for near-real-time editorial decision-making.

The Conversion Pipeline: From Trending Topic to Content Brief

The most powerful part isn't the radar itself — it's what happens when you click "Write Blog on This".

When you click the button, the system:

1. Creates a permanent SeoOpportunity record in PostgreSQL with type: CONTENT_GAP, priority: HIGH, and actionType: CREATE_PAGE.

2. Triggers the SeoAiService which sends the topic to Groq's openai/gpt-oss-120b model to generate a full content brief.

3. Returns an AI Content Brief containing:

   • Recommended SEO title and H1

   • Complete article structure with section headings

   • Key questions the article should answer

   • Required original value (code samples, benchmarks, diagrams)

   • Internal linking suggestions

   • Suggested call-to-action

The opportunity then appears in the Opportunities tab, where it flows through the full lifecycle: DETECTED → ANALYZED → RECOMMENDED → APPROVED → IMPLEMENTED → DEPLOYED → MEASURED.

Once the blog is published and deployed, the system can pull its Google Search Console performance data and compare impressions, clicks, CTR, and position deltas over 7-day and 30-day windows.

The GraphQL API Layer

The entire system is exposed through two GraphQL operations:

Query: Live Trending Radar

query GetLiveTrendingRadar($category: String) {
  getLiveTrendingRadar(category: $category) {
    lastUpdated
    items {
      title
      source
      url
      summary
      tags
      score
      commentsCount
      publishedAt
      suggestedAngle
    }
  }
}

Mutation: Convert to Content Opportunity

mutation ConvertTrendingToOpportunity(
  $title: String!
  $tags: [String]
  $source: String
  $url: String
) {
  convertTrendingToOpportunity(
    title: $title
    tags: $tags
    source: $source
    url: $url
  ) {
    id
    targetQuery
    opportunityScore
    priority
    status
    contentRecommendation {
      contentBrief {
        recommendedTitle
        recommendedH1
        suggestedStructure
        questionsToAnswer
        requiredOriginalValue
      }
    }
  }
}

Both operations are protected by GqlAuthGuard and RolesGuard with @Roles('ADMIN'), ensuring only authenticated administrators can access the radar and create content opportunities.

The Frontend: A Real-Time Dashboard Tab

The frontend is a Next.js 16 React component that provides:

• Source filtering — Toggle between All Sources, Hacker News, Dev.to, Google Search, and GitHub

• Category filtering — Filter by AI & LLMs, React & Next.js, Python & Backend, DevOps, Rust, or PostgreSQL

• Search — Full-text search across titles, tags, and summaries

• Source badges — Color-coded indicators showing where each trending item originated, with real engagement metrics (HN points, Dev.to reactions, GitHub star counts)

• Relative timestamps — Each card shows when the item was published ("23h ago", "2d ago") or "Live Query" for Google Autocomplete items that have no publish date

• Dynamic engagement labels — Instead of a static "High Viral Potential" on every card, each item gets a label based on its actual score:

  • Viral Buzz (≥500 points/reactions) — red

  • High Engagement (≥200) — orange

  • Rising Interest (≥50) — green

  • Emerging Topic (<50) — gray

  • Live Search Demand (Google Autocomplete) — blue

• Suggested writing angles — Content angle suggestions tailored to each trending topic

• One-click conversion — The "Write Blog on This" button that triggers the full content brief pipeline

The timeAgo() helper formats real ISO timestamps from each API into human-readable relative dates:

function timeAgo(dateStr: string | null | undefined): string {
  if (!dateStr) return 'Live Query';
  const seconds = Math.floor((Date.now() - new Date(dateStr).getTime()) / 1000);
  if (seconds < 60) return 'just now';
  if (seconds < 3600) return `${Math.floor(seconds / 60)}m ago`;
  if (seconds < 86400) return `${Math.floor(seconds / 3600)}h ago`;
  if (seconds < 604800) return `${Math.floor(seconds / 86400)}d ago`;
  return new Date(dateStr).toLocaleDateString('en-US', { month: 'short', day: 'numeric' });
}

The component uses Next.js Server Actions for the data flow. When the radar loads, fetchLiveTrendingRadarAction is called, which hits the backend GraphQL endpoint. When a user clicks "Write Blog on This", convertTrendingToOpportunityAction persists the opportunity and triggers AI analysis — all through Server Actions without any client-side GraphQL setup.

The Hybrid Keyword Discovery Engine

Alongside the real-time radar, the system includes a keyword discovery engine that works without requiring Google Ads API credentials — a common blocker for independent developers and small publications.

If Google Ads credentials are configured and working, the system uses the official Keyword Planner API. If they're not available, it falls back to:

1. Google Autocomplete — Fetching suggested queries for each seed keyword

2. Groq AI — Expanding those queries and classifying intent. Any volume, CPC, competition, or trend figures produced by the LLM are estimates, not measured Google keyword metrics.

The fallback produces keyword ideas that can be actionable for research, but any LLM-generated search-volume figures are estimates and should not be treated as measured demand. Each keyword is saved to the seo_keywords table, and the opportunity engine can create CONTENT_GAP opportunities for keywords that meet the configured threshold when a trustworthy avgMonthlySearches value is available.

The Data Model

The content opportunity lifecycle is tracked across three main entities:

The SeoOpportunity entity stores the full AI analysis and content recommendation as JSONB columns, making them queryable and flexible without requiring schema migrations for every new field the AI model returns.

Lessons Learned

1. Promise.allSettled() Over Promise.all()

External APIs are unreliable. GitHub might rate-limit you. Hacker News might be slow. Using Promise.allSettled() means you always get results from the sources that responded, instead of failing entirely because one source timed out.

2. In-Memory Cache for Live Feeds, PostgreSQL for Decisions

The radar feed changes every few minutes. Caching it for 5 minutes in memory prevents excessive API calls while keeping the data fresh. But once a user decides "I want to write about this topic," that decision is persisted permanently. The radar is ephemeral; content strategy is persistent.

3. Math.round() Before PostgreSQL Integer Columns

A subtle bug taught me this the hard way. TypeORM @Column({ type: 'int' }) columns in PostgreSQL strictly reject floating-point numbers. If you calculate impressions * 1.5 = 109.5 and try to save it, PostgreSQL throws invalid input syntax for type integer: "109.5". Always wrap computed values with Math.round() before saving to integer columns.

4. Google Ads Test Accounts Return Bucketed Ranges

Depending on account access and API response, Google Ads Keyword Planner metrics may be returned as ranges rather than exact values; use the values provided by the API rather than inventing precision. For a developer blog, the AI + Google Suggest fallback can still be useful for discovery when measured Keyword Planner data is unavailable.

5. Cannibalization Detection Needs High Thresholds

On a new site, nearly every query appears on multiple pages because you have so few pages. With a threshold of impressions >= 1, every single query was flagged as a "cannibalization alert." Raising the threshold to impressions >= 200 && urls.length > 1 eliminated the false positives entirely.

6. Never Fake Engagement Metrics

My first version hardcoded score: 350 on every Google Autocomplete result to make them appear alongside Hacker News posts (which have real point counts of 200-800). This was misleading — it made autocomplete suggestions look like they had engagement they didn't have, and it broke the sorting logic by inflating Google items above genuinely viral HN posts.

The fix was simple: set score: 0 for autocomplete items and handle the display differently in the frontend. Google Autocomplete signals are valuable for a completely different reason (search intent) than HN posts (community validation). They shouldn't compete on the same axis. The frontend now shows "Live Search Demand" in blue for these items instead of trying to rank them by a fake score.

The Tech Stack

What's Next

The radar currently streams and displays. The next evolution is to add:

• Automated daily digests — A BullMQ job that runs the radar every morning and emails the top 10 trending topics with pre-generated content briefs

• Trend velocity scoring — Tracking how fast a topic is accelerating across sources (a topic trending on HN, Dev.to, AND Google simultaneously gets a higher signal score)

• Competitor content gap analysis — Cross-referencing trending topics against what competitors have already published to find uncovered angles

• Auto-draft generation — Using the AI content brief to generate a full first draft that goes straight into the CMS as a review-ready post

Try It Yourself

The entire system is built with publicly available APIs. You don't need any paid API keys to get started:

• Hacker News: https://hn.algolia.com/api/v1/search?tags=front_page

• Dev.to: https://dev.to/api/articles?per_page=30&top=7

• Google Autocomplete: https://suggestqueries.google.com/complete/search?client=chrome&q=YOUR_TERM

• GitHub: https://api.github.com/search/repositories?q=created:>DATE&sort=stars

The value isn't in the individual data sources — it's in aggregating them into a single decision interface and connecting that interface to a content pipeline that turns attention signals into published articles.

Stop relying entirely on yesterday's SEO data. Use live community and search signals to find what is emerging now.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

What OpenAI shipped

OpenAI released GPT-6 Sol and GPT-6 Luna on September 22, 2026, slotting them beneath the flagship GPT-6 Astra model it shipped earlier in the month, according to OpenAI's announcement and confirmed by TechCrunch. Sol is built for complex, multi-step work such as coding and analysis, while Luna targets high-volume, low-latency tasks like summarization and extraction, a split both outlets describe as consistent with the first Sol and Luna generation released earlier this year.

Pricing dropped by half

Source: OpenAI's pricing table

OpenAI describes this as permanent pricing rather than an introductory offer. VentureBeat reports that an OpenAI spokesperson confirmed the GPT-6 Sol and Luna rates are not promotional, unlike the GPT-5.6 pricing they replace. That distinction matters if you are forecasting a production pipeline around these numbers, since a promotional rate can be pulled without warning while a standard rate is a more stable planning assumption.

Benchmarks: real gains, mixed picture

On AutomationBench 1.0.6, a Zapier-run test of end-to-end business workflows across 47 tools, GPT-6 Sol at maximum reasoning effort scores 33.2% at $0.27 per task, according to OpenAI's results table. For comparison, OpenAI reports GPT-6 Astra at low effort scoring 30.3% at 3.9 times Sol's cost, and Claude Opus 5 at max effort scoring 26.9% at 11.1 times Sol's cost per task. GPT-6 Luna, run at high effort, improves 5.4 percentage points over its predecessor while costing 58% less per task.

OpenAI flags its own comparison as incomplete: it notes that Claude Fable 5.1 scored 31.4% on the same chart, but that figure understates Fable's real cost because it excludes the Opus 5 fallback calls OpenAI says occurred on roughly 40% of Fable's tasks. That kind of caveat is worth reading closely any time a vendor benchmarks its own model against a competitor's.

On DeepSWE v1.1, a long-horizon software-engineering benchmark, GPT-6 Sol at max effort scores 68.8%, within 1.1 points of Claude Fable 5's best reported score of 69.9%, at roughly 80% lower cost per task, per OpenAI. GPT-6 Luna at max effort scores 66.6%, comparable to Claude Opus 5 and Fable 5 at medium effort, while costing 93% less per task than Opus 5 and 96% less than Fable 5, OpenAI reports.

On OSWorld 2.0 offline, which grades agents on long, realistic computer-use workflows, GPT-6 Sol at xhigh effort scores 60.5% versus Claude Opus 5's 60.3% at medium effort, at about 80% lower cost, according to OpenAI. GPT-6 Luna at max effort beats GPT-5.6 Sol at medium effort while costing one-tenth as much. Astra still leads on computer use overall.

On Agents' Last Exam, spanning 55 professional sub-industries, GPT-6 Sol at max effort scores 56.4%, above Claude Opus 5's best score in that evaluation, at 60% lower cost per task, OpenAI says.

The New Stack notes that, unlike past releases, OpenAI did not send it a full benchmark packet ahead of this launch. The comparisons also went stale fast: per TechCrunch, Anthropic released Claude Opus 5.5 just 90 minutes before OpenAI's announcement, cutting its own price to \(4 and \)20 per million tokens, which The New Stack calculates still leaves it twice as expensive as GPT-6 Sol, though no outlet has run the two head-to-head yet.

Prompt caching is the part worth wiring into your agent

For anyone running an agent loop that reuses a long system prompt or tool schema, the caching changes may matter more than the headline token prices. OpenAI says it raised default cache hit rates and added explicit cache breakpoints, letting you control exactly where a cached prefix ends, per OpenAI's announcement. Cached input-token reads now get a 90% discount, and you can change reasoning effort or toggle tool availability mid-conversation without invalidating the cache, useful if a job queue dynamically adjusts effort per task type.

At the ecosystem level, GitHub told OpenAI that these caching changes cut the share of prompt tokens requiring fresh processing by more than half across billions of requests over the past several months, per OpenAI's announcement. If your workload sends a large, mostly static system prompt on every call, checking your cache hit rate in the Prompt Caching Dashboard is likely worth more than agonizing over Sol versus Luna.

Where the new models are live

GPT-6 Sol and Luna are available now in ChatGPT Work and Codex for Plus, Pro, Business, Enterprise, and Edu users, with Luna also reaching Free and Go users in the desktop app; neither model is in the base ChatGPT Chat surface yet, according to OpenAI. In the API they are addressable as gpt-6-sol and gpt-6-luna.

GitHub Copilot added both models this week alongside Astra: Sol is available on Copilot Pro+, Max, Business, and Enterprise plans, and Luna extends down to the base Pro plan, per GitHub's changelog. Rollout in ChatGPT is gradual, so do not be surprised if a model takes a day to appear in your account.

Alignment numbers worth reading before you widen an agent's autonomy

If you are building anything that acts with less supervision, the alignment testing deserves a closer look than the launch framing suggests. On OpenAI's internal coding-deception test, Sol's rate of misleading claims fell to 1.3% from 10.4%, and Luna's fell to 2.8% from 9.5%, per VentureBeat. Given a deliberately broken tool, Sol's failure-to-disclose rate fell to 5.4% from 77.8%; Luna's fell to 30.2% from 78.3%.

On a reviewer-bypass test, where an agent tries to route around an automated safety check in Codex, Sol made no attempts to bypass the reviewer; Luna attempted one in 0.3% of runs, down from 3.5%, with no successful bypasses, per VentureBeat.

The less reassuring number: when told to respect an explicit warning such as an access-denied message, Sol still tried to work around it in 64.4% of runs, barely down from 68.2% for GPT-5.6 Sol. Luna improved to 42.4%, though The New Stack and VentureBeat report slightly different prior rates for Luna here (78.5% and 76.5%, respectively).

On a simulated forum seeded with unauthorized instructions, The New Stack reports Sol acted on them in 11.3% of runs where it found the forum, down from 51.9%; Luna and Astra took no unauthorized action, though Luna also found the forum less often.

OpenAI is explicit that these are deliberately adversarial, low-stakes scenarios run without the production-level safeguards that sit in front of ChatGPT and the API. Treat the numbers as a relative signal across model generations rather than an estimate of real-world failure rates, and keep human review on anything an agent does that is hard to undo.

Which model to reach for

The practical move for most teams is to benchmark Sol and Luna against your own tasks rather than trust any single published score, since OpenAI's own AutomationBench footnote shows how much a methodology choice can shift a headline number. Reach for Luna when the job is high-volume and simple: classification, extraction, short summaries.

Reach for Sol when the task benefits from iteration and multi-step reasoning but does not need Astra's ceiling. Reserve Astra for work where getting it wrong is expensive, and re-run your own evaluation whenever a competing model ships, since the field moved twice in one day this week.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

This post is an attempt at the actual explanation. Not a feature grid. Not a paragraph that says "Vercel is great for beginners" and calls it a day. Three platforms, three genuinely different execution models, real pricing numbers, and a straight answer on when each one makes sense.

The Three Machines

Before the pricing matters, the architecture matters. These platforms do not just charge differently -- they run your code differently, and that distinction drives every other trade-off.

Vercel is a serverless platform. When a request hits a Vercel function, the platform boots a sandboxed execution environment, runs your code, and tears it down. Nothing persists between requests unless you explicitly store state externally. Vercel built this model to handle arbitrary traffic spikes without any configuration from you, and it works -- but the model shapes what your application can and cannot do. Long-running processes, persistent in-memory caches, WebSocket connections, background workers: none of these are natural fits for a request-scoped function runtime.

A VPS is a rented Linux box. You get root. You install what you want, run what you want, and keep it running as long as you want. The process that handles your HTTP request at 3 AM is the same process that was running at noon. Nothing spins up on demand. Nothing tears down after the response. This model gives you total control and zero automatic scaling. If traffic doubles at midnight and your box cannot handle it, the box falls over.

Fly.io occupies a third position that most comparisons skip. It runs your workloads inside Firecracker microVMs -- hardware-virtualized containers that boot in sub-second timeframes, much faster than a traditional VM but with stronger isolation than a plain Docker container. You push a Docker image, specify regions, and Fly handles scheduling, TLS termination, health checks, and routing. The process that handles your request is a long-lived container, not an ephemeral function. You get persistent state, WebSockets, and background workers. You do not get bare root access to the underlying host, and you do not manage nginx.

The distinction matters because it determines what breaks first as you scale, what you can build, and where the surprise costs hide.

Cold Starts, and What Vercel Did About Them

The canonical knock on Vercel serverless functions has always been cold starts. When a function has not been invoked recently, the platform boots a fresh execution environment before it can respond. Historically, users reported delays of 2 to 3 seconds on cold paths, and that number climbed further if your function opened a database connection, because serverless cannot share persistent connection pools between invocations.

In April 2025, Vercel shipped a new execution model called Fluid Compute and made it the default for all projects created after that date. The core change is that warm function instances can now serve multiple concurrent requests rather than being locked to a single request lifecycle. According to Vercel's own knowledge base, this means most production requests land on an already-running instance, making a true cold start the exception rather than the rule for active workloads.

This is a meaningful improvement, but it has limits. Fluid Compute helps on active, traffic-receiving functions. If you have dozens of routes that each see sporadic traffic, cold starts still happen on the quiet ones. Regional cold starts also persist: a function warmed in us-east-1 is cold in ap-southeast-1.

Fly.io's situation is different because the underlying model is different. A Fly Machine is a container that stays running. Requests hit a warm process every time, with no boot sequence required. Fly does support scale-to-zero as an opt-in feature -- machines can suspend when idle and resume on the next request -- with a reported wake time of 200 to 500ms for lightweight apps. That is still faster than a traditional serverless cold start, and it is a deliberate trade-off you make, not a platform default you accept.

A VPS has no cold starts in any meaningful sense. Your process is either running or it is not. The latency floor depends on your server's geographic proximity to your users, your Node (or whatever runtime) startup time on deploys, and how well you have configured process management. nginx with PM2 or systemd handling a Node process adds no cold start overhead. What a VPS does not give you is automatic global distribution -- your server is in one datacenter.

Real Cost at Three Traffic Tiers

Abstract pricing pages are not useful. Let's ground this in a specific scenario: a Next.js application serving 500,000 requests per month, with 200GB of bandwidth, one developer on the team, and a Postgres database.

Tier 1: Low traffic, single developer

Vercel's free Hobby plan covers this traffic comfortably on paper, but commercial use is prohibited on Hobby -- a restriction most developers miss until they re-read the terms. If the app is commercial in any sense, you are on Pro: \(20 per seat per month. Your 200GB bandwidth sits well inside the 1TB monthly allowance, and Fluid Compute keeps you inside the included function execution budget for a 500k-request app. Total: \)20/month.

A Hetzner CX22 -- 2 vCPU, 4GB RAM, 40GB NVMe, 20TB bandwidth -- costs \(4.39 per month. Add self-managed Postgres on the same box and you stay at \)4.39. Add Supabase Pro for a hands-off database and you are at $29/month. Either way, the figure assumes you know nginx, Let's Encrypt, deployment pipelines, and Postgres backup management. That knowledge has a real cost even if it does not appear on an invoice.

A Fly.io minimal setup with one shared-cpu-1x machine and a Fly Postgres instance runs \(13 to \)20 per month all-in. You push Docker images with flyctl deploy. TLS, routing, and health checks are included. The gap between Vercel Pro and Fly.io at this tier is roughly zero to seven dollars, depending on how you count the database. Fly's advantage here is not price -- it is that you are building on a model that does not punish you later.

Tier 2: Mid-scale SaaS, 3 million requests per month, 600GB bandwidth, 3 developers

On Vercel Pro at three seats, the base cost is \(60/month. Your 600GB bandwidth sits inside the 1TB cap. The function execution story depends on what your routes do -- an SSR-heavy app with database queries on most routes burns execution time faster than a mostly-static site. A real-world example from pagepro.co showed one service consuming 1,276 GB-hours against a 1,000 GB-hour Pro inclusion, generating \)160 in overages. Realistically: \(60 to \)150/month.

A Hetzner CX32 at \(7.88/month handles this traffic on a 4 vCPU, 8GB RAM box without breaking a sweat. The 20TB bandwidth inclusion means you will never pay an egress overage on Hetzner for this workload. Add a managed database -- whether on the same box or offloaded to Neon or PlanetScale -- and call it \)30 to $50/month depending on your setup. The catch is operational overhead: three developers means you need a shared deployment pipeline, runbooks, and someone to handle the server when it misbehaves at 2 AM.

Fly.io at this tier with two production machines, a Postgres cluster, and multi-region deployment: roughly \(40 to \)70/month. You get the operational simplicity of a PaaS -- flyctl deploy, automatic rolling updates, built-in metrics via VictoriaMetrics -- without the function execution meter running.

Tier 3: High-traffic application, 20 million requests per month, 3TB bandwidth, 5 developers

This is where Vercel's pricing model shows its architecture. Your bandwidth overage at 3TB on Pro is \(0.15/GB for the 2TB above the 1TB inclusion: \)300 in bandwidth alone. Five seats at \(100. Edge request overages depending on your geographic distribution. A realistic Vercel bill at this tier: \)500 to $900/month, with genuine uncertainty on the execution overage side if your workload is compute-heavy.

A Hetzner CX42 at \(19/month gives you 8 vCPU, 16GB RAM, and 20TB of bandwidth -- enough for this workload on a single box with a reasonable horizontal scaling plan. A proper production setup with load balancing, a Postgres primary/replica, and automated deployments: \)80 to \(120/month. The gap between Vercel and VPS at this tier is \)400 to $700 per month. That is real money, and it represents the point where the VPS operational overhead genuinely pays for itself.

Fly.io at high traffic with multiple regions: \(150 to \)250/month, depending on how many regions and what instance sizes you need. More expensive than raw VPS, meaningfully cheaper than Vercel, and with global distribution built in that a single-region VPS cannot match.

The Ops Tax

Every article about VPS hosting undersells what managing one actually costs. The financial comparison above uses Hetzner's invoice, not the real cost -- which includes developer time spent on things that do not ship features.

Here is a realistic list of what a production VPS setup requires that Vercel and Fly.io handle for you: nginx configuration and maintenance, Let's Encrypt SSL with automatic renewal, systemd or PM2 for process management, firewall rules (UFW, fail2ban for brute-force protection), OS patching and security updates, database backup automation, deployment pipelines that handle zero-downtime restarts, log aggregation, and uptime monitoring with alerting.

None of this is intellectually hard if you already know it. But the learning curve, if you are coming from PaaS tools, runs 20 to 100 hours depending on your starting point. And the ongoing cost -- monthly patches, the occasional nginx debugging session, the SSL renewal that did not auto-renew -- adds up to meaningful opportunity cost even for experienced teams.

Tools like Coolify and Dokploy have changed this calculus somewhat. Coolify gives you a Heroku-like deployment UI on top of your own server, with 280+ one-click app templates, and it is free. But you are still running it on infrastructure you own, and when Coolify breaks, it is your problem. The abstraction is better than raw nginx; it is not the same as Vercel's zero-configuration deploy-and-forget model.

Fly.io's ops surface is narrow by design. flyctl deploy handles rolling updates. TLS is automatic. You do not write nginx configs. You do not manage process supervisors. The operational surface you own is your Dockerfile and your fly.toml configuration -- both of which are portable and readable. The trade-off is that Fly's Postgres is self-managed in the sense that you handle backups and failover configuration yourself; it is a Postgres instance running inside a Fly Machine, not a hands-off managed database service like RDS.

What Vercel Actually Locks You Into

Vendor lock-in on Vercel is not a conspiracy. It is the natural result of building a framework and a platform simultaneously, with each one optimized for the other. Understanding exactly what you cannot take with you is more useful than a vague warning about it.

next/image routes image optimization requests through Vercel's proprietary processing pipeline. The component API is standard React -- <Image src="..." /> -- but the underlying optimization (format conversion to AVIF/WebP, responsive variants, lazy loading) runs on Vercel's infrastructure. Moving to Cloudflare Pages requires either Cloudflare Images ($5/month) or a custom loader. Moving to a VPS means rebuilding with Sharp or an external service like imgix. The code change is small; the infrastructure decision is not.

ISR (Incremental Static Regeneration) assumes persistent filesystem access for cache storage. Vercel handles this transparently. On a VPS or another platform, ISR requires external cache storage -- Redis or S3 -- and configuration that Vercel abstracts away. The framework assumes infrastructure that Vercel provides, which is fine on Vercel and works on other platforms, but the work is real.

Vercel Middleware runs on a proprietary edge runtime. The API is similar to the Web Platform's Request/Response model, but it is not identical. Migrating middleware to Cloudflare Workers requires a port -- not a copy-paste -- and Netlify's Deno-based edge functions are different again. Apps heavily using Next.js 15.1+ features on Vercel can break on non-Vercel deployments for reasons that are non-obvious until you try.

Preview deployments are not a lock-in risk, but they are worth mentioning as a genuine Vercel advantage. Every pull request gets a unique URL with your full application running against it. Teams that have built review workflows around preview URLs face a real workflow change when migrating -- Fly.io can replicate this with some scripting, a VPS requires significant CI/CD work to match it.

The honest summary: if your Next.js app uses next/image, ISR, and Edge Middleware heavily, and it is a year into production, migrating off Vercel is a one to three month engineering project. Not impossible. Not trivial.

When to Move, and Where to Go

The signals that it is time to leave Vercel are more predictable than most teams realize.

Stay on Vercel when: you are shipping a Next.js app and your team is under 5 developers. You do not have a DevOps function. Your bandwidth stays under 1TB per month. You use ISR, next/image, and Edge Middleware, and you have not built around workarounds for their limitations. The $20/seat cost is lower than the opportunity cost of managing infrastructure for a small team with feature velocity as the top priority.

Move to Fly.io when: your application has backend requirements that do not fit the serverless model -- WebSockets, long-running jobs, persistent in-memory state, or a custom runtime. When you need global distribution but cannot afford the Vercel bandwidth bill at scale. When your stack is Docker-native and you want a platform that deploys containers without asking you to reshape your architecture around function execution limits. Fly's anycast routing handles geographic routing automatically; you deploy to ord, ams, and syd with a single command.

Move to a VPS when: cost efficiency is the primary constraint, you have the DevOps knowledge (or time to acquire it), your traffic pattern is predictable enough that autoscaling is not a survival requirement, and you are running a stack that benefits from persistent server processes -- a Rails app, a Django monolith, a Go binary, a NestJS service with a BullMQ worker. Hetzner at \(4.39 to \)19/month for the CX series, with 20TB of included bandwidth, represents genuinely cheap compute. The operational overhead is the real cost.

The moment most teams miss is not when they first hit the Vercel bill -- it is when they start building architectural workarounds for Vercel's model. The first time you reach for Redis because your serverless function cannot hold state. The first time you find yourself warming a function on a cron job. The first time you discover that your queue consumer cannot stay alive between requests -- because Vercel terminates functions once the response completes, and no process persists until the next one arrives. Those are the signals. Each workaround is a quiet vote to move to a model that fits what you are building.

A Concrete Decision Matrix

The audit step in that last row matters. Before migrating off Vercel, check your usage dashboard. Middleware misuse is the most common source of unexpected Vercel bills -- middleware that a developer added after reading a tutorial and forgot is a documented billing culprit. Unoptimized server-side rendering that reruns on every request when it could be cached is the second. The migration often waits until you have eliminated the inefficiencies, because those same inefficiencies will make your VPS or Fly.io bill higher than it needs to be.

The Honest Summary

Vercel is genuinely excellent infrastructure for what it does. The problem is that developers pick it for projects that will eventually need what it cannot provide without workarounds, and they do not notice until the workarounds have accumulated into a migration. Fly.io occupies a position most comparisons skip: a server model (long-lived process, persistent state, real containers) with PaaS ergonomics (push to deploy, automatic TLS, anycast routing). A VPS gives you the most compute per dollar, the most control, and the largest ops surface.

None of these are wrong choices. They are choices that suit different moments in a product's life. Knowing which moment you are in is the actual skill.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

What changed is how hard that wall is enforced. What most coverage skips: whether any of it applies to you, if you're building on the Claude API rather than claude.ai directly.

Building something that might reach teenagers, a tutoring app, a coding tool for a high school, a support bot with a mixed-age audience? Two separate systems apply here: the consumer 18+ rule, and your obligations as a developer, each under its own terms. Conflating them is the most common error in how this story gets told.

This piece separates the two, covers how the enforcement side got here, and ends with a practical checklist for shipping something that isn't strictly adults-only.

Claude.ai enforcement flow: from sign-up affirmation through classifier flagging to ID-based appeal.

1. What Actually Changed, and When

The age floor isn't news, and neither is the conversation-scanning classifier: Anthropic doesn't publish a start date, and the post usually cited as this story's origin actually describes it as already-standard practice, not something newly switched on.

What that post (titled "Protecting the wellbeing of our users,", published December 18, 2025) announced as genuinely new was a second classifier in development for subtler signals (beyond an explicit "I'm in 8th grade"), plus Anthropic joining the Family Online Safety Institute (FOSI). Age policy was actually a small subsection there, not the focus: the post was mainly about suicide/self-harm safeguards and reducing sycophancy.

OpenAI made a parallel announcement the same week with a sharply different approach (covered in section 7). So "Claude is no longer available for minors" describes a policy that predates this news cycle by years, and the enforcement behind it isn't as new as the coverage implies either. The real escalation is biometric identity verification, a few months later.

2. How Anthropic Enforces the Wall on Claude.ai

The mechanics, as Anthropic has described them, stack up like this:

• Sign-up affirmation. Every Claude.ai account holder checks a box confirming they're 18 or older.

• Conversational classifiers. Self-identifying as under 18 in a chat gets the conversation flagged for human review, which can lead to suspension.

• App-store age signals. In certain US states, the App Store or Play Store now passes age signals directly to Claude's mobile app, blocking sign-up or sign-in before a conversation happens, a parallel checkpoint that bypasses the classifier entirely.

• Formal ID and age verification. In June 2026, Anthropic added a "Verification Data" category to its privacy policy, giving itself grounds to ask any Free, Pro, or Max user to confirm age or identity "in certain circumstances." Two vendors: Yoti handles age estimation (selfie estimate, ID scan, or "over 18" credential); Persona handles fuller identity checks, including a government-ID scan and a facial-geometry template some states classify as biometric data. Effective July 8, 2026.

The detail most writeups bury: this update explicitly does not apply to commercial Team, Enterprise, or API customers: it's Free/Pro/Max only. That line is the hinge the rest of this article turns on.

3. When Enforcement Backfired: the April 2026 False-Flag Wave

Active monitoring has a false-positive problem. Starting in April 2026, Reddit and X filled up with reports from adult, paying Pro-plan users incorrectly flagged and locked out. The suspension email told them: "Our team found signals that your account was used by a child," with a 30-day window to verify age through Yoti before the link expired. Some lost access to project histories in the process.

MediaNama put a formal list of questions to Anthropic: what signals the classifiers rely on beyond self-identification, the measured false-positive rate, how often suspensions get overturned. Anthropic hadn't answered publicly as of that reporting.

The backlash wasn't only about accuracy. An open letter signed by 400+ scientists and researchers warns that age-verification systems expand collection of sensitive data (biometrics, behavioral signals, context) and add risk of misuse, third-party access, and breach. The precedent: an October 2025 breach at Discord exposed roughly 70,000 government IDs submitted for age verification.

Users also flagged Persona itself, Anthropic's identity-verification vendor: it's backed by Founders Fund, the firm Peter Thiel co-founded, which is also an Anthropic investor, a disclosed potential conflict of interest, not an established wrongdoing, but part of why the rollout landed badly.

There's more behind that unease than the funding chain. In February 2026, two months before Anthropic named Persona as its vendor, researchers found part of Persona's front-end code exposed on a government-linked server, revealing capabilities including facial-recognition watchlist screening and the anti-money-laundering (AML) and know-your-customer (KYC) checks Persona also sells to financial and other institutions.

Persona's CEO disputed the core claims to Fortune, denying that Persona links biometrics to law-enforcement databases or has government or Palantir ties, and saying the files weren't a real vulnerability. Discord, which had been piloting Persona, ended the arrangement, though both said the trial had already wound down before the files surfaced. Anthropic picked Persona anyway, two months later.

4. The July 2026 ID Verification Rollout

Anthropic frames the July identity-verification rollout defensively: it lets flagged users prove their age rather than face a flat ban. But the timing stands out: published mid-June 2026, effective July 8, squarely inside a separate, unrelated dispute over model access.

That dispute: the U.S. Department of Commerce ordered Anthropic to suspend Claude Mythos 5 and Fable 5 over export-control concerns: launched June 9, suspended June 12, restriction lifted June 30, access restored July 1 (Anthropic's account here).

TechCrunch reported that being able to verify exactly who its users are gave Anthropic a stronger hand in that standoff. Whether that was a motivation or a side effect is interpretation; the timing overlap is record.

Practically, the data this flow can collect: a government-ID image plus its details (ID number, date of birth), a selfie photo or video, and a facial-geometry template, data that some jurisdictions, including Illinois under its biometric privacy law, treat as legally sensitive by default.

5. What the Claude API Actually Requires

The split that matters if you're a developer: Claude.ai (Free, Pro, Max, including Claude Code from those plans) falls under Consumer Terms and the 18+ wall above. Claude for Work/Government/Education and direct API access (including Bedrock and Vertex AI) fall under Commercial Terms and the Usage Policy instead, with no blanket age wall.

Anthropic's own policy page puts it cleanly: Claude.ai isn't offered to under-18 users, but developers on the API are bound by the Usage Policy: those serving minors face additional requirements, not a prohibition.

That's deliberate design, not a loophole. Anthropic updated its Usage Policy to let organizations build for minors on the API if they implement safety features and disclose the AI system to users. The operative document, Guidelines for Organizations Serving Minors, lays out four categories: age verification, content moderation and filtering, monitoring and reporting, and regulatory compliance and disclosure. There's no fixed template: what's appropriate scales with your product.

Separately, Anthropic's child safety guidance for developers spells out what's non-negotiable: the Usage Policy bans creating or distributing CSAM (including AI-generated), facilitating grooming, trafficking, or sextortion of a minor, and sexualizing minors "in any context, including fiction or roleplay," regardless of deployment size. You're responsible for your own end users misusing the product this way; Anthropic monitors API usage independently and takes its own enforcement action.

One more layer: Claude's own constitution instructs the model, absent signals otherwise, to treat messages as coming from what it calls a "relatively (but not unconditionally) trusted adult member of the public," but to adjust for strong signals of a minor. That's a useful backstop baked into training, but it isn't age verification and isn't a substitute for your own safeguards.

6. Building a Youth-Facing Product on Claude: a Checklist

If your product might have under-18 users on the Claude API rather than claude.ai, the four safeguard categories from section 5 break down into seven concrete things worth building. Three (age assurance, disclosure, monitoring) are less checklist items than small pieces of code. Here's how they wire together in a NestJS app.

The parts worth coding

Start with a guard gating the feature on your own age-assurance signal, not Anthropic's: the consumer classifiers only run against claude.ai conversations, so API traffic gets none of that by default:

import { Injectable, CanActivate, ExecutionContext, ForbiddenException } from '@nestjs/common';

type AgeBand = 'under13' | '13-17' | '18plus';

@Injectable()
export class AgeAssuranceGuard implements CanActivate {
  canActivate(context: ExecutionContext): boolean {
    const req = context.switchToHttp().getRequest();
    const ageBand: AgeBand | undefined = req.user?.ageBand;

    if (!ageBand) {
      throw new ForbiddenException('Age assurance required before using this feature.');
    }
    if (ageBand === 'under13') {
      throw new ForbiddenException('This feature is not available for this account.');
    }

    req.moderationTier = ageBand === '13-17' ? 'strict' : 'standard';
    return true;
  }
}

ageBand comes from whatever your product does for age assurance: self-attestation, a parent or guardian flow, a KYC vendor. The Usage Policy doesn't mandate the mechanism, just that one exists and matches your risk profile.

That guard only matters if something downstream reads what it sets. Here's the controller wiring it to the service call (the piece most samples skip):

import { Controller, Post, Body, Req, UseGuards } from '@nestjs/common';
import type { Request } from 'express';
import { AgeAssuranceGuard } from './age-assurance.guard';
import { ClaudeMinorSafeService } from './claude-minor-safe.service';

interface ChatRequestDto {
  message: string;
}

interface RequestWithModeration extends Request {
  user: { id: string };
  moderationTier: 'standard' | 'strict';
}

@Controller('chat')
export class ChatController {
  constructor(private readonly claude: ClaudeMinorSafeService) {}

  @UseGuards(AgeAssuranceGuard)
  @Post()
  async chat(@Req() req: RequestWithModeration, @Body() dto: ChatRequestDto): Promise<{ reply: string }> {
    const reply = await this.claude.reply(dto.message, req.moderationTier, req.user.id);
    return { reply };
  }
}

Inside reply(), disclosure and moderation both key off that same moderationTier, since both depend on who's on the other end:

import { Injectable } from '@nestjs/common';
import Anthropic from '@anthropic-ai/sdk';
import { AuditLogService } from './audit-log.service';

type ModerationTier = 'standard' | 'strict';

// Your own moderation heuristics or classifier calls go here. This list
// is illustrative, not a real safety layer on its own.
const FLAGGED_PATTERNS: RegExp[] = [/meet\s?up in person/i, /don't tell (your|my) parents/i];

@Injectable()
export class ClaudeMinorSafeService {
  private readonly client = new Anthropic();

  constructor(private readonly auditLog: AuditLogService) {}

  async reply(userMessage: string, moderationTier: ModerationTier, userId: string): Promise<string> {
    const disclosure =
      'You are an AI assistant, not a human; say so plainly if asked. ' +
      (moderationTier === 'strict'
        ? 'This user is a teenager. Keep responses age-appropriate: no mature themes, ' +
          'no unsupervised meetup suggestions, no requests for personal contact details.'
        : '');

    const response = await this.client.messages.create({
      model: 'claude-sonnet-5',
      max_tokens: 1024,
      system: disclosure,
      messages: [{ role: 'user', content: userMessage }],
    });

    const block = response.content[0];
    const text = block?.type === 'text' ? block.text : '';

    if (moderationTier === 'strict' && this.looksFlaggable(userMessage, text)) {
      await this.auditLog.record({ userId, userMessage, reply: text, reason: 'strict-tier pattern match' });
    }

    return text;
  }

  private looksFlaggable(input: string, output: string): boolean {
    return FLAGGED_PATTERNS.some((pattern) => pattern.test(input) || pattern.test(output));
  }
}

FLAGGED_PATTERNS is deliberately weak: a placeholder, not a moderation system; route this to a real classifier past prototype stage. What matters structurally is that a flagged interaction goes somewhere a human can act on it (the piece most implementations skip):

import { Inject, Injectable, Logger } from '@nestjs/common';
import { randomUUID } from 'crypto';

interface FlaggedInteraction {
  userId: string;
  userMessage: string;
  reply: string;
  reason: string;
}

// Wherever your restricted-access review store actually lives -- a
// dedicated table, a queue, a ticketing system. Full conversation
// content belongs only here, never in general application logs.
export interface ReviewQueue {
  enqueue(caseId: string, entry: FlaggedInteraction): Promise<void>;
}

@Injectable()
export class AuditLogService {
  private readonly logger = new Logger(AuditLogService.name);

  constructor(@Inject('REVIEW_QUEUE') private readonly reviewQueue: ReviewQueue) {}

  async record(entry: FlaggedInteraction): Promise<void> {
    const caseId = randomUUID();
    await this.reviewQueue.enqueue(caseId, entry);
    // The application log gets a case ID only -- never the minor's
    // message or Claude's reply. Those live in the restricted queue above.
    this.logger.warn('Flagged interaction queued for review', { caseId, reason: entry.reason });
  }
}

Notice what doesn't happen: the raw conversation never touches the application logger, only a case ID does. Logging a minor's actual message into general-purpose logs would be its own compliance problem. ReviewQueue is wherever your trust-and-safety process actually watches; a flagged interaction nobody reviews creates a paper trail showing you knew and didn't act, worse than not detecting it.

The rest is process, not code

• Map your COPPA, GDPR-K, and state-law obligations before you ship: these vary by where your users are, not your company.

• Treat the Guidelines for Organizations Serving Minors as a floor, not a template: the specifics are on you.

• Don't assume Claude's defaults fit your audience: baseline safety training is calibrated for a general adult; your moderation tier has to do the rest.

• Decide who owns escalation before you need them: wiring up the service is easy, having someone actually watch it is what teams skip.

7. How This Compares: OpenAI and Google

OpenAI took a different shape from the same December 2025 news cycle: rather than a hard wall, it updated ChatGPT's Model Spec with new principles for detected users under 18: age-tiered defaults for 13-17, safety prioritized over other goals, use prohibited outright only under 13. Age indication at sign-up has historically been voluntary, versus Anthropic's classifier-driven approach to catching self-IDs after the fact.

Google goes further the other way: Gemini reaches under-13 children via supervised Family Link accounts, plus a separate "teen experience" for 13-17-year-olds on their own accounts, the most permissive of the three, not a middle path.

The pattern: on the consumer side the three companies landed in genuinely different places: hard wall, age-tiered access, supervised access. On the developer side, all three converge on the same idea: build safeguards for whatever audience you're actually serving, since the consumer-facing rules won't do it for you.

8. The Regulatory Backdrop

None of this is a vacuum: the EU's Digital Services Act, Australia's under-16 ban (since December 2025), Malaysia's Online Safety Act (since January 2026, with an under-16 ban and eKYC mandate), and a US state-law wave tracing to Utah's 2023 Social Media Regulation Act (the same wave behind the app-store signal in section 2) all point one way: Anthropic is getting ahead of a locked-in trend, not inventing one. Context for section 3's enforcement problems, not an excuse for them.

9. What This Means for Your Roadmap

Two separate systems: the 18+ wall on Claude.ai is real, actively enforced, and (based on the false-positive wave) still rough. None of it applies to what you build on the API. If your product might reach a minor, you're not blocked; you're handed specific obligations under the Usage Policy and the Guidelines for Organizations Serving Minors, and implementing them is on you, not Anthropic's classifiers.

Given how fast the regulatory backdrop moves (the EU, Australia, Malaysia, a growing list of US states), the durable move is decoupling your own age-verification and consent layer from whatever Anthropic's consumer product is doing at any given moment. Build to the direction of travel, not today's ruleset, and revisit the linked sources periodically: every policy here has already changed at least once in the past year.

Two governance tracks: Consumer Terms with the 18+ wall versus Commercial Terms under the Usage Policy.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

Consistent hashing exists to prevent exactly that — it's why DynamoDB, Cassandra, and most distributed caches can scale without a self-inflicted thundering herd.

The Problem With hash(key) % N

Modulo-based sharding is intuitive: hash the key, take the remainder when divided by the number of servers, and that remainder tells you which server owns it. The problem is that N is baked directly into every single assignment. Change N by even one, and the remainder for almost every key changes along with it, because the modulo operation has no memory of what the previous assignment was.

Putting Servers and Keys on the Same Ring

Consistent hashing changes the model entirely. Picture a clock face instead of a number line: both the servers and the keys get hashed onto the same circle, so everything ends up sitting at some position on the dial. (A real implementation uses billions of positions instead of twelve, but the clock-face idea is all you need.)

To find which server owns a key: hash the key to find its spot on the dial, then move clockwise until you land on a server. Whichever one you hit first, owns that key. That's the entire lookup.

In practice, nothing actually "walks" anywhere — positions live in a sorted list, and finding the right server is one binary search: O(log V). Even a ring with millions of points needs just a handful of comparisons.

When a new server joins, it only intercepts keys sitting in the small stretch of the dial right before it — everyone else's assignment stays exactly the same. When a server leaves, its keys simply spill over to whichever server is next going clockwise, and nothing else changes.

A Tiny, Walkable Example

Forget real hash functions for a moment and use a ring of just 100 positions (0 to 99). Three servers sit on it: A at 10, B at 40, C at 75. Three keys hash to positions 25, 45, and 95.

Walking clockwise from each key to the first server: key 25 lands on B, key 45 lands on C, and key 95 wraps past 99 back to 0 and lands on A.

Now add a fourth server, D, at position 50:

• Key 25 still walks straight into B at 40 — D is further away and never comes into play. No change.

• Key 45 now hits D at 50 before it ever reaches C at 75. It moves, from C to D.

• Key 95 still wraps around to A at 10, completely unaffected by anything near position 50. No change.

Only the key sitting in the stretch immediately behind D — between B and D — ever noticed a new server showed up. Everything else on the ring didn't have to care.

Proving It: Real Rehashing Numbers

Take 100,000 keys, assign them across 4 servers, then add a 5th, and count how many keys land on a different server than before.

import hashlib, bisect

def h(key):
    return int(hashlib.md5(key.encode()).hexdigest(), 16)

def naive_assign(keys, num_servers):
    return {key: h(key) % num_servers for key in keys}

class ConsistentHashRing:
    def __init__(self, servers, vnodes=150):
        self.ring = {}
        for server in servers:
            for i in range(vnodes):
                pos = h(f"{server}#{i}")
                self.ring[pos] = server
        self.sorted_positions = sorted(self.ring.keys())

    def get_server(self, key):
        pos = h(key)
        idx = bisect.bisect(self.sorted_positions, pos)
        if idx == len(self.sorted_positions):
            idx = 0
        return self.ring[self.sorted_positions[idx]]

Naive hash%N:         80,087/100,000 keys moved  (80.1%)
Consistent hashing:   18,519/100,000 keys moved  (18.5%)

Naive modulo hashing reshuffles 80% of the entire dataset for a 25% capacity increase. Consistent hashing moves almost exactly what theory predicts it should — 1 divided by the new server count (1/5 = 20%) — and nothing more. Every key that didn't need to move, didn't.

The Catch: Uneven Load, and the Fix

Placing each server at just one random point on the ring has a problem of its own: with only a handful of random points, the arcs between them can be wildly uneven in size, and so can the load.

vnodes=  1:  min=1,332   max=40,206   stdev=18,565
vnodes=150:  min=18,340  max=22,565   stdev=1,766

With one ring position per server, one server ended up owning 40,206 keys while another owned just 1,332 — a 30x imbalance, from the same 5 servers and the same 100,000 keys.

The fix is virtual nodes: instead of placing each physical server once, place it 100 to 200 times at different hashed positions on the ring. More points per server means the randomness has more chances to average out, so the arcs even out. When a server does leave, its load spreads across many other servers instead of dumping entirely onto its single ring neighbor.

Weighted Consistent Hashing

Virtual nodes assume every server is equally capable, but real clusters are rarely that tidy — one box might have three times the RAM or CPU of another. The fix is a small variation on the same idea: give bigger servers proportionally more virtual nodes instead of an equal share.

import hashlib, bisect

def h(key):
    return int(hashlib.md5(key.encode()).hexdigest(), 16)

class WeightedRing:
    def __init__(self, server_weights, base_vnodes=50):
        self.ring = {}
        for server, weight in server_weights.items():
            for i in range(base_vnodes * weight):
                pos = h(f"{server}#{i}")
                self.ring[pos] = server
        self.sorted_positions = sorted(self.ring.keys())

    def get_server(self, key):
        pos = h(key)
        idx = bisect.bisect(self.sorted_positions, pos)
        if idx == len(self.sorted_positions):
            idx = 0
        return self.ring[self.sorted_positions[idx]]

weights = {"A": 1, "B": 1, "C": 3}  # C is a 3x-larger box
ring = WeightedRing(weights)

Testing that against 100,000 keys, where A and B are equal-sized and C is sized for three times the load:

A (weight 1): expected 20.0%, got 19.7%  (19,689 keys)
B (weight 1): expected 20.0%, got 22.8%  (22,829 keys)
C (weight 3): expected 60.0%, got 57.5%  (57,482 keys)

Close enough to the target split to be genuinely useful in practice. The small deviation — B ended up a bit ahead of A despite an identical weight — is just the expected noise from a finite, randomly-hashed sample, and it shrinks further as vnode counts increase. This is exactly how a cluster with mismatched hardware avoids either starving its small boxes or overloading its big ones.

Where This Shows Up in Production

• Amazon's Dynamo (and DynamoDB after it) popularized consistent hashing with virtual nodes as the core partitioning strategy for a distributed key-value store designed to scale without downtime.

• Apache Cassandra uses consistent hashing with virtual nodes to spread data across the cluster and rebalance automatically as nodes join or leave.

• Memcached client libraries (the well-known libketama implementation) use it so that adding a cache server doesn't invalidate the entire cache at once.

• Load balancers and CDNs use it to route requests so a given client or resource consistently lands on the same backend, without every backend change reshuffling all the traffic.

A Related Approach: Rendezvous Hashing

Consistent hashing isn't the only way to solve this problem. Rendezvous hashing, also called highest random weight (HRW), skips the ring entirely: for a given key, compute a combined hash of (key, server) for every server, and assign the key to whichever server produces the highest score.

Adding or removing a server changes that comparison, not any stored structure — the same minimal-movement result through a different mechanism. It trades the ring's O(log V) lookup for an O(N) scan, checking every server one by one: fine for a small cluster, less appealing for a very large one.

The Takeaway

Consistent hashing doesn't eliminate data movement when your cluster changes; a fifth of the keys above still had to move, and that's expected and fine. What it eliminates is the unnecessary movement — the other 80% that a naive modulo would have shuffled around for no real reason.

Combined with virtual nodes (weighted, if your hardware isn't uniform) to keep the load even, it's the reason a distributed system can grow or shrink its capacity as a routine operation instead of a disruptive one.

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

At least three other projects share the name:

• an archived poki-archive/three-ui canvas layer

• a small unrelated three-ui npm package

• petekp/three-ui, an experimental WebGL library exploring Chrome's HTML-in-canvas origin trial

Looser matches exist too (an AR-focused three-ui-ar package, for one). The namespace is crowded — say so if you meant a different project.

This review is checked directly against the live GitHub repo: README, file tree, and repo metadata, as of this writing. Pro internals aren't published, so this isn't a source-code audit of the private main project.

One limit worth naming: the install commands below are copied from the README, not executed. This environment has no network access to run npm install directly, and a registry search for @designcodeio/threeui didn't return an independent hit either way.

1. What it is

ThreeUI is a catalog of shader-driven UI pieces: hero sections, WebGL backgrounds, animated buttons, and full page templates. It ships three ways — a browsable site, an installable npm package, and, for Pro subscribers, a CLI that writes raw source into your project.

The public repo is the Community edition: 50 parent components across 111 routes, totaling 164 browse results (141 free variants plus 23 singleton components). It runs the same app shell, navigation, live renderers, and variant pickers as the main product.

The Community edition isn't Pro features switched off — it contains no authentication, account-state, or checkout code at all. The only thing actually missing is the Pro and Beta component implementations themselves.

2. Getting started

Prerequisites: not stated anywhere in the public docs. Neither the README nor the repo's rendered file tree exposes a Node engine constraint or a React peer-dependency range. Check package.json directly in the repo before pinning versions in a real project.

Option A — install as a dependency

npm install @designcodeio/threeui

import { AtTheHorizon } from "@designcodeio/threeui";
import "@designcodeio/threeui/style.css";

export function Hero() {
  return <AtTheHorizon />;
}

For a smaller import graph, pull components from their subpath instead of the package root:

import { AtTheHorizon } from "@designcodeio/threeui/components/AtTheHorizon";

One gotcha: some components render a full HTML document. They expect their runtime assets at fixed root-relative URLs — the same ones the hosted preview uses.

Copy the relevant files from node_modules/@designcodeio/threeui/lib-dist/assets/ into your public directory, or override the component's sourceUrl/assetBaseUrl prop. Treat this as an integration step, not a drop-in install.

Option B — run the Community catalog locally

npm install
npm run dev

Before shipping any change, run the full check:

npm run build

This runs the publication-boundary, type, and production-build checks together.

Getting Pro components

Pro source isn't published to npm. Subscribers authenticate in the browser, then pull an entitled source bundle through the CLI:

npx @designcodeio/threeui-cli add cross-beam

The CLI won't overwrite modified project files unless you pass --force. Run npx @designcodeio/threeui-cli --help for login, logout, and destination options.

3. Architecture review

Two build targets, one source tree

The repo carries separate vite.config.js and vite.lib.config.js files, plus split tsconfig.json / tsconfig.lib.json configs.

The catalog website and the publishable npm library are built from one shared component tree but two separate configs. A change to library-only build settings — externals, output format — doesn't touch the app build, and vice versa.

The Community edition is generated, not forked

A private main repo holds everything — Community, Pro, and Beta. This public repo is refreshed from it by a sync job: npm run sync:community -- /path/to/main-threeui.

That job:

• fails closed instead of erring open

• filters Pro and Beta content before building the public import graph

• strips restricted font assets

• writes three generated files:

  • public/community-sync-report.json — a parity/count report

  • public/source-code.json — the source bundle behind the in-app "Code" tab

  • src/data/shaders.tsx — the catalog/import file

The private repo runs this after every push to its main branch. Rather than pushing straight here, it commits to an automation/community-sync branch and opens a reviewed pull request. A sync with no public-facing change doesn't open one at all, so the public history stays clean.

If the filter step fails, the job errors instead of publishing a partial result. The specific failure it's built to prevent: Pro source leaking into the public repo through an incomplete sync.

Releases infer their own version

A versioned sync PR infers its own semver:

• new public components, variants, or controls → minor

• removals → major

• compatible source-only changes → patch

Merging that PR publishes to npm through trusted publishing with provenance, not a long-lived npm token sitting in CI.

A separate gate runs first: clean build, boundary audit, package creation, anonymous install smoke test. The Pro CLI installer is versioned and released independently of Pro content — a new Pro component doesn't force a CLI bump.

Entitlement checks run live

The CLI's OAuth+PKCE flow fits a public, secretless client — there's no client secret to protect. Its session is stored with owner-only file permissions.

Entitlement is checked on every server request, not once at login. A lapsed subscription stops working immediately, not after some cached grace window.

The license boundary

• App code, Community component code, and ThreeUI-authored Community imagery: MIT

• Bundled fonts: SIL Open Font License 1.1

• Bundled Three.js runtime files: their own MIT license

• Remote catalog thumbnails and previews loaded live from threeui.com: not redistributed under the repo's license

The boundary sits between what's checked into the repo and what's fetched live from the site at render time. Everything in the repo — code and imagery alike — ships under MIT; only the live-loaded assets fall outside it.

4. Things to watch

• node_modules won't show the full picture. Pro source arrives through an authenticated CLI download, not an npm dependency. A dependency scan of a Pro-using project won't surface it.

• Full-document components assume a hosting layout. Their default asset URLs mirror the hosted preview site's paths. Plan for that coupling during integration.

• Some files in the tree are generated, not source. public/community-sync-report.json, public/source-code.json, and src/data/shaders.tsx are written by the sync job. A PR editing them directly would likely get overwritten by the next sync — confirm contribution scope with maintainers first.

• Activity is light but present. At last check the repo carried 5 open issues and 4 open pull requests against 22 commits total — enough to gauge maintenance pace, not enough to call it a heavily trafficked issue tracker either way.

• Third-party listings here are unverifiable. 21st.dev claims ThreeUI publishes an official shadcn registry at threeui.com, but its own component count for that listing swung between 109, 44, and 33 across fetches minutes apart. threeui.com renders client-side, so its content isn't independently fetchable either. Treat this channel as unresolved, not confirmed.

Further reading

• https://github.com/MengTo/threeui

• https://threeui.com

Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.