<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[ZyVOP]]></title><description><![CDATA[One platform to write, cross-post to Dev.to, Medium, WordPress, Hashnode, and Bluesky, and protect your SEO with canonical links. Zero paywalls and full content]]></description><link>https://blog.zyvop.com</link><image><url>https://cdn.hashnode.com/uploads/logos/6a1714c0badcd8afcb06dd34/c81de29c-f10d-4d66-8ba9-33b6c7dedc0a.png</url><title>ZyVOP</title><link>https://blog.zyvop.com</link></image><generator>RSS for Node</generator><lastBuildDate>Wed, 11 Nov 2026 13:08:16 GMT</lastBuildDate><atom:link href="https://blog.zyvop.com/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[TypeScript for AI Engineers: The Language of AI Web Apps]]></title><description><![CDATA[Most AI products need a web front end, and that means TypeScript. This post teaches it to Python programmers: the JavaScript gotchas, the type system, Zod…]]></description><link>https://blog.zyvop.com/typescript-for-ai-engineers-the-language-of-ai-web-apps-2ge48</link><guid isPermaLink="true">https://blog.zyvop.com/typescript-for-ai-engineers-the-language-of-ai-web-apps-2ge48</guid><category><![CDATA[#aiengineer]]></category><category><![CDATA[JavaScript]]></category><category><![CDATA[Node.js]]></category><category><![CDATA[streaming]]></category><category><![CDATA[TypeScript]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Wed, 11 Nov 2026 07:37:40 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/fd25c8b2-4576-49cd-afed-2ff9974e5db5.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>You can build a great AI career in Python alone. But people use web pages, and the chat boxes, dashboards, and prompt playgrounds on those pages are usually built with <strong>TypeScript</strong>. Learn it, and you can ship the whole product yourself.</p>
<p>This post is core. It's a fast path for Python programmers, not a full course, and by the end you'll have written a typed, tested client for the API from Post 20, streaming included.</p>
<p>Everything here ran on Node 22 and TypeScript 6.0.3, so the outputs and error messages are real. The exception is the ecosystem section (React, Next.js, ESLint), which I describe from knowledge because I couldn't install those here.</p>
<h2>Why TypeScript, and what it actually is</h2>
<p><strong>JavaScript</strong> is the language that runs in every web browser, and, through a program called <strong>Node.js</strong>, on servers too. It's flexible and a bit loose, which is great for small scripts and painful for large projects.</p>
<p><strong>TypeScript</strong> is JavaScript <em>plus types</em>. You write type annotations, much like the type hints from Post 10, and a tool called the compiler (<code>tsc</code>) checks them and catches mistakes before your code runs. Then the types are <strong>erased</strong>, leaving plain JavaScript that runs anywhere. That last detail has big consequences, and we'll get to them later.</p>
<p>Where does it fit in an AI app? Here's the usual shape:</p>
<pre><code class="language-rust">browser (TypeScript/React)  --&gt;  your server (TypeScript or Python)  --&gt;  model API
         the chat screen              holds the API key, adds logic           the model
</code></pre>
<p>The browser should never hold your API key (Post 20), so your server sits in the middle, keeps the secrets, and streams results back. That server can be TypeScript or Python, and the pieces talk to each other over HTTP, which you already understand.</p>
<h2>Setting up</h2>
<p>You need Node.js (get the LTS version from nodejs.org). Check it:</p>
<pre><code class="language-bash">node --version
npm --version
</code></pre>
<p><code>npm</code> is Node's package manager. It does for JavaScript what <code>pip</code> and <code>uv</code> do for Python. Create a project:</p>
<pre><code class="language-bash">mkdir chat-api-client
cd chat-api-client
npm init -y
npm install --save-dev typescript tsx @types/node
</code></pre>
<p>That installs three development tools:</p>
<ul>
<li><p><code>typescript</code> is the compiler and type checker.</p>
</li>
<li><p><code>tsx</code> runs TypeScript files directly, without a separate build step. It's the easiest way to run TS while learning.</p>
</li>
<li><p><code>@types/node</code> teaches TypeScript about Node's built-in features.</p>
</li>
</ul>
<p>Notice a folder called <code>node_modules/</code> appears, full of installed packages, along with <code>package.json</code> (your project's description and dependencies, like <code>pyproject.toml</code> from Post 12) and <code>package-lock.json</code> (the exact versions installed, like <code>uv.lock</code>). <strong>Commit the lock file, and add</strong> <code>node_modules/</code> <strong>to your</strong> <code>.gitignore</code> (Post 22). <code>node_modules</code> can be enormous, and anyone can rebuild it by running <code>npm install</code>.</p>
<p>Add <code>"type": "module"</code> to <code>package.json</code>, so your files use the modern import syntax. And create a <code>tsconfig.json</code> in the project folder to configure the compiler:</p>
<pre><code class="language-json">{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "noEmit": true,
    "skipLibCheck": true,
    "lib": ["ES2022", "DOM", "DOM.Iterable"],
    "types": ["node"]
  },
  "include": ["src"]
}
</code></pre>
<p>The most important line is <code>"strict": true</code>. It turns on the strictest checks, including the one that catches <code>null</code> and <code>undefined</code> mistakes. Always use it. TypeScript's defaults change between versions, so spelling out your settings, as above, keeps behavior predictable. <code>noEmit</code> means "just check, don't produce files," since <code>tsx</code> handles running.</p>
<p>Now you have two commands, and the difference between them is important:</p>
<pre><code class="language-bash">npx tsx src/hello.ts      # RUN the file. Does NOT check types.
npx tsc --noEmit          # CHECK the types. Doesn't run anything.
</code></pre>
<p><code>tsx</code> strips the types and runs the code, even if there are type errors. <code>tsc</code> does the checking. Get in the habit of running the check regularly, and add both to <code>package.json</code> so they're easy to remember:</p>
<pre><code class="language-json">"scripts": {
  "start": "tsx src/client.ts",
  "check": "tsc --noEmit",
  "test": "tsx --test src/*.test.ts"
}
</code></pre>
<p>Then <code>npm run check</code> and <code>npm test</code> do the work. (Newer versions of Node can also run TypeScript files directly by stripping types, but the details depend on your Node version, so I'll stick with <code>tsx</code> here.)</p>
<h2>JavaScript for Python programmers</h2>
<p>TypeScript <em>is</em> JavaScript, so you need the JavaScript basics first. Here's a translation table, and then the gotchas that bite Python programmers.</p>
<table>
<thead>
<tr>
<th>Python</th>
<th>TypeScript / JavaScript</th>
</tr>
</thead>
<tbody><tr>
<td>x = 5</td>
<td>const x = 5 (or let x = 5 if it will change)</td>
</tr>
<tr>
<td>None</td>
<td>null or undefined (two kinds of "nothing")</td>
</tr>
<tr>
<td>True / False</td>
<td>true / false</td>
</tr>
<tr>
<td>f"Hello, {name}"</td>
<td><code>Hello, ${name}</code> (backticks)</td>
</tr>
<tr>
<td>len(items)</td>
<td>items.length</td>
</tr>
<tr>
<td>x in items</td>
<td>items.includes(x)</td>
</tr>
<tr>
<td>for x in items:</td>
<td>for (const x of items) { ... }</td>
</tr>
<tr>
<td>[n * n for n in nums if n % 2 == 0]</td>
<td>nums.filter(n =&gt; n % 2 === 0).map(n =&gt; n * n)</td>
</tr>
<tr>
<td>{"a": 1} (dict)</td>
<td>{ a: 1 } (an object), or a Map</td>
</tr>
<tr>
<td>print(x)</td>
<td>console.log(x)</td>
</tr>
<tr>
<td>def f(x): ...</td>
<td>function f(x) { ... } or const f = (x) =&gt; ...</td>
</tr>
<tr>
<td>try / except</td>
<td>try / catch</td>
</tr>
<tr>
<td>async def / await</td>
<td>async function / await</td>
</tr>
<tr>
<td>dataclass / Pydantic</td>
<td>interface or type, plus Zod for runtime checks</td>
</tr>
<tr>
<td>pip / uv, pyproject.toml</td>
<td>npm, package.json</td>
</tr>
<tr>
<td>pytest</td>
<td>Vitest, Jest, or Node's built-in test runner</td>
</tr>
</tbody></table>
<p>I ran each of the following. Here they are, with the output.</p>
<p><strong>One number type.</strong> There's no separate <code>int</code> and <code>float</code>:</p>
<pre><code class="language-typescript">console.log(typeof 1, typeof 1.5, 0.1 + 0.2, 7 / 2, Math.floor(7 / 2));
// number number 0.30000000000000004 3.5 3
</code></pre>
<p>Note that <code>7 / 2</code> is <code>3.5</code>, as in Python 3. For whole-number division, use <code>Math.floor</code>. The <code>0.1 + 0.2</code> quirk from Post 4 is the same.</p>
<p><strong>Two kinds of "nothing".</strong> <code>null</code> means "deliberately empty," and <code>undefined</code> means "never set." You'll see both:</p>
<pre><code class="language-typescript">let a: string | null = null;
let b: string | undefined;
console.log(a, b, a == b, a === b);
// null undefined true false
</code></pre>
<p><strong>Always use</strong> <code>===</code><strong>.</strong> The double-equals <code>==</code> converts types before comparing, with surprising results:</p>
<pre><code class="language-typescript">console.log(0 == "", 0 === "", "1" == 1);
// true false true
</code></pre>
<p><code>0 == ""</code> is <code>true</code>. Don't use <code>==</code>. Happily, TypeScript flags many of these comparisons for you: it reports <code>This comparison appears to be unintentional because the types 'number' and 'string' have no overlap.</code></p>
<p><strong>Truthiness differs from Python.</strong> An empty array and an empty object are <strong>truthy</strong> in JavaScript, while they're falsy in Python:</p>
<pre><code class="language-typescript">console.log([] ? "empty array is truthy" : "falsy");     // empty array is truthy
console.log([0, "", null, undefined, NaN].map((v) =&gt; Boolean(v)));
// [ false, false, false, false, false ]
</code></pre>
<p>The falsy values are <code>false</code>, <code>0</code>, <code>""</code>, <code>null</code>, <code>undefined</code>, and <code>NaN</code>. To check for an empty array, test <code>items.length === 0</code>. TypeScript even complains about <code>[] ? ...</code>, with <code>This kind of expression is always truthy.</code></p>
<p><code>??</code> <strong>versus</strong> <code>||</code><strong>.</strong> This one causes real bugs. <code>||</code> falls back for <em>any</em> falsy value, including a legitimate <code>0</code>. The "nullish coalescing" operator <code>??</code> falls back only for <code>null</code> and <code>undefined</code>:</p>
<pre><code class="language-typescript">const retries = 0;
console.log("with ||:", retries || 3, "| with ??:", retries ?? 3);
// with ||: 3 | with ??: 0
</code></pre>
<p>If someone sets <code>retries</code> to <code>0</code> on purpose, <code>||</code> quietly turns it into <code>3</code>. Use <code>??</code> for defaults. Its partner is <strong>optional chaining</strong>, <code>?.</code>, which stops safely if something is missing:</p>
<pre><code class="language-typescript">const user: { profile?: { email?: string } } = {};
console.log(user.profile?.email, user.profile?.email ?? "no email");
// undefined no email
</code></pre>
<p>Without <code>?.</code>, <code>user.profile.email</code> would crash, as <code>None.email</code> does in Python.</p>
<p><strong>Array methods</strong> replace comprehensions:</p>
<pre><code class="language-typescript">const nums = [1, 2, 3, 4, 5, 6];
console.log(nums.filter((n) =&gt; n % 2 === 0).map((n) =&gt; n * n));   // [ 4, 16, 36 ]
console.log(nums.reduce((sum, n) =&gt; sum + n, 0));                 // 21
console.log(nums.find((n) =&gt; n &gt; 3), nums.some((n) =&gt; n &gt; 5), nums.every((n) =&gt; n &gt; 0));
// 4 true true
</code></pre>
<p><strong>The sorting gotcha.</strong> <code>.sort()</code> with no argument sorts as <em>text</em>:</p>
<pre><code class="language-typescript">console.log([10, 9, 1, 100].sort());                    // [ 1, 10, 100, 9 ]
console.log([10, 9, 1, 100].sort((x, y) =&gt; x - y));     // [ 1, 9, 10, 100 ]
</code></pre>
<p>Always pass a comparison function for numbers.</p>
<p><strong>Destructuring and spread</strong> are lovely, and you'll use them constantly:</p>
<pre><code class="language-typescript">const { role, content } = { role: "user", content: "hi", extra: 1 };
const [first, ...rest] = [1, 2, 3];
const merged = { ...{ model: "x", temperature: 0.7 }, temperature: 0.2 };
console.log(role, content, first, rest, merged);
// user hi 1 [ 2, 3 ] { model: 'x', temperature: 0.2 }
</code></pre>
<p>The spread <code>...</code> copies an object's fields, and later fields win, so <code>merged</code> has <code>temperature: 0.2</code>. That's the standard way to apply overrides to default settings.</p>
<p><strong>Objects as dictionaries, and</strong> <code>Map</code><strong>.</strong> A plain object works for string keys. A <code>Map</code> is better when keys vary or when you add and remove a lot:</p>
<pre><code class="language-typescript">const counts: Record&lt;string, number&gt; = {};
for (const word of "the cat the dog".split(" ")) {
  counts[word] = (counts[word] ?? 0) + 1;
}
console.log(counts);     // { the: 2, cat: 1, dog: 1 }
</code></pre>
<p>That's the word counter from Post 5 again, with <code>??</code> playing the role of <code>.get(word, 0)</code>.</p>
<p><strong>JSON is built in</strong>, and speaks the language natively:</p>
<pre><code class="language-typescript">const text = JSON.stringify({ role: "user", content: "hi" });
console.log(text, JSON.parse(text).content);
// {"role":"user","content":"hi"} hi
</code></pre>
<p><code>JSON.parse</code> throws a <code>SyntaxError</code> on bad input, so wrap it in <code>try/catch</code> when the data comes from outside.</p>
<p><strong>Big numbers lose precision.</strong> All numbers are 64-bit floats, so whole numbers are only exact up to a point:</p>
<pre><code class="language-typescript">console.log(Number.MAX_SAFE_INTEGER, 9007199254740993, 9007199254740993n);
// 9007199254740991 9007199254740992 9007199254740993n
</code></pre>
<p>The middle value was written as <code>...993</code> but printed as <code>...992</code>. It silently changed. The <code>n</code> suffix makes a <code>BigInt</code>, which is exact. This matters when a database ID or a huge counter passes through JavaScript: keep such IDs as strings.</p>
<p><code>for...of</code> <strong>versus</strong> <code>for...in</code><strong>.</strong> Use <code>for (const x of items)</code> to loop over <em>values</em>. <code>for...in</code> loops over keys, which for an array means the indexes <em>as strings</em>, which is almost never what you want:</p>
<pre><code class="language-typescript">for (const x of ["a", "b"]) console.log("of:", x);       // of: a, of: b
for (const i in ["a", "b"]) console.log("in:", i, typeof i);   // in: 0 string, in: 1 string
</code></pre>
<h2>TypeScript basics</h2>
<p>Now the types. If you've done Post 10, much of this will feel familiar.</p>
<h3>Describing shapes</h3>
<p>An <strong>interface</strong> (or a <code>type</code>) describes the shape of an object. Here's a chat message, using a <strong>literal union</strong> for the role, the TypeScript cousin of Python's <code>Literal</code>:</p>
<pre><code class="language-typescript">type Role = "user" | "assistant" | "system";

interface Message {
  role: Role;
  content: string;
  tokens?: number;     // the ? makes it optional
}

function tokenCount(message: Message): number {
  return message.tokens ?? Math.ceil(message.content.length / 4);
}

console.log(tokenCount({ role: "user", content: "How do generators work?" }));   // 6
console.log(tokenCount({ role: "user", content: "hi", tokens: 9 }));              // 9
</code></pre>
<p>You don't have to annotate everything. TypeScript <strong>infers</strong> types, so <code>const n = 5</code> is already known to be a number. Annotate function parameters and return types, and let inference handle the rest.</p>
<h3>Letting the compiler catch your mistakes</h3>
<p>Here's where it pays off. I wrote a file full of deliberate mistakes using that <code>Message</code> type, and ran <code>tsc</code>. These are the real messages:</p>
<pre><code class="language-python">Type '"robot"' is not assignable to type '"user" | "assistant" | "system"'.
</code></pre>
<p>You gave a role that isn't allowed.</p>
<pre><code class="language-python">Property 'content' is missing in type '{ role: "user"; }' but required in type 'Message'.
</code></pre>
<p>A required field is missing.</p>
<pre><code class="language-python">'m.tokens' is possibly 'undefined'.
</code></pre>
<p>You used <code>m.tokens + 1</code>, but <code>tokens</code> is optional. This is the classic one, the cousin of Python's <code>NoneType has no attribute</code> error from Post 11, except it's caught <em>before</em> you run anything.</p>
<pre><code class="language-python">'text' is possibly 'null'.
</code></pre>
<p>Calling <code>text.toUpperCase()</code> when <code>text</code> might be <code>null</code>.</p>
<pre><code class="language-typescript">Object literal may only specify known properties, but 'tokns' does not exist in type 'Message'. Did you mean to write 'tokens'?
</code></pre>
<p>A typo in a field name, and TypeScript even suggests the fix.</p>
<pre><code class="language-python">Argument of type 'string' is not assignable to parameter of type 'number'.
</code></pre>
<p>Calling <code>double("21")</code> on a function that takes a number.</p>
<pre><code class="language-python">'value' is of type 'unknown'.
</code></pre>
<p>Using a value of type <code>unknown</code> without checking it first (more on that below).</p>
<pre><code class="language-typescript">Function lacks ending return statement and return type does not include 'undefined'.
</code></pre>
<p>This one appeared in a <code>switch</code> that forgot a case, which brings us to the next feature.</p>
<p>Read each message from the start: it names the problem, and the position tells you the line. Most are short and clear.</p>
<h3>Unions that tell you what you're holding</h3>
<p>A <strong>discriminated union</strong> is a set of shapes, each with a field that says which one it is. It's perfect for streaming events, where a stream can send tokens, errors, or a "done" signal:</p>
<pre><code class="language-typescript">type StreamEvent =
  | { type: "token"; token: string }
  | { type: "error"; message: string }
  | { type: "done" };

function describe(event: StreamEvent): string {
  switch (event.type) {
    case "token":
      return `token: ${event.token}`;
    case "error":
      return `error: ${event.message}`;
    case "done":
      return "done";
  }
}
</code></pre>
<p>Inside each <code>case</code>, TypeScript <em>narrows</em> the type: in the <code>"token"</code> branch, it knows <code>event</code> has a <code>token</code> field, and in the <code>"error"</code> branch it knows about <code>message</code>. And if you forget a case, as I did on purpose in my error file, the compiler objects with the "lacks ending return statement" error above. Add a new event type later, and the compiler shows you every place that needs updating.</p>
<h3><code>unknown</code>, not <code>any</code></h3>
<p>TypeScript has a type called <code>any</code> that switches checking off completely. <strong>Avoid it.</strong> It defeats the purpose, and it spreads. When you genuinely don't know what something is, like data from <code>JSON.parse</code> or a network response, use <code>unknown</code>. It means "could be anything, and you must check before using it":</p>
<pre><code class="language-typescript">function isMessage(value: unknown): value is Message {
  if (typeof value !== "object" || value === null) return false;
  const v = value as Record&lt;string, unknown&gt;;
  return (
    (v.role === "user" || v.role === "assistant" || v.role === "system") &amp;&amp;
    typeof v.content === "string"
  );
}

const incoming: unknown = JSON.parse('{"role": "assistant", "content": "Hello"}');
if (isMessage(incoming)) {
  console.log("valid message:", incoming.content.toUpperCase());   // valid message: HELLO
}
console.log(isMessage({ role: "robot", content: "x" }), isMessage(null), isMessage("text"));
// false false false
</code></pre>
<p>The <code>value is Message</code> in the signature makes <code>isMessage</code> a <strong>type guard</strong>. When it returns <code>true</code>, TypeScript treats the value as a <code>Message</code> inside that <code>if</code>. It's your own runtime check, with the type system's blessing.</p>
<h3>Generics and handy utilities</h3>
<p><strong>Generics</strong> let a function work with any type while staying type-safe, like <code>list[T]</code> in Python's hints:</p>
<pre><code class="language-typescript">function first&lt;T&gt;(items: T[]): T | undefined {
  return items[0];
}

const n = first([1, 2, 3]);       // TypeScript knows: number | undefined
const s = first(["a", "b"]);      // string | undefined
</code></pre>
<p>You'll meet generics constantly in the form <code>Promise&lt;string&gt;</code> ("a promise that will produce a string"), <code>Array&lt;number&gt;</code>, and <code>Record&lt;string, number&gt;</code> (an object with string keys and number values). A few more tools you'll use:</p>
<pre><code class="language-typescript">const ROLES = ["user", "assistant", "system"] as const;    // a fixed list
type RoleFromList = (typeof ROLES)[number];                // "user" | "assistant" | "system"

type Options = { temperature: number; maxTokens: number; stream: boolean };
const overrides: Partial&lt;Options&gt; = { temperature: 0.2 };   // every field optional
</code></pre>
<p><code>as const</code> freezes a list into exact literal types, so you can derive a union from it. <code>Partial&lt;T&gt;</code> makes every field of <code>T</code> optional, which is ideal for "settings you may override."</p>
<h2>Types vanish at runtime</h2>
<p>Here's the most important lesson, and it echoes Post 10. TypeScript types exist only while you write code. They're <strong>erased</strong> before the program runs, so they can't check data that arrives from outside, like a network response or a model's JSON. Watch what happens when you simply <em>tell</em> TypeScript what the data is, using a cast:</p>
<pre><code class="language-typescript">const trusted = JSON.parse('{"role": "robot", "content": 42}') as Message;
console.log(trusted.role, typeof trusted.content);
// robot number
</code></pre>
<p>TypeScript believed us. The "message" has a role that isn't allowed and content that isn't a string, and nothing complained. A cast (<code>as Message</code>) is a promise you make to the compiler, and it doesn't verify anything. So for data from outside your program, never cast. Validate.</p>
<p>You can write a type guard by hand, as above, but for anything bigger, use a library like <strong>Zod</strong>, the TypeScript equivalent of Pydantic. You describe the shape <em>once</em>, and you get both runtime validation and the TypeScript type:</p>
<pre><code class="language-typescript">import { z } from "zod";

const MessageSchema = z.object({
  role: z.enum(["user", "assistant", "system"]),
  content: z.string(),
  tokens: z.number().int().nonnegative().optional(),
});
type ValidatedMessage = z.infer&lt;typeof MessageSchema&gt;;

const good: ValidatedMessage = MessageSchema.parse({ role: "user", content: "hi" });
console.log(good);     // { role: 'user', content: 'hi' }

const result = MessageSchema.safeParse({ role: "robot", content: 42 });
if (!result.success) {
  for (const issue of result.error.issues) {
    console.log(" -", issue.path.join("."), issue.message);
  }
}
// - role Invalid enum value. Expected 'user' | 'assistant' | 'system', received 'robot'
// - content Expected string, received number
</code></pre>
<p><code>parse</code> returns the clean, typed data, or throws a <code>ZodError</code>. <code>safeParse</code> returns a result object instead of throwing, with <code>success</code> telling you which it was, so you can handle failure without <code>try/catch</code>. This is exactly the "validate the model's reply" pattern from Post 10, and it's how you'd check structured output in a TypeScript app.</p>
<p>(I ran this with Zod 3.23.8, installed with <code>npm install zod</code>. Zod's error-message wording changes between major versions, so yours may read differently, and the basic API is the same.)</p>
<h2>Async, fetch, and the web</h2>
<p>If you've done Post 14, JavaScript's async will feel like home. In fact, it's simpler, because JavaScript has had an event loop built in from the start. There's no <code>asyncio.run</code>. You just write <code>async</code> functions and <code>await</code> them, and in modern modules you can even <code>await</code> at the top level of a file.</p>
<p>The equivalent of <code>asyncio.gather</code> is <code>Promise.all</code>. I made three delayed values of different types:</p>
<pre><code class="language-typescript">async function delay&lt;T&gt;(value: T, ms: number): Promise&lt;T&gt; {
  await new Promise((resolve) =&gt; setTimeout(resolve, ms));
  return value;
}

const results = await Promise.all([delay("a", 30), delay(2, 10), delay(true, 20)]);
console.log(results);
// [ 'a', 2, true ]
</code></pre>
<p>They run concurrently, and the results come back in the original order, just like <code>gather</code>. TypeScript even tracks each element's type.</p>
<h3>fetch, and its most important gotcha</h3>
<p>JavaScript has <code>fetch</code> built in (it's in browsers, and in Node 18 and newer). It's the equivalent of Python's <code>requests</code>:</p>
<pre><code class="language-typescript">const response = await fetch("http://localhost:8000/ask", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ question: "What is HTTP?" }),
  signal: AbortSignal.timeout(5000),
});
</code></pre>
<p><code>AbortSignal.timeout(5000)</code> gives the request a five-second limit, the equivalent of <code>timeout=5</code> in Python, and just as essential.</p>
<p>Now the gotcha, which catches everyone coming from <code>requests</code>: <code>fetch</code> <strong>does not throw on HTTP errors.</strong> A 404 or a 500 is still a successful <em>conversation</em>, so the promise resolves normally. Only network failures and timeouts throw. You must check <code>response.ok</code> yourself, which is the job <code>raise_for_status()</code> did in Python. I sent an empty request to the server from Post 20:</p>
<pre><code class="language-typescript">fetch resolved normally with status 422 ok = false { error: "missing 'question'" }
</code></pre>
<p>No exception. If you forget the <code>response.ok</code> check, your code carries on treating an error message as a good answer.</p>
<h2>Streaming in TypeScript</h2>
<p>Streaming is where TypeScript shines, because it's what powers chat screens. A <code>fetch</code> response has a <code>body</code>, which is a stream of bytes you can read piece by piece. Here's the plan, and it should remind you of Posts 7 and 20:</p>
<ol>
<li><p>Get a <strong>reader</strong> from <code>response.body</code>.</p>
</li>
<li><p>Read chunks as they arrive, and decode the bytes into text.</p>
</li>
<li><p>Collect the text in a <strong>buffer</strong>, and pull out complete lines.</p>
</li>
<li><p>For each <code>data:</code> line, parse the JSON and <strong>yield</strong> the token.</p>
</li>
</ol>
<p>Step 3 deserves a warning. A chunk from the network can end <em>in the middle of a line</em>, or even in the middle of a word, so you can't just parse each chunk as it comes. You keep a buffer, and only process lines once you've seen their newline. We'll test exactly that in the mini project.</p>
<p>The natural way to expose this is an <strong>async generator</strong>, the TypeScript version of the generators from Post 7 and the <code>async for</code> from Post 14. You write <code>async function*</code>, and consume it with <code>for await</code>:</p>
<pre><code class="language-typescript">for await (const token of streamTokens("Streaming makes replies feel faster")) {
  process.stdout.write(token);
}
</code></pre>
<p>(In a browser, you'd append each token to a page element instead. The browser's built-in <code>EventSource</code> for server-sent events only makes <code>GET</code> requests and can't set headers, as we saw in Post 20, so web apps typically read the stream with <code>fetch</code>, using the same technique.)</p>
<h2>The ecosystem, briefly</h2>
<p>I can't teach all of web development here, so here's a map of what you'll meet. This part is from my knowledge, and these tools change quickly, so check current documentation before choosing.</p>
<ul>
<li><p><strong>React</strong> is the most widely used library for building user interfaces, and <strong>Next.js</strong> is a popular framework built on it that also lets you write server code in the same project. Learning React basics is the natural next step after this post if you want to build chat screens.</p>
</li>
<li><p><strong>Servers</strong> in Node are commonly built with frameworks like Express, Fastify, or Hono. For simple needs, Next.js's built-in server routes are enough.</p>
</li>
<li><p><strong>SDKs.</strong> Model providers publish TypeScript SDKs, there are higher-level toolkits for building AI apps in TypeScript, and an official TypeScript SDK exists for the Model Context Protocol, which we'll meet in Post 56.</p>
</li>
<li><p><strong>Testing.</strong> Vitest and Jest are popular. Node also has a built-in test runner (<code>node:test</code>), which is what we'll use below.</p>
</li>
<li><p><strong>Formatting and linting.</strong> Prettier formats code, and ESLint (or Biome) lints it, like the Python tools from Post 24.</p>
</li>
</ul>
<p>And a decision guide, since people often ask which language to use:</p>
<table>
<thead>
<tr>
<th>Choose</th>
<th>When</th>
</tr>
</thead>
<tbody><tr>
<td>Python for the backend</td>
<td>The work is data, ML, evals, notebooks, or training, and you need Python's libraries</td>
</tr>
<tr>
<td>TypeScript for the backend</td>
<td>The product is mostly web, you want one language across front and back end, or you're deploying to serverless or edge platforms</td>
</tr>
<tr>
<td>Both</td>
<td>Very common: a Python service does the ML work, a TypeScript app builds the product, and they talk over HTTP</td>
</tr>
</tbody></table>
<h2>Mini project: a typed client for the Post 20 API</h2>
<p>Let's put it all together. We'll write a fully typed TypeScript client for the server from Post 20: asking questions, streaming tokens, and retrying on rate limits, with real tests. The project needs the <code>package.json</code> scripts and the <code>tsconfig.json</code> from above, and then two files. Create <code>src/client.ts</code>:</p>
<pre><code class="language-typescript">const BASE = process.env.API_BASE ?? "http://localhost:8000";

export class ApiError extends Error {
  constructor(
    public status: number,
    message: string,
  ) {
    super(message);
    this.name = "ApiError";
  }
}

type FetchFn = typeof fetch;

function isAnswer(value: unknown): value is { answer: string } {
  return (
    typeof value === "object" &amp;&amp;
    value !== null &amp;&amp;
    typeof (value as Record&lt;string, unknown&gt;).answer === "string"
  );
}

export async function ask(question: string, fetchFn: FetchFn = fetch): Promise&lt;string&gt; {
  const response = await fetchFn(`${BASE}/ask`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ question }),
    signal: AbortSignal.timeout(5000),
  });
  if (!response.ok) {
    throw new ApiError(response.status, `ask failed with status ${response.status}`);
  }
  const data: unknown = await response.json();
  if (!isAnswer(data)) {
    throw new Error("unexpected response shape from /ask");
  }
  return data.answer;
}

export async function* streamTokens(text: string, fetchFn: FetchFn = fetch): AsyncGenerator&lt;string&gt; {
  const response = await fetchFn(`${BASE}/stream`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ text }),
    signal: AbortSignal.timeout(10000),
  });
  if (!response.ok || response.body === null) {
    throw new ApiError(response.status, `stream failed with status ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      buffer += decoder.decode(value, { stream: true });

      // a chunk can end in the middle of a line, so only handle complete lines
      let newline: number;
      while ((newline = buffer.indexOf("\n")) !== -1) {
        const line = buffer.slice(0, newline).trimEnd();
        buffer = buffer.slice(newline + 1);
        if (!line.startsWith("data: ")) continue;

        const payload = line.slice("data: ".length);
        if (payload === "[DONE]") return;
        const event: unknown = JSON.parse(payload);
        if (typeof event === "object" &amp;&amp; event !== null &amp;&amp; "token" in event) {
          yield String((event as { token: unknown }).token);
        }
      }
    }
  } finally {
    reader.releaseLock();
  }
}

const sleep = (ms: number) =&gt; new Promise((resolve) =&gt; setTimeout(resolve, ms));

export async function getWithRetries&lt;T&gt;(
  path: string,
  attempts = 4,
  fetchFn: FetchFn = fetch,
  sleepFn: (ms: number) =&gt; Promise&lt;unknown&gt; = sleep,
): Promise&lt;T&gt; {
  for (let attempt = 1; attempt &lt;= attempts; attempt++) {
    const response = await fetchFn(`${BASE}${path}`, { signal: AbortSignal.timeout(5000) });

    if (response.status === 429 || response.status &gt;= 500) {
      const retryAfter = Number(response.headers.get("Retry-After"));
      const waitSeconds = Number.isFinite(retryAfter) &amp;&amp; retryAfter &gt; 0 ? retryAfter : 0.1 * 2 ** attempt;
      console.log(`Attempt ${attempt}: got ${response.status}, waiting ${waitSeconds.toFixed(1)} s`);
      await sleepFn(waitSeconds * 1000);
      continue;
    }
    if (!response.ok) {
      throw new ApiError(response.status, `${path} failed with status ${response.status}`);
    }
    return (await response.json()) as T;
  }
  throw new Error(`${path} still failing after ${attempts} attempts`);
}

async function main() {
  console.log(await ask("What is HTTP?"));

  const start = performance.now();
  let firstTokenAt: number | null = null;
  for await (const token of streamTokens("Streaming makes replies feel faster")) {
    firstTokenAt ??= (performance.now() - start) / 1000;
    process.stdout.write(token);
  }
  const total = (performance.now() - start) / 1000;
  console.log(`\nfirst token after ${firstTokenAt?.toFixed(2)} s, finished after ${total.toFixed(2)} s`);

  console.log(await getWithRetries&lt;{ status: string; calls: number }&gt;("/flaky"));

  try {
    await fetch(`${BASE}/ask`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: "{}",
    }).then(async (r) =&gt; {
      console.log("fetch resolved normally with status", r.status, "ok =", r.ok, await r.json());
    });
    await ask("");
  } catch (error) {
    if (error instanceof ApiError) {
      console.log(`ApiError: status ${error.status}`);
    } else {
      throw error;
    }
  }
}

if (import.meta.url === `file://${process.argv[1]}`) {
  await main();
}
</code></pre>
<p>Take a moment with the design. Several ideas from earlier posts are hiding in there:</p>
<ul>
<li><p><strong>Dependency injection</strong> (Post 24). Each function takes a <code>fetchFn</code> parameter that defaults to the real <code>fetch</code>. In production you ignore it. In tests you pass a fake. That one default parameter is what makes everything testable without a server. <code>getWithRetries</code> also takes a <code>sleepFn</code>, so tests don't have to wait through real delays.</p>
</li>
<li><p><strong>Validation at the edge.</strong> <code>ask</code> doesn't trust the response. It reads it as <code>unknown</code>, and a type guard checks the shape before returning <code>data.answer</code>.</p>
</li>
<li><p><strong>A typed error.</strong> <code>ApiError</code> carries the status code, so callers can react to a 422 differently from a 500. This is the custom-exception pattern from Post 11.</p>
</li>
<li><p><strong>A discriminated retry rule.</strong> 429 and 5xx are retried, honoring <code>Retry-After</code>. Other errors, like a 404, are not (Post 20).</p>
</li>
<li><p><strong>Cleanup.</strong> A <code>finally</code> block releases the stream reader even if something goes wrong.</p>
</li>
<li><p><strong>The</strong> <code>main</code> <strong>guard</strong> at the bottom runs the demo only when you run the file directly, the TypeScript counterpart of <code>if __name__ == "__main__":</code> from Post 12.</p>
</li>
</ul>
<p>Now the tests, using Node's built-in test runner, so there's nothing extra to install. Create <code>src/client.test.ts</code>:</p>
<pre><code class="language-typescript">import assert from "node:assert/strict";
import test from "node:test";

import { ApiError, ask, getWithRetries, streamTokens } from "./client.js";

function jsonResponse(status: number, body: unknown, headers: Record&lt;string, string&gt; = {}): Response {
  return new Response(JSON.stringify(body), {
    status,
    headers: { "Content-Type": "application/json", ...headers },
  });
}

// a fake fetch that returns queued responses and remembers how often it was called
function fakeFetch(...responses: Response[]) {
  const queue = [...responses];
  const calls: string[] = [];
  const fn = (async (input: string | URL | Request) =&gt; {
    calls.push(String(input));
    const next = queue.shift();
    if (!next) throw new Error("fake fetch ran out of responses");
    return next;
  }) as typeof fetch;
  return { fn, calls };
}

const noSleep = async () =&gt; {};

test("getWithRetries retries on 429 and then succeeds", async () =&gt; {
  const { fn, calls } = fakeFetch(
    jsonResponse(429, { error: "slow down" }, { "Retry-After": "1" }),
    jsonResponse(429, { error: "slow down" }, { "Retry-After": "1" }),
    jsonResponse(200, { status: "finally worked" }),
  );
  const result = await getWithRetries&lt;{ status: string }&gt;("/flaky", 4, fn, noSleep);
  assert.equal(result.status, "finally worked");
  assert.equal(calls.length, 3);
});

test("getWithRetries does not retry a 404", async () =&gt; {
  const { fn, calls } = fakeFetch(jsonResponse(404, { error: "nope" }));
  await assert.rejects(
    () =&gt; getWithRetries("/missing", 4, fn, noSleep),
    (error: unknown) =&gt; error instanceof ApiError &amp;&amp; error.status === 404,
  );
  assert.equal(calls.length, 1);
});

test("getWithRetries gives up after the last attempt", async () =&gt; {
  const { fn, calls } = fakeFetch(jsonResponse(503, {}), jsonResponse(503, {}));
  await assert.rejects(() =&gt; getWithRetries("/down", 2, fn, noSleep), /still failing after 2 attempts/);
  assert.equal(calls.length, 2);
});

test("ask returns the answer", async () =&gt; {
  const { fn } = fakeFetch(jsonResponse(200, { answer: "42" }));
  assert.equal(await ask("What is 6 x 7?", fn), "42");
});

test("ask turns an HTTP error into an ApiError", async () =&gt; {
  const { fn } = fakeFetch(jsonResponse(422, { error: "missing 'question'" }));
  await assert.rejects(
    () =&gt; ask("", fn),
    (error: unknown) =&gt; error instanceof ApiError &amp;&amp; error.status === 422,
  );
});

test("ask rejects a response with the wrong shape", async () =&gt; {
  const { fn } = fakeFetch(jsonResponse(200, { unexpected: true }));
  await assert.rejects(() =&gt; ask("hi", fn), /unexpected response shape/);
});

test("streamTokens handles chunks that split a line in half", async () =&gt; {
  const pieces = ['data: {"tok', 'en": "Hel"}\n\ndata: {"token": "lo"}\n\nda', "ta: [DONE]\n\n"];
  const encoder = new TextEncoder();
  const body = new ReadableStream&lt;Uint8Array&gt;({
    start(controller) {
      for (const piece of pieces) controller.enqueue(encoder.encode(piece));
      controller.close();
    },
  });
  const { fn } = fakeFetch(new Response(body, { status: 200 }));

  const tokens: string[] = [];
  for await (const token of streamTokens("ignored", fn)) tokens.push(token);
  assert.deepEqual(tokens, ["Hel", "lo"]);
});
</code></pre>
<p>(The <code>./client.js</code> import path looks odd, because the file is <code>client.ts</code>. That's the convention in modern TypeScript: you import the name the compiled JavaScript will have, and tools like <code>tsx</code> and <code>tsc</code> understand the mapping.)</p>
<p>The last test is the interesting one. It cuts the stream in awkward places, even in the middle of the word <code>data</code>, and checks that the parser still reassembles <code>["Hel", "lo"]</code>, with no real network involved.</p>
<h3>Running it</h3>
<p>First, start the Python server from Post 20 (<code>python server.py</code>) in one terminal. Then, in the TypeScript project:</p>
<pre><code class="language-bash">npm run check
npm test
npm start
</code></pre>
<p><code>npm run check</code> should print nothing but the command header, meaning <strong>zero type errors under strict mode</strong>. <code>npm test</code> should end with:</p>
<pre><code class="language-markdown"># tests 7
# pass 7
# fail 0
</code></pre>
<p>And <code>npm start</code> runs the client against the live server:</p>
<pre><code class="language-yaml">You asked: What is HTTP?
Streaming makes replies feel faster 
first token after 0.01 s, finished after 1.01 s
Attempt 1: got 429, waiting 1.0 s
Attempt 2: got 429, waiting 1.0 s
{ status: 'finally worked', calls: 3 }
fetch resolved normally with status 422 ok = false { error: "missing 'question'" }
ApiError: status 422
</code></pre>
<p>That matches the Python client's output in Post 20: the same fast first token, the same <code>Retry-After</code> retries, and the same 422 handling, from a different language. (The flaky endpoint only fails on its first two calls, so restart the server before running again.)</p>
<p>Finally, a sanity check on the tests themselves. I deliberately broke the retry logic, stopping it from retrying on 429, and ran the tests. One failed, naming the exact behavior I'd broken, and after I restored the code, all seven passed again. Tests that can fail are tests worth having.</p>
<h3>Make it yours</h3>
<ol>
<li><p>Replace the hand-written <code>isAnswer</code> guard with a Zod schema (<code>npm install zod</code>), and compare the two.</p>
</li>
<li><p>Add jitter (a little randomness) to the retry wait, and update the tests to keep them deterministic.</p>
</li>
<li><p>Add a <code>listCourses()</code> function with a proper <code>Course</code> type, plus validation.</p>
</li>
<li><p>Make <code>streamTokens</code> support cancellation with an <code>AbortSignal</code> that the caller can trigger.</p>
</li>
<li><p>Write a small HTML page with a text box that calls the stream and appends tokens to a <code>&lt;div&gt;</code>. You'll hit CORS (Post 20), so serve the page from the same server, or add the right headers.</p>
</li>
<li><p>Port the retry helper to also accept an <code>isRetryable</code> function, the strategy pattern from Post 24.</p>
</li>
</ol>
<h2>Practice</h2>
<p>About 20 minutes of real effort on each before asking for help.</p>
<ol>
<li><p>Translate the word counter from Post 5, <code>chunk_words</code> from Post 17, and the <code>retry</code> helper from Post 6 into TypeScript. Type-check each with <code>strict</code> on.</p>
</li>
<li><p>Write the types and a type guard for a model request and response, then write the same thing with Zod, and compare how much code each takes.</p>
</li>
<li><p>Trigger five of the compiler errors listed in this post on purpose, then fix them.</p>
</li>
<li><p>Port the <code>batches</code> generator from Post 7 into a TypeScript async generator, and consume it with <code>for await</code>.</p>
</li>
<li><p>Write <code>mapWithLimit(items, limit, fn)</code>, which runs async work with at most <code>limit</code> tasks at a time, the TypeScript equivalent of the semaphore from Post 14.</p>
</li>
<li><p>Make a file containing every JavaScript gotcha from this post (<code>==</code>, <code>[]</code> truthiness, <code>.sort()</code>, <code>||</code> versus <code>??</code>), run <code>tsc</code> on it, and see which ones TypeScript catches for you.</p>
</li>
<li><p>Add a test for the client where the server sends a malformed JSON event in the stream. Decide what the right behavior is, and implement it.</p>
</li>
<li><p>Explain in your own words why <code>as Message</code> is dangerous, and what you'd use instead.</p>
</li>
</ol>
<p>Then ask an AI tutor to review your answers, and don't let it rewrite them.</p>
<h2>What to take away</h2>
<ul>
<li><p>TypeScript is JavaScript plus types. The types are erased, so they catch mistakes at development time and do nothing at runtime.</p>
</li>
<li><p><code>tsx</code> runs your code without checking it, and <code>tsc --noEmit</code> checks without running. Use <code>strict: true</code>, and run the check often.</p>
</li>
<li><p>The JavaScript gotchas to remember: use <code>===</code>, know that <code>[]</code> is truthy, use <code>??</code> instead of <code>||</code> for defaults, pass a comparator to <code>.sort()</code>, and watch out for big numbers.</p>
</li>
<li><p>Interfaces, literal unions, optional fields, and discriminated unions let the compiler catch typos, missing fields, and unhandled cases.</p>
</li>
<li><p>Use <code>unknown</code> instead of <code>any</code>, and never cast data from outside. Validate it with a type guard or Zod.</p>
</li>
<li><p><code>async</code>/<code>await</code> and <code>Promise.all</code> work like their Python counterparts. <code>fetch</code> doesn't throw on HTTP errors, so always check <code>response.ok</code>, and give every request a timeout.</p>
</li>
<li><p>Streaming is a reader plus a buffer plus an async generator. Chunks can split lines, so buffer them.</p>
</li>
<li><p>Injecting <code>fetch</code> as a parameter makes everything testable with fakes.</p>
</li>
<li><p>Python and TypeScript work well together, with HTTP as the bridge.</p>
</li>
</ul>
<hr />
<p><strong>Previous:</strong> Post 24, <em>Clean Code &amp; Design Patterns</em></p>
<p><strong>Next in the series:</strong> Post 26, <em>Intro to C++/Rust</em>, which is optional. It covers only what you need to read the performance-critical code that sits underneath AI libraries.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/typescript-for-ai-engineers-the-language-of-ai-web-apps-2ge48?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[The 15 Git Commands That'll Save Your Sanity (and Your Code)]]></title><description><![CDATA[Most Git tutorials read like documentation. This one reads like advice from a developer who's already made every mistake in it — the 15 commands you'll…]]></description><link>https://blog.zyvop.com/the-15-git-commands-that-ll-save-your-sanity-and-your-code-7yygk</link><guid isPermaLink="true">https://blog.zyvop.com/the-15-git-commands-that-ll-save-your-sanity-and-your-code-7yygk</guid><category><![CDATA[Git]]></category><category><![CDATA[Gitcommands]]></category><category><![CDATA[gitforbeginners]]></category><category><![CDATA[GitWorkflow]]></category><category><![CDATA[versioncontrol]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Wed, 11 Nov 2026 06:43:08 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/df4714c0-21de-4080-bdf1-093115f1a7f6.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Most developers don't learn Git by reading the manual. They learn it the first time they delete something important, commit the wrong files, or discover that the branch they thought they were working on was not the branch they were actually working on.</p>
<p>That is not necessarily a bad way to learn. Git becomes much easier once you've seen what these commands are trying to protect you from.</p>
<p>The problem is that Git has accumulated a reputation for being complicated because there are dozens of commands, countless flags, and a lot of terminology that makes perfect sense once you already understand Git. When you're new to it, though, <code>add</code>, <code>commit</code>, <code>push</code>, <code>pull</code>, <code>reset</code>, <code>restore</code>, and <code>revert</code> can look like variations of the same thing.</p>
<p>They aren't.</p>
<p>This guide focuses on the commands you'll actually use in day-to-day development and, more importantly, the situations that tell you which one to reach for. You don't need to memorize every Git flag to be productive. You need a good mental model, a handful of reliable commands, and enough understanding to avoid turning a small mistake into a much bigger one.</p>
<h2>The Git Mental Model</h2>
<p>Before getting into the commands, it helps to understand the four places your code can exist during normal Git work.</p>
<p><img src="https://mermaid.ink/img/pako:eNpNzkFLw0AQBeC_Ms55cxA85SA0LUVBL03VQ-phkkySpbuZsDu11ib_XVJL8fgeH7x3xkpqxhQbJ8eqo6CwXe16AIBF8SFhb_sWtoH581pCkjyOrVWguh4hK3KldjaLwHQ12c1U4r3VEZbFi1TkYMODRKsSTle6vNHhELsRVsWGvSj_l2jQc_Bka0zPqB37-W_NDR2covlr3ilYKh3H2TTS65q8dSdMMaFhcJzEU1T2BjJn-_0rVfklr6VXAzvMuRWGt-cdGthIKSoGnth9sdqKDCyCJWcgUh-TyME2aC4juf2Zv9w_DN84TQbLdilOAqZ4d-ysMk6_2-l2Rg?type=png" alt="Mermaid Diagram" /></p>
<p>Your <strong>working tree</strong> is where you actually edit files. The <strong>staging area</strong> is where you decide which of those changes belong in your next commit. A <strong>commit</strong> records that staged state in your local repository. The <strong>remote repository</strong> — often hosted on GitHub, GitLab, or another server — is where you share those commits with other people or systems.</p>
<p>Most of the Git commands in this article are really just ways of inspecting, moving, combining, or undoing changes between those states.</p>
<p>Once that clicks, Git stops feeling like a collection of unrelated commands.</p>
<hr />
<h2>1. <code>git init</code></h2>
<p>When you're starting a project from scratch and want Git to track its history, you create a repository with <code>git init</code>.</p>
<pre><code class="language-bash">git init
</code></pre>
<p>Git creates a hidden <code>.git</code> directory inside the project. That directory is where Git keeps the information it needs to manage the repository: commits, references, configuration, and the other internal data that makes version control possible.</p>
<p>It is worth separating two ideas here because beginners often conflate them. Running <code>git init</code> creates the repository; it does <strong>not</strong> automatically add every file in the directory to version control. Your files are still untracked until you stage them.</p>
<p>A typical first setup therefore looks like this:</p>
<pre><code class="language-bash">git init
git add .
git commit -m "Initial commit"
</code></pre>
<p>You can also ask Git to create the project directory for you:</p>
<pre><code class="language-bash">git init my-new-project
</code></pre>
<p>You generally only need to initialize a repository once. When you're not sure whether you're already inside one, <code>git status</code> is a safer check than running <code>git init again</code>.</p>
<hr />
<h2>2. <code>git clone</code></h2>
<p><code>git clone</code> is the starting point for an existing project. Instead of creating a new repository, you're making a local copy of one that already exists somewhere else.</p>
<pre><code class="language-bash">git clone https://github.com/username/repository.git
</code></pre>
<p>A normal clone gives you the project's files and repository history and sets up information about the remote repository so you can fetch and push changes later. It is the command you'll usually run when joining a team, contributing to an open-source project, or moving an existing repository onto a new machine.</p>
<p>Git normally creates a directory based on the repository name, but you can choose your own:</p>
<pre><code class="language-bash">git clone https://github.com/username/repository.git my-project
</code></pre>
<p>There is one practical consideration worth knowing about. A repository with ten years of commits can be considerably larger than the current source code suggests, because you're bringing historical data along with it. When you only need a limited amount of history, a shallow clone can be useful:</p>
<pre><code class="language-bash">git clone --depth 1 https://github.com/username/repository.git
</code></pre>
<p>That gives you a much smaller starting point, although it also means you don't have the complete history available locally.</p>
<p>The simple rule is:</p>
<p><strong>Use</strong> <code>init</code> <strong>when you're creating the repository. Use</strong> <code>clone</code> <strong>when the repository already exists.</strong></p>
<hr />
<h2>3. <code>git status</code></h2>
<p>If there is one Git command worth learning so well that you stop thinking about it, it is <code>git status</code>.</p>
<pre><code class="language-bash">git status
</code></pre>
<p>When you're not sure what Git thinks is happening, ask Git. The command shows your current branch and tells you about changes that are modified, staged, or untracked.</p>
<p>A simplified example might look like this:</p>
<pre><code class="language-text">On branch main

Changes not staged for commit:
        modified:   index.html

Untracked files:
        style.css
</code></pre>
<p>In this case, <code>index.html</code> is already part of the repository but has local changes that haven't been staged. <code>style.css</code> is a new file that Git isn't tracking yet.</p>
<p><code>git status</code> is deliberately boring, and that's exactly why it is useful. It doesn't change your files or history. Run it before you commit. Run it when you've just pulled someone else's changes. Run it before a reset. Run it after you've been switching branches and can no longer remember what you were doing.</p>
<p>When Git starts feeling confusing, don't guess. Start with <code>git status</code>.</p>
<hr />
<h2>4. <code>git add</code></h2>
<p>The staging area is one of Git's most useful ideas, and also one of the first things beginners find strange.</p>
<p>Suppose you've changed three files, but only two of those changes belong in the next commit. Git gives you the option to decide exactly what gets recorded.</p>
<p>That's what <code>git add</code> does.</p>
<pre><code class="language-bash">git add filename.txt
</code></pre>
<p>You can stage everything below the current directory:</p>
<pre><code class="language-bash">git add .
</code></pre>
<p>or stage matching files:</p>
<pre><code class="language-bash">git add *.css
</code></pre>
<p>Think of staging as preparing a package before you send it. Your working tree can contain many changes, but only the staged changes are going into the next commit.</p>
<p>That distinction becomes particularly useful when you're doing unrelated work in the same project. You don't have to create a giant commit simply because several files happen to be modified at once.</p>
<p>One caution: <code>git add .</code> is convenient, but convenience can hide mistakes. It may stage files you didn't intend to include, which is why <code>.gitignore</code> should be set up early for things such as dependency directories, local environment files, generated output, and operating-system clutter.</p>
<hr />
<h2>5. <code>git commit</code></h2>
<p>A commit is a recorded snapshot of the changes you've staged.</p>
<pre><code class="language-bash">git commit -m "Add user authentication"
</code></pre>
<p>This is the point where your work becomes part of the repository's history. You can compare it later, inspect it, revert it, or use it as a known point to return to.</p>
<p>That makes commit quality matter more than many beginners realize. A repository filled with messages like <code>changes</code>, <code>fix</code>, and <code>final</code> technically works, but it is much harder to understand months later.</p>
<p>Compare:</p>
<pre><code class="language-text">Fix bug
</code></pre>
<p>with:</p>
<pre><code class="language-text">Fix null pointer when cart is empty
</code></pre>
<p>The second message tells the next person — often future you — what the commit was actually about.</p>
<p>There is also a shortcut worth knowing:</p>
<pre><code class="language-bash">git commit -am "Fix navbar spacing on mobile"
</code></pre>
<p>The <code>-a</code> option stages modifications and deletions to files Git already tracks, then creates the commit. It does <strong>not</strong> include new files, so a brand-new file still needs <code>git add</code> first.</p>
<p>And when the latest commit is almost right but not quite, <code>--amend</code> lets you update it:</p>
<pre><code class="language-bash">git commit --amend -m "Add user authentication and tests"
</code></pre>
<p>Amending is handy while a commit is still local. Once you've pushed it and other people may have based work on it, rewriting that commit becomes a different conversation.</p>
<hr />
<h2>6. <code>git branch</code></h2>
<p>Branches are how you isolate work without putting unfinished changes directly onto the branch other people may be using.</p>
<pre><code class="language-bash">git branch
</code></pre>
<p>lists your local branches.</p>
<p>To create one:</p>
<pre><code class="language-bash">git branch new-feature
</code></pre>
<p>To delete a branch that has already been safely merged:</p>
<pre><code class="language-bash">git branch -d old-feature
</code></pre>
<p>And when you know you want to remove it regardless of whether Git considers it merged:</p>
<pre><code class="language-bash">git branch -D old-feature
</code></pre>
<p>You can rename a branch with:</p>
<pre><code class="language-bash">git branch -m new-name
</code></pre>
<p>A useful thing to understand is that a branch isn't a separate copy of your entire project. It is essentially a movable reference to a commit, which is why creating branches is cheap.</p>
<p>There is one beginner mistake worth remembering: <code>git branch new-feature</code> creates the branch, but it does <strong>not</strong> switch you to it. To create and move to a branch in one step, modern Git gives you:</p>
<pre><code class="language-bash">git switch -c new-feature
</code></pre>
<p>That leads naturally to the next command family.</p>
<hr />
<h2>7. <code>git checkout</code>, <code>git switch</code>, and <code>git restore</code></h2>
<p>For years, <code>git checkout</code> handled several unrelated jobs. You could use it to switch branches, create a branch, or restore a file to an earlier state.</p>
<pre><code class="language-bash">git checkout new-feature
git checkout -b another-feature
git checkout -- filename.txt
</code></pre>
<p>The command still works, and you will continue to encounter it in older documentation and existing projects. The problem is that one command having several very different meanings makes it harder for beginners to understand what is happening.</p>
<p>Modern Git separates those operations.</p>
<p>Use <code>git switch</code> when you're working with branches:</p>
<pre><code class="language-bash">git switch new-feature
git switch -c another-feature
git switch -
</code></pre>
<p>The last form switches back to the branch you were previously on, which is surprisingly useful when you're jumping between two tasks.</p>
<p>Use <code>git restore</code> when you're working with files:</p>
<pre><code class="language-bash">git restore filename.txt
</code></pre>
<p>That discards unstaged changes to the file.</p>
<p>To remove a file from the staging area without throwing away its edits:</p>
<pre><code class="language-bash">git restore --staged filename.txt
</code></pre>
<p>So the simplest mental model is:</p>
<pre><code class="language-text">git switch   → branches
git restore  → files
git checkout → older multi-purpose command
</code></pre>
<p>You don't need to treat <code>checkout</code> as obsolete. Just recognize that <code>switch</code> and <code>restore</code> make your intention clearer when you're writing new commands.</p>
<hr />
<h2>8. <code>git merge</code></h2>
<p>Eventually, isolated work needs to come back together. That's what merging does.</p>
<p>For example:</p>
<pre><code class="language-bash">git switch main
git merge new-feature
</code></pre>
<p>If Git can reconcile the two histories automatically, the merge may be uneventful. When it can't, you'll get a conflict.</p>
<p>A conflict usually means that both sides changed overlapping parts of a file, or made changes Git cannot safely combine on its own. The repository hasn't been corrupted; Git has simply reached a decision that requires a human.</p>
<p>You'll see something like:</p>
<pre><code class="language-text">&lt;&lt;&lt;&lt;&lt;&lt;&lt; HEAD
your version
=======
their version
&gt;&gt;&gt;&gt;&gt;&gt;&gt; new-feature
</code></pre>
<p>Open the file, decide what the final version should actually be, remove the conflict markers, and then stage the resolved file:</p>
<pre><code class="language-bash">git add filename.txt
git commit
</code></pre>
<p>The first few merge conflicts can feel alarming because the file suddenly contains strange markers and Git seems to stop in the middle of the operation. In practice, conflict resolution is a normal part of collaborative development. The important thing is to understand that Git is waiting for a decision, not reporting a fatal failure.</p>
<hr />
<h2>9. <code>git push</code></h2>
<p>Your commits are local until you send them somewhere shared.</p>
<pre><code class="language-bash">git push origin main
</code></pre>
<p>That command pushes your local commits to the <code>main</code> branch on the <code>origin</code> remote.</p>
<p>When you create a new branch and push it for the first time, you will often use:</p>
<pre><code class="language-bash">git push -u origin new-feature
</code></pre>
<p>The <code>-u</code> option establishes the upstream relationship between your local branch and the remote branch. After that, a plain:</p>
<pre><code class="language-bash">git push
</code></pre>
<p>is usually enough.</p>
<p>Force-pushing is where things get serious.</p>
<pre><code class="language-bash">git push --force
</code></pre>
<p>can replace remote history, including commits you weren't aware another developer had added. There are legitimate reasons to rewrite history, particularly on personal branches after an amend or rebase, but it should never be a reflex.</p>
<p>When a force-push is genuinely necessary, this is generally safer:</p>
<pre><code class="language-bash">git push --force-with-lease
</code></pre>
<p>It checks that the remote branch is still in the state your local repository expects. If someone else has moved it since your last update, the push can be rejected instead of silently overwriting their work.</p>
<p>Even with that protection, shared branches deserve caution.</p>
<hr />
<h2>10. <code>git pull</code></h2>
<p><code>git pull</code> is the command you use when the remote repository has changes you need to bring into your current branch.</p>
<pre><code class="language-bash">git pull origin main
</code></pre>
<p>Conceptually, the operation starts by fetching updates from the remote and then integrating the relevant remote branch into your current branch. Exactly how that integration happens depends on your configuration and options.</p>
<p>A common alternative is rebasing:</p>
<pre><code class="language-bash">git pull --rebase origin main
</code></pre>
<p>Instead of creating a merge as part of the pull, Git replays your local commits on top of the updated remote history. That can produce a cleaner, more linear history, but rebasing also rewrites the commits being replayed.</p>
<p>Neither workflow is universally right. Some teams merge, some rebase, and some use stricter fast-forward policies. The important thing is consistency: know what your project expects before changing the default behavior.</p>
<p>For example:</p>
<pre><code class="language-bash">git config pull.rebase true
</code></pre>
<p>can make rebase the default for the relevant configuration scope.</p>
<p>One subtle point is worth remembering: <code>git pull origin main</code> does not necessarily mean “update my local <code>main</code> branch.” It integrates <code>origin/main</code> into <strong>whichever branch you're currently on</strong>. That distinction matters when you're working on a feature branch.</p>
<hr />
<h2>11. <code>git log</code></h2>
<p>Every Git repository eventually reaches the point where you need to know not just what the code looks like now, but how it got there.</p>
<p>That's what <code>git log</code> is for.</p>
<pre><code class="language-bash">git log
</code></pre>
<p>The default output contains commit metadata as well as the commit message, which can get noisy in a large repository. In practice, you'll often want a more compact view:</p>
<pre><code class="language-bash">git log --oneline
</code></pre>
<p>Or, when you're trying to understand branching and merges:</p>
<pre><code class="language-bash">git log --oneline --graph --all
</code></pre>
<p>A few flags cover most everyday needs:</p>
<table>
<thead>
<tr>
<th>Flag</th>
<th>What it does</th>
</tr>
</thead>
<tbody><tr>
<td>--oneline</td>
<td>Compact commit summary</td>
</tr>
<tr>
<td>--graph</td>
<td>Draw an ASCII representation of branches and merges</td>
</tr>
<tr>
<td>-p</td>
<td>Show the changes introduced by commits</td>
</tr>
<tr>
<td>--author="name"</td>
<td>Filter commits by author</td>
</tr>
<tr>
<td>-n 5</td>
<td>Show only the latest five commits</td>
</tr>
</tbody></table>
<p>The real value of <code>git log</code> isn't the list itself. It is the context. When a piece of code looks strange and nobody remembers why it exists, history can often answer the question faster than another hour of searching through the codebase.</p>
<hr />
<h2>12. <code>git diff</code></h2>
<p>Before you commit a change, you should know what you're actually about to record.</p>
<p>That's the job of <code>git diff</code>.</p>
<pre><code class="language-bash">git diff
</code></pre>
<p>A simple example:</p>
<pre><code class="language-diff">- &lt;h1&gt;Welcome&lt;/h1&gt;
+ &lt;h1&gt;Welcome to My Site&lt;/h1&gt;
</code></pre>
<p>The minus line represents the old content; the plus line represents the new content.</p>
<p>There is a second version you'll use often:</p>
<pre><code class="language-bash">git diff --staged
</code></pre>
<p>That shows what is currently in the staging area and therefore what the next commit is likely to contain.</p>
<p>You can also compare two branch tips directly:</p>
<pre><code class="language-bash">git diff branch1 branch2
</code></pre>
<p>A five-second diff check before committing is one of the best habits you can build. It catches accidental formatting changes, forgotten debugging statements, edits to the wrong file, and all the tiny mistakes that otherwise end up being discovered during code review.</p>
<hr />
<h2>13. <code>git stash</code></h2>
<p>Stash is useful when your current work is unfinished but your working tree needs to be clean.</p>
<p>Maybe a production issue needs attention. Maybe you need to inspect another branch. Maybe you simply aren't ready to create a commit yet.</p>
<p>You can temporarily put your work aside with:</p>
<pre><code class="language-bash">git stash
</code></pre>
<p>Git stores the changes and returns the working tree to a clean state.</p>
<p>You can see what you've saved with:</p>
<pre><code class="language-bash">git stash list
</code></pre>
<p>and bring the most recent stash back with:</p>
<pre><code class="language-bash">git stash pop
</code></pre>
<p>If you'd rather reapply it without removing the stash entry:</p>
<pre><code class="language-bash">git stash apply
</code></pre>
<p>And when you know you no longer need one:</p>
<pre><code class="language-bash">git stash drop
</code></pre>
<p>One detail surprises people: untracked files aren't included in a normal stash. When you need those as well, use:</p>
<pre><code class="language-bash">git stash -u
</code></pre>
<p>If you regularly have several stashes in flight, name them:</p>
<pre><code class="language-bash">git stash push -m "search filter work"
</code></pre>
<p>That small bit of discipline makes a huge difference when you come back to the stash list later and discover that every entry is just another anonymous <code>WIP</code>.</p>
<hr />
<h2>14. <code>git reset</code> and <code>git revert</code></h2>
<p>These two commands are often introduced together because both can be used to undo work. They are not interchangeable, and the difference becomes particularly important once your commits are being shared with other people.</p>
<h3><code>git reset</code></h3>
<p>Reset moves the current branch reference backward.</p>
<pre><code class="language-bash">git reset --soft HEAD~1
git reset --mixed HEAD~1
git reset --hard HEAD~1
</code></pre>
<p>The three modes mainly differ in what happens to your index and working tree:</p>
<table>
<thead>
<tr>
<th>Mode</th>
<th>Branch</th>
<th>Staging area</th>
<th>Working tree</th>
</tr>
</thead>
<tbody><tr>
<td>--soft</td>
<td>Moves</td>
<td>Keeps changes staged</td>
<td>Keeps changes</td>
</tr>
<tr>
<td>--mixed</td>
<td>Moves</td>
<td>Unstages changes</td>
<td>Keeps changes</td>
</tr>
<tr>
<td>--hard</td>
<td>Moves</td>
<td>Resets</td>
<td>Resets</td>
</tr>
</tbody></table>
<p><code>--mixed</code> is the default.</p>
<p>The command that deserves the most respect is:</p>
<pre><code class="language-bash">git reset --hard HEAD~1
</code></pre>
<p>It can discard changes from tracked files in your working tree. If you haven't checked what Git thinks is going to happen, this is a very bad time to start experimenting.</p>
<p>The other important consideration is whether the commit has already been shared.</p>
<p>If the commit exists only locally, resetting can be perfectly reasonable. If you've already pushed it and other people may have based their work on it, moving the branch backward creates a history rewrite that affects everyone downstream.</p>
<h3><code>git revert</code></h3>
<p>For an already-shared commit, <code>git revert</code> is often the safer choice:</p>
<pre><code class="language-bash">git revert HEAD
</code></pre>
<p>Rather than moving the branch backward, it creates a <strong>new commit</strong> that reverses the earlier change.</p>
<p>The easiest way to remember the distinction is:</p>
<pre><code class="language-text">reset   → move the branch backward
revert  → add a new commit that undoes an earlier one
</code></pre>
<p>Git also keeps a reflog locally, which can sometimes help recover from mistakes involving commands such as <code>reset</code>. That's useful to know, but it shouldn't be treated as permission to run destructive commands carelessly.</p>
<hr />
<h2>15. <code>git remote</code></h2>
<p>A Git repository can work entirely locally, but most development eventually involves a remote repository somewhere else. <code>git remote</code> lets you inspect and manage those connections.</p>
<p>To see what is configured:</p>
<pre><code class="language-bash">git remote -v
</code></pre>
<p>To add a remote:</p>
<pre><code class="language-bash">git remote add origin https://github.com/user/repo.git
</code></pre>
<p>To remove one:</p>
<pre><code class="language-bash">git remote remove origin
</code></pre>
<p>To rename one:</p>
<pre><code class="language-bash">git remote rename origin upstream
</code></pre>
<p><code>origin</code> is a convention, not a special Git command or keyword. When you clone a repository, Git normally names the remote <code>origin</code> simply because that is the conventional default.</p>
<p>Fork-based workflows often use another common convention:</p>
<pre><code class="language-text">origin   → your fork
upstream → original repository
</code></pre>
<p>That arrangement lets you push your own work to <code>origin</code> while fetching updates from the original project through <code>upstream</code>.</p>
<hr />
<h1>Five Habits That Make Git Much Easier</h1>
<p>Knowing commands is useful. Knowing when to stop and inspect the repository is even more useful.</p>
<h3>Check before you change things</h3>
<p>When you're uncertain, start here:</p>
<pre><code class="language-bash">git status
git diff
</code></pre>
<p>Those two commands answer a remarkable number of “what just happened?” questions without changing anything themselves.</p>
<h3>Keep commits focused</h3>
<p>Small commits are easier to review and easier to undo. They also make <code>git log</code> useful instead of turning it into a museum of commits named <code>final</code>, <code>final2</code>, and <code>really-final-this-time</code>.</p>
<h3>Use <code>.gitignore</code> early</h3>
<p>A typical project might ignore things such as:</p>
<pre><code class="language-text">node_modules/
.env
.DS_Store
dist/
</code></pre>
<p>Just remember that <code>.gitignore</code> doesn't remove a file that has already been committed. If a credential or API key has already been pushed to a remote repository, assume it has been exposed and rotate it.</p>
<h3>Treat force-push as an exception</h3>
<p>Before using:</p>
<pre><code class="language-bash">git push --force
</code></pre>
<p>ask yourself whether anyone else could have based work on the remote branch.</p>
<p>On a private feature branch, rewriting history may be perfectly reasonable. On a shared branch, it can create a mess for everyone.</p>
<h3>Practice recovery, not memorization</h3>
<p>The fastest way to understand Git is to use it somewhere safe.</p>
<p>Create a throwaway repository. Make a few commits. Create a branch. Modify a file. Stash it. Undo something. Create a merge conflict and resolve it.</p>
<p>Once you've recovered from a mistake in a repository you don't care about, the same commands feel much less intimidating in a repository you do.</p>
<hr />
<h1>Git Commands by Situation</h1>
<p>This is the part worth bookmarking because it answers the question beginners actually have: <strong>“What do I type now?”</strong></p>
<table>
<thead>
<tr>
<th>Situation</th>
<th>Reach for</th>
</tr>
</thead>
<tbody><tr>
<td>Starting a project</td>
<td>git init</td>
</tr>
<tr>
<td>Getting an existing project</td>
<td>git clone</td>
</tr>
<tr>
<td>Not sure what's happening</td>
<td>git status</td>
</tr>
<tr>
<td>Choosing what belongs in the next commit</td>
<td>git add</td>
</tr>
<tr>
<td>Saving a checkpoint</td>
<td>git commit</td>
</tr>
<tr>
<td>Starting isolated work</td>
<td>git branch / git switch</td>
</tr>
<tr>
<td>Moving between branches</td>
<td>git switch</td>
</tr>
<tr>
<td>Throwing away a local file change</td>
<td>git restore</td>
</tr>
<tr>
<td>Combining branch histories</td>
<td>git merge</td>
</tr>
<tr>
<td>Sharing your commits</td>
<td>git push</td>
</tr>
<tr>
<td>Getting remote changes</td>
<td>git pull</td>
</tr>
<tr>
<td>Investigating history</td>
<td>git log</td>
</tr>
<tr>
<td>Inspecting exactly what changed</td>
<td>git diff</td>
</tr>
<tr>
<td>Temporarily setting work aside</td>
<td>git stash</td>
</tr>
<tr>
<td>Undoing an unshared commit</td>
<td>git reset</td>
</tr>
<tr>
<td>Undoing a shared commit</td>
<td>git revert</td>
</tr>
<tr>
<td>Checking repository connections</td>
<td>git remote</td>
</tr>
</tbody></table>
<hr />
<h1>Quick-Reference Cheat Sheet</h1>
<table>
<thead>
<tr>
<th>#</th>
<th>Command</th>
<th>What it does</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>git init</td>
<td>Create a repository</td>
</tr>
<tr>
<td>2</td>
<td>git clone</td>
<td>Copy an existing repository</td>
</tr>
<tr>
<td>3</td>
<td>git status</td>
<td>Inspect repository state</td>
</tr>
<tr>
<td>4</td>
<td>git add</td>
<td>Stage changes</td>
</tr>
<tr>
<td>5</td>
<td>git commit</td>
<td>Record a snapshot</td>
</tr>
<tr>
<td>6</td>
<td>git branch</td>
<td>Manage branches</td>
</tr>
<tr>
<td>7</td>
<td>git checkout / git switch / git restore</td>
<td>Switch branches or restore files</td>
</tr>
<tr>
<td>8</td>
<td>git merge</td>
<td>Combine branch histories</td>
</tr>
<tr>
<td>9</td>
<td>git push</td>
<td>Send commits to a remote</td>
</tr>
<tr>
<td>10</td>
<td>git pull</td>
<td>Fetch and integrate remote changes</td>
</tr>
<tr>
<td>11</td>
<td>git log</td>
<td>Inspect commit history</td>
</tr>
<tr>
<td>12</td>
<td>git diff</td>
<td>Compare changes</td>
</tr>
<tr>
<td>13</td>
<td>git stash</td>
<td>Temporarily store unfinished work</td>
</tr>
<tr>
<td>14</td>
<td>git reset / git revert</td>
<td>Undo changes in different ways</td>
</tr>
<tr>
<td>15</td>
<td>git remote</td>
<td>Manage remote connections</td>
</tr>
</tbody></table>
<h1>Where to Go From Here</h1>
<p>You don't need to memorize all fifteen commands before you start building things.</p>
<p>For most everyday work, you'll spend a surprising amount of time with just:</p>
<pre><code class="language-bash">git status
git add
git commit
git push
git pull
</code></pre>
<p>Branching becomes important as soon as you're working on multiple features or collaborating with other developers. <code>stash</code>, <code>reset</code>, and <code>revert</code> become important when something gets messy.</p>
<p>That's actually the better way to learn Git: let the situations teach you the commands instead of trying to memorize the entire manual up front.</p>
<p>And when you do get into trouble, remember the most useful Git habit of all:</p>
<p><strong>Don't guess what Git is doing. Inspect the state first.</strong></p>
<pre><code class="language-bash">git status
git diff
</code></pre>
<p>Git isn't really about memorizing commands.</p>
<p>It's about knowing where your changes are, understanding what you want to happen next, and choosing the command that gets you there without making the problem worse.</p>
<p>Once that mental model becomes second nature, Git stops feeling mysterious.</p>
<p>It becomes a safety net.</p>
<p>And that's what it was supposed to be all along.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/the-15-git-commands-that-ll-save-your-sanity-and-your-code-7yygk?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Webhook Security: Verify Signatures, Stop Replays, Handle Retries]]></title><description><![CDATA[Anyone who finds your webhook URL can post to it. Here's how to verify HMAC signatures, stop replayed requests, and handle duplicate and retried…]]></description><link>https://blog.zyvop.com/webhook-security-verify-signatures-stop-replays-handle-retries-x05wj</link><guid isPermaLink="true">https://blog.zyvop.com/webhook-security-verify-signatures-stop-replays-handle-retries-x05wj</guid><category><![CDATA[APIDesign]]></category><category><![CDATA[FastAPI]]></category><category><![CDATA[Python]]></category><category><![CDATA[Security]]></category><category><![CDATA[webhooks]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Wed, 11 Nov 2026 05:43:50 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/aa753dcf-cdcc-499f-8dc2-b35091186129.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A webhook endpoint is a public URL that your server trusts to make things happen. A payment provider posts to it, and you credit an account. A Git host posts to it, and you deploy.</p>
<p>The URL isn't secret for long. It shows up in logs, dashboards and screenshots. And anyone who knows it can send it a request.</p>
<p>That leaves three questions you have to answer on every delivery:</p>
<ol>
<li><p><strong>Is this really from the sender?</strong> That's forgery, and signatures answer it.</p>
</li>
<li><p><strong>Is someone replaying a real request they captured?</strong> That's replay, and timestamps plus event IDs answer it.</p>
</li>
<li><p><strong>Is the sender delivering the same event again, late, or out of order?</strong> That's retries, and a fast acknowledgement plus an idempotent handler answer it.</p>
</li>
</ol>
<p>This post builds the receiving side in Python. The signature code is checked against test vectors that GitHub and the Standard Webhooks spec publish, so you can see that it computes what the real senders compute.</p>
<h2>What you're defending against</h2>
<table>
<thead>
<tr>
<th>Attack or problem</th>
<th>What happens</th>
<th>The defense</th>
</tr>
</thead>
<tbody><tr>
<td>Forgery</td>
<td>Someone posts a made-up event</td>
<td>Verify an HMAC signature made with a shared secret</td>
</tr>
<tr>
<td>Tampering</td>
<td>Someone changes a real request on the way</td>
<td>The signature covers the exact body</td>
</tr>
<tr>
<td>Replay</td>
<td>Someone re-sends a captured, validly signed request</td>
<td>A timestamp window, plus event ID deduplication</td>
</tr>
<tr>
<td>Timing attack</td>
<td>Guessing a signature one byte at a time from response times</td>
<td>Constant-time comparison</td>
</tr>
<tr>
<td>Resource exhaustion</td>
<td>Huge bodies, or floods of junk</td>
<td>A size limit before any crypto, plus rate limits at the edge</td>
</tr>
<tr>
<td>A leaked secret</td>
<td>The shared secret escapes</td>
<td>Rotation, so you accept the old and new secret for a while</td>
</tr>
</tbody></table>
<p>And here's the pipeline every request will pass through:</p>
<p><img src="https://mermaid.ink/img/pako:eNpdkctu2zAQRX9lOmsKttOsjCJF_GqyKFpUaRHA9GIkjiwiFEcgabuq4n8v5BQx1OU87j2XnB5LMYxzrJycyppCgqeV9gAA91uN37_lTzA5cVGLvMRJS13DPkWNO8iyO1j0GhdiOkgi4Cjs-bPG85t8MWy8auw4anyF59lW4-3so8bdeO5lGC97jQ9MhkOENnBkn4C8-VSEyV2yDcdETQvWR2sYUs1wst7I6YpbjuyebwbadPpOW47TrHqNud17SofAcCRnDVQSLjjxDFKBHAJELgOneKWs_qOMu__M173G9XF4weMKyAUm00FMEthcndZjzWar8WY6BXNonS0p8Xvy9Qj5ZasxH6zAJjV8hIdBRmXJbWKjcYcKGw4NWYPzHlPNzXBewxUdXEL11vlFwVLhOA47lfi0oca6DueYUds6zmIXEzcKFs76l69U5pd6Iz4p0JjzXhh-PmpU8EMKSaLggd2Rky1JwX2w5BRE8jGLHGyF6gLJ7Z8hy-y2_Y3ns8JivxQnAef44VTbxHj-C4Ch0VA?type=png" alt="Mermaid Diagram" /></p>
<h2>How signing works</h2>
<p>The sender and you share a secret. For each delivery, the sender computes an <strong>HMAC</strong> of the request using that secret, and sends the result in a header. HMAC-SHA256 is a keyed hash. Only someone holding the secret can produce the right value for a given body.</p>
<p>You recompute it from what you received and compare. If the two match, the sender knew the secret and the body wasn't changed.</p>
<p>The idea is the same everywhere, but the details differ:</p>
<table>
<thead>
<tr>
<th></th>
<th>GitHub</th>
<th>Stripe</th>
<th>Standard Webhooks</th>
</tr>
</thead>
<tbody><tr>
<td>Header</td>
<td>X-Hub-Signature-256</td>
<td>Stripe-Signature</td>
<td>webhook-signature</td>
</tr>
<tr>
<td>What's signed</td>
<td>The body</td>
<td>timestamp.body</td>
<td>id.timestamp.body</td>
</tr>
<tr>
<td>Format</td>
<td>sha256= and hex</td>
<td>t=...,v1=... with hex</td>
<td>v1, and base64</td>
</tr>
<tr>
<td>Key</td>
<td>The secret as text</td>
<td>The whsec_... secret as text</td>
<td>The base64 part of whsec_..., decoded</td>
</tr>
<tr>
<td>Timestamp</td>
<td>None</td>
<td>In the header and signed</td>
<td>Its own header, and signed</td>
</tr>
<tr>
<td>Where's the event ID?</td>
<td>X-GitHub-Delivery header</td>
<td>id in the JSON body</td>
<td>webhook-id header, and signed</td>
</tr>
</tbody></table>
<p>Standard Webhooks is an open spec that several services follow. We'll build on it, because it signs the ID and the timestamp as well as the body. That gives you replay protection almost for free. GitHub signs only the body, so with GitHub the delivery ID does all the replay work.</p>
<h2>Step 1: Verify the signature</h2>
<p>Create <code>signing.py</code>:</p>
<pre><code class="language-python">import base64
import hashlib
import hmac
from dataclasses import dataclass

class InvalidWebhook(Exception):
    """The request can't be trusted. `reason` is for your logs, not for the sender."""

    def __init__(self, reason):
        super().__init__(reason)
        self.reason = reason

@dataclass(frozen=True)
class Verified:
    message_id: str
    timestamp: int

def _lower(headers):
    return {str(name).lower(): value for name, value in headers.items()}

def _equal(a: str, b: str) -&gt; bool:
    """Constant-time comparison. Bytes, so a non-ASCII header can't raise a TypeError."""
    return hmac.compare_digest(a.encode("utf-8", "replace"), b.encode("utf-8", "replace"))

def _fresh(timestamp: str, now: float, tolerance: int) -&gt; int:
    try:
        seconds = int(timestamp)
    except ValueError:
        raise InvalidWebhook("bad-timestamp") from None
    if abs(now - seconds) &gt; tolerance:  # too old (a replay) or too far in the future
        raise InvalidWebhook("stale-timestamp")
    return seconds

# ---- Standard Webhooks: webhook-id, webhook-timestamp, webhook-signature --------------------

def standard_signature(secret: str, message_id: str, timestamp: str, body: bytes) -&gt; str:
    key = base64.b64decode(secret.removeprefix("whsec_"))
    signed = f"{message_id}.{timestamp}.".encode() + body  # id.timestamp.raw-body
    return "v1," + base64.b64encode(hmac.new(key, signed, hashlib.sha256).digest()).decode()

def verify_standard(secrets, headers, body: bytes, now: float, tolerance: int = 300) -&gt; Verified:
    """`secrets` is a list, so you can accept the old and new secret while rotating."""
    h = _lower(headers)
    try:
        message_id, timestamp, signatures = h["webhook-id"], h["webhook-timestamp"], h["webhook-signature"]
    except KeyError:
        raise InvalidWebhook("missing-headers") from None
    seconds = _fresh(timestamp, now, tolerance)  # the cheap check goes first
    for secret in secrets:
        expected = standard_signature(secret, message_id, timestamp, body)
        for candidate in signatures.split():  # the header can hold several: "v1,aaa v1,bbb"
            if candidate.startswith("v1,") and _equal(candidate, expected):
                return Verified(message_id, seconds)
    raise InvalidWebhook("bad-signature")

# ---- GitHub: X-Hub-Signature-256 (no timestamp, so dedupe on X-GitHub-Delivery) -------------

def verify_github(secret: str, headers, body: bytes) -&gt; None:
    given = _lower(headers).get("x-hub-signature-256")
    if given is None:
        raise InvalidWebhook("missing-headers")
    expected = "sha256=" + hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    if not _equal(given, expected):
        raise InvalidWebhook("bad-signature")

# ---- Stripe: Stripe-Signature: t=...,v1=...,v0=... ------------------------------------------

def verify_stripe(secret: str, header: str, body: bytes, now: float, tolerance: int = 300) -&gt; int:
    parts = [item.split("=", 1) for item in header.split(",") if "=" in item]
    timestamps = [value for key, value in parts if key.strip() == "t"]
    signatures = [value for key, value in parts if key.strip() == "v1"]  # ignore v0 and anything else
    if not timestamps or not signatures:
        raise InvalidWebhook("missing-headers")
    seconds = _fresh(timestamps[0], now, tolerance)
    expected = hmac.new(secret.encode(), f"{timestamps[0]}.".encode() + body, hashlib.sha256).hexdigest()
    if not any(_equal(signature, expected) for signature in signatures):
        raise InvalidWebhook("bad-signature")
    return seconds
</code></pre>
<p>All three schemes are in there. Each one raises <code>InvalidWebhook</code> with a reason, and the reason is for your logs.</p>
<p>The code follows six rules. Each one stops a real mistake.</p>
<h3>1. Verify the raw bytes</h3>
<p>The signature covers the exact bytes the sender transmitted. If your framework parses the JSON and you serialize it again, the bytes can change. Whitespace, key order and number formatting are all fair game.</p>
<p><code>test_re_serialised_json_breaks_the_signature</code> shows it. A body with two spaces in it verifies fine. Parse it and dump it again, and the signature no longer matches.</p>
<p>In FastAPI, take the <code>Request</code> and call <code>await request.body()</code>. Don't declare a Pydantic body parameter, because that parses the body before you ever see the raw bytes.</p>
<h3>2. Compare in constant time</h3>
<p><code>==</code> stops at the first difference. Given enough tries, an attacker can use response time to find a valid signature byte by byte. <code>hmac.compare_digest</code> is designed to prevent that. The Python docs recommend it for exactly this.</p>
<p>One detail: both arguments must be the same type, and a <code>str</code> must be ASCII-only. A non-ASCII character in a header would raise a <code>TypeError</code>, and your endpoint would return a 500 instead of a rejection. So the code compares bytes. <code>test_a_non_ascii_signature_is_rejected_not_a_crash</code> covers it.</p>
<h3>3. Check the timestamp, in both directions</h3>
<p>A request that's minutes old is suspicious, and so is one dated in the future. The window here is five minutes either way, which is Stripe's default and what Standard Webhooks recommends. Stripe warns against setting the tolerance to zero, and says to keep your server's clock synced with NTP.</p>
<p>The cheap timestamp check runs before the HMAC.</p>
<h3>4. Accept more than one secret</h3>
<p>When you rotate a secret, there's a gap where the sender uses the old one and you've deployed the new one, or the other way round. Stripe handles this by sending a signature for each active secret. A header can hold several signatures, and one valid match is enough.</p>
<p>So <code>verify_standard</code> takes a list of secrets, and it tries every signature in the header. Once the old secret is retired, take it off the list. <code>test_secret_rotation_accepts_old_and_new_and_ignores_unknown_versions</code> checks both sides.</p>
<h3>5. Trust only the version you know</h3>
<p>Signature headers carry a version label, like <code>v1</code>. Ignore any scheme you don't recognize. Stripe also sends a <code>v0</code> signature for test events, and says never to trust it for live traffic.</p>
<h3>6. Say nothing useful to attackers</h3>
<p>Every failure gets the same <code>400</code> and the same body. The reason (bad signature, stale timestamp, missing headers) goes to your logs. Otherwise your error messages become a guide for working out which check someone is failing.</p>
<h2>Checked against the real thing</h2>
<p>A verifier that agrees with itself proves little. So the tests check two published vectors.</p>
<p>GitHub's docs publish a secret (<code>It's a Secret to Everybody</code>), a payload (<code>Hello, World!</code>) and the signature they should produce. The Standard Webhooks spec publishes a secret, message ID, timestamp, payload and signature. The code in this post reproduces both signatures exactly.</p>
<p>You can see it in <code>test_github_matches_its_published_vector</code> and <code>test_standard_webhooks_matches_its_published_vector</code>.</p>
<h2>Step 2: Stop replays</h2>
<p>Say an attacker captures a real, correctly signed request. They can't forge a new one, but they can send the captured one again. If it credited an account once, will it credit it twice?</p>
<p>Two layers answer that, and they cover different things.</p>
<p><strong>The timestamp window limits how long a captured request stays useful.</strong> After five minutes it's rejected. This only works if the timestamp is inside the signed content, because otherwise the attacker just edits it. Stripe and Standard Webhooks sign it. GitHub's signature has no timestamp at all.</p>
<p><strong>Deduplicating event IDs makes a replay inside the window harmless.</strong> If the ID has been seen, the request is acknowledged and nothing else happens.</p>
<p><code>test_a_replay_is_harmless_inside_the_window_and_rejected_after_it</code> shows both. The same bytes sent 100 seconds later are acknowledged as a duplicate. Sent 400 seconds after that, they're rejected as stale.</p>
<p>The two windows are not the same length. The timestamp window defends against attackers, so it's short. The ID window defends against the sender's own retries, so it has to last as long as the sender can redeliver. Stripe retries for up to three days in live mode. Keep IDs at least that long.</p>
<h2>Step 3: Handle retries</h2>
<p>Senders retry, and not always the way you'd expect:</p>
<table>
<thead>
<tr>
<th>Sender</th>
<th>When it retries</th>
<th>Notes</th>
</tr>
</thead>
<tbody><tr>
<td>Stripe</td>
<td>After any response that isn't a 2xx, including redirects, and after timeouts. In live mode, for up to three days with exponential backoff</td>
<td>Each attempt gets a new timestamp and signature</td>
</tr>
<tr>
<td>GitHub</td>
<td>It doesn't redeliver failed deliveries on its own. You redeliver them from the UI or the API</td>
<td>You must answer with a 2xx within 10 seconds. A redelivery keeps the same X-GitHub-Delivery ID</td>
</tr>
<tr>
<td>Standard Webhooks senders</td>
<td>Typically exponential backoff with jitter</td>
<td>The webhook-id stays the same across attempts, while the timestamp and signature are fresh</td>
</tr>
</tbody></table>
<p>Two things follow.</p>
<p>First, a retry is a new request with a valid signature, a fresh timestamp and an ID you've already seen. The replay defenses above will accept it. That's right, because the dedupe layer turns it into a harmless duplicate.</p>
<p>Second, the sender is probably waiting on you. A slow handler leads to a timeout, which leads to a retry, which leads to more load on the slow handler.</p>
<h3>Acknowledge fast, process later</h3>
<p>So do as little as possible in the request. Verify, store, answer:</p>
<pre><code class="language-python">import logging
import time

from signing import InvalidWebhook, verify_standard

log = logging.getLogger("webhooks")

class WebhookReceiver:
    """Verify, store, answer. Nothing slow happens here."""

    def __init__(self, inbox, secrets, clock=time.time, tolerance=300, max_body=1_000_000, on_reject=None):
        self.inbox, self.secrets, self.clock = inbox, secrets, clock
        self.tolerance, self.max_body = tolerance, max_body
        self.on_reject = on_reject or (lambda reason: log.warning("rejected webhook: %s", reason))

    def handle(self, headers, body: bytes):
        """Returns (status code, JSON body)."""
        if len(body) &gt; self.max_body:
            return 413, {"error": "payload too large"}  # refuse before spending CPU on an HMAC
        try:
            verified = verify_standard(self.secrets, headers, body, self.clock(), self.tolerance)
        except InvalidWebhook as error:
            self.on_reject(error.reason)  # the reason goes to your logs...
            return 400, {"error": "invalid webhook"}  # ...and the sender learns nothing
        is_new = self.inbox.store("standard", verified.message_id, body)
        return 200, {"status": "accepted" if is_new else "duplicate"}  # a duplicate is still a success
</code></pre>
<p>The handler never runs here. The event goes into an inbox table, and a worker picks it up afterwards:</p>
<p><img src="https://mermaid.ink/img/pako:eNp1kluL2zAQhf_KqZ4SUCC79MkPgV5YmodeiNuGQqBMrEkyRJZcaezEXfa_F8cNy5L2UaNvjs4czaOpomNTmMy_Wg4VvxfaJ6o3AQAaSiqVNBQUJSij5OA43V6uhssfsU1IXLF0_2KWA7MM23jGxJHSljJPb7H1gK1jOl41ytlisSrw5XP5FdxxUEzEgTv9efe3fTUSHSfZ9ciyD6RtYovqwNURKjVnpbp5ppcFssbEo8y1PlssygL38zmoqrhRdhYSUIv3krmKweURXY8Snikz9MAIfFa4lkeDz9C6QGoD-iGZAwXnr0N9isqIHSesC8gOotiR-GyRWFMPT8oJJ9EDCPsUTxL2cOypf-GgpnQcel0M_CKtUSXuxgExGUzmy-ehlpzZXWzfz-fT_4WCCfnE5PqBTDy9Dcm1jZeKlI01NaeaxJni0eiB62GjHO-o9WrsWPlOSWjrOQ_MLgZ9oFp8bwozo6bxPMt9Vq4t3noJx49UlZfzQwxqsTEl7yPj23JjLFZxGzVafGDfsUpFFm-SkLfIFPIsD3tg7OWRUn4PXu5eN2fz9GTNdv8u-phMYV6dDqJsnv4AW1z7vA?type=png" alt="Mermaid Diagram" /></p>
<p>What to return in each case:</p>
<table>
<thead>
<tr>
<th>Situation</th>
<th>Status</th>
<th>Why</th>
</tr>
</thead>
<tbody><tr>
<td>Valid, new</td>
<td>200</td>
<td>You've stored it, so you've got it</td>
</tr>
<tr>
<td>Valid, already seen</td>
<td>200</td>
<td>A duplicate isn't an error. A non-2xx here would just make the sender retry again</td>
</tr>
<tr>
<td>Bad signature, stale timestamp, missing headers</td>
<td>400</td>
<td>The same answer for all of them</td>
</tr>
<tr>
<td>Body too large</td>
<td>413</td>
<td>Refused before any crypto</td>
</tr>
<tr>
<td>You couldn't store it</td>
<td>500</td>
<td>The one time you want a retry</td>
</tr>
<tr>
<td>An event type you don't handle</td>
<td>200</td>
<td>Acknowledge and ignore. Don't make the sender retry something you'll never process</td>
</tr>
</tbody></table>
<h3>The inbox</h3>
<p>Here's the table behind it:</p>
<pre><code class="language-python">import random
import sqlite3
import time
from contextlib import closing
from dataclasses import dataclass

@dataclass
class Job:
    source: str
    event_id: str
    body: bytes
    attempts: int  # which attempt this is, starting at 1

class Inbox:
    """Every verified delivery lands here first. The primary key makes duplicates impossible."""

    def __init__(self, path, lease=60, max_attempts=6, clock=time.time, rng=random.random):
        self.path, self.lease, self.max_attempts, self.clock, self.rng = path, lease, max_attempts, clock, rng
        with closing(self._connect()) as db:
            db.execute(
                """CREATE TABLE IF NOT EXISTS inbox (
                       source          TEXT NOT NULL,
                       event_id        TEXT NOT NULL,
                       body            BLOB NOT NULL,
                       status          TEXT NOT NULL,       -- pending, done or dead
                       attempts        INTEGER NOT NULL,
                       next_attempt_at REAL NOT NULL,
                       last_error      TEXT,
                       received_at     REAL NOT NULL,
                       PRIMARY KEY (source, event_id)
                   )"""
            )

    def _connect(self):
        return sqlite3.connect(self.path, timeout=10, isolation_level=None)

    def store(self, source: str, event_id: str, body: bytes) -&gt; bool:
        """True for a new event, False for a duplicate."""
        now = self.clock()
        with closing(self._connect()) as db:
            return db.execute(
                "INSERT OR IGNORE INTO inbox VALUES (?, ?, ?, 'pending', 0, ?, NULL, ?)",
                (source, event_id, body, now, now),
            ).rowcount == 1

    def next_job(self):
        """Lease the next due job. If the worker dies, the lease runs out and it comes back."""
        now = self.clock()
        with closing(self._connect()) as db:
            db.execute("BEGIN IMMEDIATE")  # one worker at a time can take a job
            row = db.execute(
                "SELECT source, event_id, body, attempts FROM inbox"
                " WHERE status = 'pending' AND next_attempt_at &lt;= ? ORDER BY next_attempt_at LIMIT 1",
                (now,),
            ).fetchone()
            if row:
                db.execute(
                    "UPDATE inbox SET attempts = attempts + 1, next_attempt_at = ? WHERE source = ? AND event_id = ?",
                    (now + self.lease, row[0], row[1]),
                )
            db.execute("COMMIT")
        return Job(row[0], row[1], row[2], row[3] + 1) if row else None

    def _update(self, job, status, next_attempt_at, error=None):
        with closing(self._connect()) as db:
            db.execute(
                "UPDATE inbox SET status = ?, next_attempt_at = ?, last_error = ? WHERE source = ? AND event_id = ?",
                (status, next_attempt_at, error, job.source, job.event_id),
            )

    def done(self, job):
        self._update(job, "done", 0)

    def dead(self, job, error):
        self._update(job, "dead", 0, error)

    def fail(self, job, error):
        if job.attempts &gt;= self.max_attempts:
            return self.dead(job, error)  # give up, and keep it for a human to look at
        self._update(job, "pending", self.clock() + self.backoff(job.attempts), error)

    def backoff(self, attempt: int) -&gt; float:
        delay = min(3600, 30 * 2 ** (attempt - 1))  # 30s, 1m, 2m, 4m ... capped at an hour
        return delay / 2 + delay / 2 * self.rng()  # jitter, so retries don't all line up

    def counts(self) -&gt; dict:
        with closing(self._connect()) as db:
            return dict(db.execute("SELECT status, COUNT(*) FROM inbox GROUP BY status"))
</code></pre>
<p>It does two jobs at once.</p>
<p><strong>Deduplication.</strong> <code>(source, event_id)</code> is the primary key, and <code>INSERT OR IGNORE</code> reports whether a row was actually inserted. The database decides who was first, so eight simultaneous copies of one delivery produce one stored event. <code>test_eight_simultaneous_copies_are_stored_once</code> fires them from eight threads at once.</p>
<p><strong>A work queue.</strong> <code>next_job</code> leases the next due event inside a transaction, so two workers never take the same one. If a worker dies mid-job, the lease runs out and the event comes back.</p>
<h3>The worker</h3>
<pre><code class="language-python">import json

class Worker:
    """Takes events out of the inbox and runs your handlers, retrying failures."""

    def __init__(self, inbox, handlers):
        self.inbox, self.handlers = inbox, handlers

    def run_once(self) -&gt; bool:
        job = self.inbox.next_job()
        if job is None:
            return False
        try:
            event = json.loads(job.body)
            kind = event["type"]
        except (ValueError, KeyError, TypeError) as error:
            self.inbox.dead(job, f"unreadable event: {error!r}")  # retrying can't fix a bad payload
            return True
        try:
            handler = self.handlers.get(kind)
            if handler is not None:  # events you don't handle are acknowledged and ignored
                handler(event)
        except Exception as error:
            self.inbox.fail(job, repr(error))  # retried later, with a growing delay
        else:
            self.inbox.done(job)
        return True
</code></pre>
<p>A failing handler isn't a reason to lose the event. It's retried with a growing delay, 30 seconds, then a minute, then two, capped at an hour, with some jitter so retries don't line up. After <code>max_attempts</code> it's set aside as a dead letter, and it stays in the table for a human to look at.</p>
<p>An unreadable payload goes straight to the dead letter pile, because retrying can't make bad JSON valid.</p>
<p>One setting to get right is the lease, which defaults to 60 seconds. Make it longer than your slowest handler. If it's shorter, a second worker can pick up the event while the first is still busy, and your handler runs twice at the same time.</p>
<h3>Make the handler idempotent</h3>
<p>Everything so far delivers each event <em>at least once</em>, never <em>exactly once</em>. The receiver dedupes, but a worker can still crash after your handler has done its work and before it marks the event done. When the lease expires, the handler runs again.</p>
<p>So the handler has to be safe to run twice. The simplest way is to record the event ID with the effect, and let the database refuse the second one:</p>
<pre><code class="language-python">import sqlite3
from contextlib import closing

class Ledger:
    """An idempotent handler: an account is credited once per event, however often it's delivered."""

    def __init__(self, path):
        self.path = path
        with closing(self._connect()) as db:
            db.execute("CREATE TABLE IF NOT EXISTS credits (event_id TEXT PRIMARY KEY, account TEXT NOT NULL, cents INTEGER NOT NULL)")

    def _connect(self):
        return sqlite3.connect(self.path, timeout=10, isolation_level=None)

    def credit(self, event_id: str, account: str, cents: int) -&gt; bool:
        with closing(self._connect()) as db:
            return db.execute("INSERT OR IGNORE INTO credits VALUES (?, ?, ?)", (event_id, account, cents)).rowcount == 1

    def balance(self, account: str) -&gt; int:
        with closing(self._connect()) as db:
            return db.execute("SELECT COALESCE(SUM(cents), 0) FROM credits WHERE account = ?", (account,)).fetchone()[0]
</code></pre>
<p><code>test_a_crash_after_the_side_effect_is_recovered_and_credits_once</code> plays out the awkward case. A worker leases an event and credits the account, then dies. The lease expires, the event runs again, and the balance is still 700 cents, not 1,400.</p>
<p>It's the same idea as idempotency keys on an API: let a unique constraint decide whether the work has already happened.</p>
<h3>Don't rely on order</h3>
<p>Stripe's docs say plainly that events can arrive out of order, and not to use the <code>created</code> timestamp to order them. Write handlers that don't care. For anything important, treat the event as a signal to go and look. Fetch the object's current state from the provider's API, and act on that.</p>
<h2>Wire it up</h2>
<p>The FastAPI route is thin:</p>
<pre><code class="language-python">import os

from fastapi import FastAPI, Request
from fastapi.concurrency import run_in_threadpool
from fastapi.responses import JSONResponse

from inbox import Inbox
from receiver import WebhookReceiver

app = FastAPI()
receiver = WebhookReceiver(Inbox("webhooks.db"), secrets=[os.environ["WEBHOOK_SECRET"]])

@app.post("/webhooks/payments")
async def payments_webhook(request: Request):
    body = await request.body()  # the RAW bytes: the signature covers exactly these
    status, payload = await run_in_threadpool(receiver.handle, request.headers, body)
    return JSONResponse(payload, status_code=status)
</code></pre>
<p>And the worker is a loop:</p>
<pre><code class="language-python">import time

from inbox import Inbox
from ledger import Ledger
from worker import Worker

ledger = Ledger("webhooks.db")

def payment_succeeded(event):
    ledger.credit(event["id"], event["account"], event["cents"])

worker = Worker(Inbox("webhooks.db"), {"payment.succeeded": payment_succeeded})

while True:
    if not worker.run_once():
        time.sleep(1)  # nothing due: wait a moment
</code></pre>
<p>Run them as two separate processes, with the secret in an environment variable:</p>
<pre><code class="language-bash">WEBHOOK_SECRET=whsec_... uvicorn main:app --port 8000
python run_worker.py
</code></pre>
<p>Also put a body size limit on your reverse proxy, such as <code>client_max_body_size</code> in nginx. The receiver's <code>max_body</code> check runs after the framework has read the whole body.</p>
<h2>See it work</h2>
<p>This script sends the receiver seven different requests, with a clock it can move forward:</p>
<pre><code class="language-python">import json
import os
import tempfile

from inbox import Inbox
from receiver import WebhookReceiver
from signing import standard_signature

SECRET = "whsec_" + "dGVzdC1zZWNyZXQtMDEyMzQ1Njc4OWFiY2RlZg=="
now = [1_800_000_000]  # a clock we can move
reasons = []
inbox = Inbox(os.path.join(tempfile.mkdtemp(), "demo.db"), clock=lambda: now[0])
receiver = WebhookReceiver(inbox, [SECRET], clock=lambda: now[0], on_reject=reasons.append)

body = json.dumps({"id": "evt_1", "type": "payment.succeeded", "account": "acct_1", "cents": 500}).encode()

def signed(message_id="msg_1", payload=body, secret=SECRET):
    timestamp = str(now[0])
    return {"webhook-id": message_id, "webhook-timestamp": timestamp,
            "webhook-signature": standard_signature(secret, message_id, timestamp, payload)}

def show(label, headers, payload=body):
    status, answer = receiver.handle(headers, payload)
    detail = answer.get("status") or f"{answer['error']} ({reasons[-1]})"
    print(f"{label:&lt;34} -&gt; {status} {detail}")

captured = signed()
show("a valid delivery", captured)
show("the same request again", captured)
now[0] += 30
show("the sender's retry (new timestamp)", signed())
show("a tampered body", signed(), body.replace(b"500", b"900"))
show("signed with the wrong secret", signed(secret="whsec_" + "d3Jvbmctc2VjcmV0LXdyb25nLXNlY3JldC0wMDAwMA=="))
show("no signature headers at all", {})
now[0] += 600
show("the captured request, 10 minutes on", captured)
print("stored events:", inbox.counts())
</code></pre>
<pre><code class="language-text">a valid delivery                   -&gt; 200 accepted
the same request again             -&gt; 200 duplicate
the sender's retry (new timestamp) -&gt; 200 duplicate
a tampered body                    -&gt; 400 invalid webhook (bad-signature)
signed with the wrong secret       -&gt; 400 invalid webhook (bad-signature)
no signature headers at all        -&gt; 400 invalid webhook (missing-headers)
the captured request, 10 minutes on -&gt; 400 invalid webhook (stale-timestamp)
stored events: {'pending': 1}
</code></pre>
<p>The first delivery is accepted. The same request again, and the sender's retry with a new timestamp, are acknowledged as duplicates. A tampered body, a wrong secret and missing headers are all rejected, each for its own reason. And the captured request replayed ten minutes later is stale. At the end, exactly one event is stored.</p>
<h2>Defense in depth</h2>
<p>Signatures are the real control. These add layers:</p>
<ul>
<li><p><strong>HTTPS only.</strong> Both GitHub and Stripe expect it, and you should verify certificates when you're the sender.</p>
</li>
<li><p><strong>IP allowlists.</strong> GitHub and Stripe publish the addresses their webhooks come from. An allowlist narrows who can reach you, but the addresses change, so it never replaces the signature.</p>
</li>
<li><p><strong>A separate endpoint and secret per integration.</strong> A leak then stays contained.</p>
</li>
<li><p><strong>No secrets in logs.</strong> Not the secret, and not full signature headers.</p>
</li>
<li><p><strong>Skip CSRF protection on the route.</strong> Webhooks aren't browser requests, and the signature takes CSRF's place.</p>
</li>
</ul>
<h2>If you're the one sending webhooks</h2>
<p>The same lessons apply from the other side:</p>
<ul>
<li><p><strong>Sign the ID and the timestamp</strong>, not just the body.</p>
</li>
<li><p><strong>Use one secret per endpoint,</strong> and support rotation by signing with both during the changeover.</p>
</li>
<li><p><strong>Retry with exponential backoff and jitter.</strong> Keep the ID the same and give each attempt a fresh timestamp and signature.</p>
</li>
<li><p><strong>Use short timeouts.</strong> Treat anything that isn't a 2xx, redirects included, as a failure.</p>
</li>
<li><p><strong>Validate customer-supplied URLs,</strong> so your system can't be tricked into calling internal addresses.</p>
</li>
</ul>
<h2>Checklist</h2>
<ul>
<li><p>Verify the signature on the <strong>raw body</strong>.</p>
</li>
<li><p>Compare with <code>hmac.compare_digest</code>, on bytes.</p>
</li>
<li><p>Reject old and future timestamps, and accept a list of secrets.</p>
</li>
<li><p>Deduplicate on the event ID, and keep IDs as long as the sender can retry.</p>
</li>
<li><p>Answer fast: verify, store, return 2xx.</p>
</li>
<li><p>Process from a queue, with backoff and a dead letter pile.</p>
</li>
<li><p>Make handlers idempotent, and don't depend on event order.</p>
</li>
<li><p>Log the reason for a rejection, and tell the sender nothing.</p>
</li>
</ul>
<h2>The tests</h2>
<p>Here are all 23. The vector tests are the first ones to read:</p>
<pre><code class="language-python">import base64
import hashlib
import hmac
import json
import os
import sys
import tempfile
import threading
import time
from unittest import mock

from inbox import Inbox
from ledger import Ledger
from receiver import WebhookReceiver
from signing import InvalidWebhook, standard_signature, verify_github, verify_standard, verify_stripe
from worker import Worker

NOW = 1_800_000_000
SECRET = "whsec_" + base64.b64encode(b"a-secret-for-tests-0123456789abc").decode()
OLD_SECRET = "whsec_" + base64.b64encode(b"the-previous-secret-0123456789ab").decode()

# The published Standard Webhooks test vector, and GitHub's from its docs.
SPEC = dict(secret="whsec_MfKQ9r8GKYqrTwjUPD8ILPZIo2LaLaSw", id="msg_p5jXN8AQM9LWM0D4loKWxJek",
            timestamp="1614265330", body=b'{"test": 2432232314}',
            signature="v1,g0hM9SsE+OTPJTGt/tmIKtSyZlE3uFJELVlNIOLJ1OE=")
GITHUB = dict(secret="It's a Secret to Everybody", body=b"Hello, World!",
              header="sha256=757107ea0eb2509fc211221cce984b8a37570b6d7586c22c46f4379c8b043e17")

class FakeClock:
    def __init__(self, now=NOW):
        self.now = now

    def __call__(self):
        return self.now

    def advance(self, seconds):
        self.now += seconds

def reason(call, *args):
    """The reason a verification was refused, or None if it passed."""
    try:
        call(*args)
    except InvalidWebhook as error:
        return error.reason
    return None

def standard_headers(message_id="msg_1", body=b"{}", timestamp=NOW, secret=SECRET):
    return {"webhook-id": message_id, "webhook-timestamp": str(timestamp),
            "webhook-signature": standard_signature(secret, message_id, str(timestamp), body)}

# ---- signatures, checked against the published vectors -------------------------------------------

def test_github_matches_its_published_vector():
    headers = {"X-Hub-Signature-256": GITHUB["header"]}
    assert reason(verify_github, GITHUB["secret"], headers, GITHUB["body"]) is None
    assert reason(verify_github, GITHUB["secret"], headers, b"Hello, World?") == "bad-signature"
    assert reason(verify_github, "wrong secret", headers, GITHUB["body"]) == "bad-signature"
    assert reason(verify_github, GITHUB["secret"], {}, GITHUB["body"]) == "missing-headers"
    sha1 = {"X-Hub-Signature-256": "sha1=" + GITHUB["header"][7:]}
    assert reason(verify_github, GITHUB["secret"], sha1, GITHUB["body"]) == "bad-signature"

def test_standard_webhooks_matches_its_published_vector():
    assert standard_signature(SPEC["secret"], SPEC["id"], SPEC["timestamp"], SPEC["body"]) == SPEC["signature"]
    headers = {"webhook-id": SPEC["id"], "webhook-timestamp": SPEC["timestamp"], "webhook-signature": SPEC["signature"]}
    verified = verify_standard([SPEC["secret"]], headers, SPEC["body"], now=1_614_265_330)
    assert (verified.message_id, verified.timestamp) == (SPEC["id"], 1_614_265_330)

def test_the_timestamp_window_is_five_minutes_either_way():
    h = {"webhook-id": SPEC["id"], "webhook-timestamp": SPEC["timestamp"], "webhook-signature": SPEC["signature"]}
    ts = int(SPEC["timestamp"])
    check = lambda now: reason(verify_standard, [SPEC["secret"]], h, SPEC["body"], now)
    assert check(ts + 300) is None and check(ts - 300) is None
    assert check(ts + 301) == "stale-timestamp" and check(ts - 301) == "stale-timestamp"
    assert reason(verify_standard, [SPEC["secret"]], {**h, "webhook-timestamp": "yesterday"}, SPEC["body"], ts) == "bad-timestamp"

def test_everything_that_was_signed_must_match():
    body = b'{"amount": 500}'
    good = standard_headers("msg_1", body)
    verify = lambda headers, payload=body, secrets=(SECRET,): reason(verify_standard, list(secrets), headers, payload, NOW)
    assert verify(good) is None
    assert verify(good, b'{"amount": 900}') == "bad-signature"                      # a changed body
    assert verify({**good, "webhook-id": "msg_2"}) == "bad-signature"                # a changed id
    assert verify({**good, "webhook-timestamp": str(NOW + 1)}) == "bad-signature"    # a changed timestamp
    assert verify(good, secrets=(OLD_SECRET,)) == "bad-signature"                    # the wrong secret

def test_secret_rotation_accepts_old_and_new_and_ignores_unknown_versions():
    body = b"{}"
    signed_with_old = standard_headers("msg_1", body, secret=OLD_SECRET)
    assert reason(verify_standard, [SECRET, OLD_SECRET], signed_with_old, body, NOW) is None
    assert reason(verify_standard, [SECRET], signed_with_old, body, NOW) == "bad-signature"  # once the old one is retired
    good = standard_headers("msg_1", body)["webhook-signature"]
    two = {**standard_headers("msg_1", body), "webhook-signature": "v1,AAAA " + good}
    assert reason(verify_standard, [SECRET], two, body, NOW) is None                        # one valid signature is enough
    other_version = {**standard_headers("msg_1", body), "webhook-signature": good.replace("v1,", "v2,")}
    assert reason(verify_standard, [SECRET], other_version, body, NOW) == "bad-signature"

def test_a_non_ascii_signature_is_rejected_not_a_crash():
    headers = {**standard_headers("msg_1"), "webhook-signature": "v1,caf\u00e9\u2713"}
    assert reason(verify_standard, [SECRET], headers, b"{}", NOW) == "bad-signature"
    assert reason(verify_github, "s", {"x-hub-signature-256": "sha256=\u2713"}, b"") == "bad-signature"

def test_header_names_are_case_insensitive():
    headers = {name.title(): value for name, value in standard_headers().items()}
    assert reason(verify_standard, [SECRET], headers, b"{}", NOW) is None

def test_signatures_are_compared_in_constant_time():
    with mock.patch.object(hmac, "compare_digest", wraps=hmac.compare_digest) as compare:
        verify_standard([SECRET], standard_headers(), b"{}", NOW)
        verify_github(GITHUB["secret"], {"x-hub-signature-256": GITHUB["header"]}, GITHUB["body"])
        assert compare.call_count == 2

def test_re_serialised_json_breaks_the_signature():
    raw = b'{"id": "evt_1",  "type": "payment.succeeded"}'  # two spaces: perfectly valid JSON
    headers = standard_headers("msg_1", raw)
    assert reason(verify_standard, [SECRET], headers, raw, NOW) is None
    parsed_and_dumped = json.dumps(json.loads(raw)).encode()
    assert parsed_and_dumped != raw
    assert reason(verify_standard, [SECRET], headers, parsed_and_dumped, NOW) == "bad-signature"

def stripe_header(secret, body, timestamp=NOW, extra=""):
    digest = hmac.new(secret.encode(), f"{timestamp}.".encode() + body, hashlib.sha256).hexdigest()
    return f"t={timestamp},v1={digest}{extra}"

def test_stripe_style_signatures():
    secret, body = "whsec_stripe_test", b'{"id": "evt_1"}'
    header = stripe_header(secret, body)
    assert reason(verify_stripe, secret, header, body, NOW) is None
    assert reason(verify_stripe, secret, header, b'{"id": "evt_2"}', NOW) == "bad-signature"
    assert reason(verify_stripe, secret, header, body, NOW + 301) == "stale-timestamp"
    assert reason(verify_stripe, secret, f"t={NOW},v0=abc", body, NOW) == "missing-headers"  # v0 is never trusted
    assert reason(verify_stripe, secret, "v1=abc", body, NOW) == "missing-headers"
    assert reason(verify_stripe, secret, stripe_header(secret, body, extra=",v1=" + "0" * 64), body, NOW) is None  # two v1s

# ---- the receiver: verify, store, answer ------------------------------------------------------------------

def make(**kwargs):
    clock = FakeClock()
    rejected = []
    inbox = Inbox(os.path.join(tempfile.mkdtemp(), "w.db"), clock=clock, rng=lambda: 1.0, **kwargs)
    receiver = WebhookReceiver(inbox, [SECRET], clock=clock, on_reject=rejected.append)
    return receiver, inbox, clock, rejected

def event(event_id="evt_1", cents=500):
    return json.dumps({"id": event_id, "type": "payment.succeeded", "account": "acct_1", "cents": cents}).encode()

def test_a_valid_delivery_is_stored_and_acknowledged():
    receiver, inbox, _, _ = make()
    body = event()
    assert receiver.handle(standard_headers("msg_1", body), body) == (200, {"status": "accepted"})
    assert inbox.counts() == {"pending": 1}  # stored for later: nothing was processed yet

def test_a_retry_is_acknowledged_but_stored_once():
    receiver, inbox, clock, _ = make()
    body = event()
    receiver.handle(standard_headers("msg_1", body, clock.now), body)
    clock.advance(30)  # the sender retries: same id, fresh timestamp, fresh signature
    status, payload = receiver.handle(standard_headers("msg_1", body, clock.now), body)
    assert (status, payload) == (200, {"status": "duplicate"})
    assert inbox.counts() == {"pending": 1}

def test_eight_simultaneous_copies_are_stored_once():
    receiver, inbox, _, _ = make()
    body, results, barrier = event(), [], threading.Barrier(8)

    def deliver():
        barrier.wait()
        results.append(receiver.handle(standard_headers("msg_1", body), body)[1]["status"])

    threads = [threading.Thread(target=deliver) for _ in range(8)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    assert sorted(results) == ["accepted"] + ["duplicate"] * 7 and inbox.counts() == {"pending": 1}

def test_rejections_say_nothing_to_the_sender_and_store_nothing():
    receiver, inbox, clock, rejected = make()
    body = event()
    forged = standard_headers("msg_1", body, secret=OLD_SECRET)
    stale = standard_headers("msg_2", body, clock.now - 1000)
    responses = [receiver.handle(forged, body), receiver.handle(stale, body), receiver.handle({}, body)]
    assert responses == [(400, {"error": "invalid webhook"})] * 3  # the same answer every time
    assert rejected == ["bad-signature", "stale-timestamp", "missing-headers"]  # the reasons are for you
    assert inbox.counts() == {}

def test_oversized_bodies_are_refused_before_any_hmac_is_computed():
    receiver, inbox, _, _ = make()
    huge = b"x" * 2_000_000
    headers = standard_headers("msg_1", huge)  # signed correctly, so only the size can be the problem
    with mock.patch.object(hmac, "new", side_effect=AssertionError("computed an HMAC")):
        assert receiver.handle(headers, huge)[0] == 413
    assert inbox.counts() == {}

def test_a_replay_is_harmless_inside_the_window_and_rejected_after_it():
    receiver, inbox, clock, rejected = make()
    body = event()
    captured = standard_headers("msg_1", body, clock.now)
    assert receiver.handle(captured, body)[1] == {"status": "accepted"}
    clock.advance(100)
    assert receiver.handle(captured, body) == (200, {"status": "duplicate"})  # the same bytes again: nothing happens
    clock.advance(300)
    assert receiver.handle(captured, body)[0] == 400 and rejected == ["stale-timestamp"]
    assert inbox.counts() == {"pending": 1}

# ---- the worker: retries, dead letters, crashes ---------------------------------------------------------------------

def deliver(receiver, clock, event_id, cents=500, message_id=None):
    body = event(event_id, cents)
    message_id = message_id or "msg_" + event_id
    return receiver.handle(standard_headers(message_id, body, clock.now), body)

def test_the_worker_runs_the_handler_and_marks_the_event_done():
    receiver, inbox, clock, _ = make()
    deliver(receiver, clock, "evt_1")
    seen = []
    worker = Worker(inbox, {"payment.succeeded": lambda e: seen.append(e["id"])})
    assert worker.run_once() is True and worker.run_once() is False
    assert seen == ["evt_1"] and inbox.counts() == {"done": 1}

def test_unknown_events_are_ignored_and_unreadable_ones_are_set_aside():
    receiver, inbox, clock, _ = make()
    unknown = json.dumps({"id": "evt_1", "type": "something.else"}).encode()
    receiver.handle(standard_headers("msg_1", unknown, clock.now), unknown)
    garbage = b"not json at all"
    receiver.handle(standard_headers("msg_2", garbage, clock.now), garbage)
    called = []
    worker = Worker(inbox, {"payment.succeeded": called.append})
    while worker.run_once():
        pass
    assert called == [] and inbox.counts() == {"done": 1, "dead": 1}  # a bad payload won't improve with retries

def test_failures_are_retried_with_growing_delays():
    receiver, inbox, clock, _ = make()
    deliver(receiver, clock, "evt_1")
    attempts = []

    def flaky(e):
        attempts.append(clock.now)
        if len(attempts) &lt; 3:
            raise RuntimeError("database is down")

    worker = Worker(inbox, {"payment.succeeded": flaky})
    assert worker.run_once() is True  # attempt 1 fails: wait 30 seconds
    assert worker.run_once() is False
    clock.advance(30)
    assert worker.run_once() is True  # attempt 2 fails: wait 60 seconds
    clock.advance(59)
    assert worker.run_once() is False
    clock.advance(1)
    assert worker.run_once() is True  # attempt 3 works
    assert len(attempts) == 3 and inbox.counts() == {"done": 1}

def test_an_event_that_keeps_failing_is_set_aside_after_the_last_attempt():
    receiver, inbox, clock, _ = make(max_attempts=3)
    deliver(receiver, clock, "evt_1")

    def broken(e):
        raise RuntimeError("always")

    worker = Worker(inbox, {"payment.succeeded": broken})
    for _ in range(3):
        assert worker.run_once() is True
        clock.advance(3600)
    assert worker.run_once() is False and inbox.counts() == {"dead": 1}

def test_a_crash_after_the_side_effect_is_recovered_and_credits_once():
    receiver, inbox, clock, _ = make()
    ledger = Ledger(os.path.join(tempfile.mkdtemp(), "ledger.db"))
    deliver(receiver, clock, "evt_1", cents=700)
    job = inbox.next_job()  # a worker leases the event...
    ledger.credit("evt_1", "acct_1", 700)  # ...runs the handler...
    del job  # ...and dies before it can mark the event done
    worker = Worker(inbox, {"payment.succeeded": lambda e: ledger.credit(e["id"], e["account"], e["cents"])})
    assert worker.run_once() is False  # still leased: nobody else touches it yet
    clock.advance(61)  # the lease runs out
    assert worker.run_once() is True  # the event runs again
    assert ledger.balance("acct_1") == 700  # but the account was credited once
    assert inbox.counts() == {"done": 1}

def test_two_workers_never_run_the_same_event():
    receiver, inbox, clock, _ = make()
    for i in range(60):
        deliver(receiver, clock, f"evt_{i}")
    handled, lock, barrier = [], threading.Lock(), threading.Barrier(8)

    def handler(e):
        with lock:
            handled.append(e["id"])

    def work():
        worker = Worker(inbox, {"payment.succeeded": handler})
        barrier.wait()  # all eight start together
        while worker.run_once():
            pass

    switch_interval = sys.getswitchinterval()
    sys.setswitchinterval(1e-6)  # make threads trade places as often as possible
    try:
        threads = [threading.Thread(target=work) for _ in range(8)]
        for t in threads:
            t.start()
        for t in threads:
            t.join()
    finally:
        sys.setswitchinterval(switch_interval)
    assert sorted(handled) == sorted(f"evt_{i}" for i in range(60)) and inbox.counts() == {"done": 60}

def test_end_to_end_retries_and_duplicates_credit_each_payment_once():
    receiver, inbox, clock, _ = make()
    ledger = Ledger(os.path.join(tempfile.mkdtemp(), "ledger.db"))
    deliver(receiver, clock, "evt_1", cents=500)
    deliver(receiver, clock, "evt_2", cents=250)
    clock.advance(30)
    deliver(receiver, clock, "evt_1", cents=500)  # the sender retried evt_1
    worker = Worker(inbox, {"payment.succeeded": lambda e: ledger.credit(e["id"], e["account"], e["cents"])})
    while worker.run_once():
        pass
    assert ledger.balance("acct_1") == 750 and inbox.counts() == {"done": 2}
</code></pre>
<p>Run them with <code>pytest -q</code>. They use SQLite and a fake clock, so they need no network and no web framework.</p>
<hr />
<p><em>Checked against the GitHub and Stripe documentation, the Standard Webhooks spec and the Python docs on November 7, 2026. Providers change their signing details occasionally, so check the current docs for the sender you integrate with.</em></p>
<p><strong>Sources</strong></p>
<ul>
<li><p><a href="https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries">GitHub: Validating webhook deliveries</a></p>
</li>
<li><p><a href="https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks">GitHub: Best practices for using webhooks</a></p>
</li>
<li><p><a href="https://docs.stripe.com/webhooks">Stripe: Receive Stripe events in your webhook endpoint</a></p>
</li>
<li><p><a href="https://docs.stripe.com/webhooks/signatures">Stripe: Check the webhook signatures</a></p>
</li>
<li><p><a href="https://www.standardwebhooks.com/">Standard Webhooks</a></p>
</li>
<li><p><a href="https://docs.python.org/3/library/hmac.html">Python docs: hmac</a></p>
</li>
</ul>
<hr />
<p><em>Published via <a href="https://zyvop.com/webhook-security-verify-signatures-stop-replays-handle-retries-x05wj?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Zero-Downtime Database Migrations: How to Change a Live Schema Safely]]></title><description><![CDATA[A migration that finishes in milliseconds on your laptop can lock a production table for an hour. This guide shows how to avoid that: the…]]></description><link>https://blog.zyvop.com/zero-downtime-database-migrations-how-to-change-a-live-schema-safely-zehkb</link><guid isPermaLink="true">https://blog.zyvop.com/zero-downtime-database-migrations-how-to-change-a-live-schema-safely-zehkb</guid><category><![CDATA[database-migrations]]></category><category><![CDATA[Devops]]></category><category><![CDATA[MySQL]]></category><category><![CDATA[PostgreSQL]]></category><category><![CDATA[ZeroDowntime]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Wed, 11 Nov 2026 05:39:20 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/92dfe1c9-68e2-49eb-96f2-c044b0137b1c.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>Expand, migrate, contract: a playbook for schema changes that never take your application offline.</em></p>
<p>A migration that finishes in milliseconds on your laptop can lock a production table for an hour. The causes are almost always the same: statements that take heavy locks, operations that rewrite an entire table, and old application code running against a schema it does not understand. This post covers a repeatable way to avoid all three.</p>
<h2>The core idea: expand and contract</h2>
<p>During a deploy, old and new versions of your code run side by side, and you need the option to roll back. So every schema state must work with both the previous and the next version of the application. The way to guarantee that is to split a risky change into three phases, each shipped separately:</p>
<ol>
<li><p><strong>Expand.</strong> Add the new structure (a column, table, or index) in a backward-compatible way. Old code ignores it.</p>
</li>
<li><p><strong>Migrate.</strong> Write to both the old and new structures, backfill existing data, then switch reads to the new one.</p>
</li>
<li><p><strong>Contract.</strong> Once nothing uses the old structure, remove it.</p>
</li>
</ol>
<p>This takes more deploys than a single ALTER statement, but each step is small, reversible, and safe to run under load.</p>
<h2>Know what a lock can do to you</h2>
<p>Most migration outages are not caused by the slow statement itself. They come from the lock queue. A DDL statement waits for a lock behind a long-running transaction, and while it waits, every new query on that table queues up behind the DDL. A change that would have taken milliseconds can freeze the table for as long as that one old transaction runs.</p>
<p>The fix is to make migrations fail fast and retry instead of waiting:</p>
<pre><code class="language-sql">-- Postgres: give up quickly rather than queue behind a long transaction
SET lock_timeout = '3s';
ALTER TABLE users ADD COLUMN display_name text;
</code></pre>
<p>MySQL has <code>lock_wait_timeout</code> for the same purpose. If the statement times out, nothing is harmed: retry in a moment with a small backoff. It also helps to check for long-running transactions before you start (in Postgres, look at <code>pg_stat_activity</code>), because they are what turn a lock wait into an outage.</p>
<h2>Safe and risky versions of common changes</h2>
<table>
<thead>
<tr>
<th>Change</th>
<th>Risky approach</th>
<th>Safer approach</th>
</tr>
</thead>
<tbody><tr>
<td>Add a column</td>
<td>A default that forces a full table rewrite, such as a volatile default</td>
<td>Add it nullable, or with a constant default where the engine makes that a metadata-only change (Postgres 11+, MySQL 8 INSTANT). Backfill separately</td>
</tr>
<tr>
<td>Add an index</td>
<td>Plain CREATE INDEX in Postgres, which blocks writes while it builds</td>
<td>CREATE INDEX CONCURRENTLY in Postgres; online DDL with ALGORITHM=INPLACE, LOCK=NONE in MySQL</td>
</tr>
<tr>
<td>Add NOT NULL</td>
<td>SET NOT NULL on a large table, which scans every row under a heavy lock</td>
<td>Add a CHECK constraint as NOT VALID, validate it separately, then set NOT NULL</td>
</tr>
<tr>
<td>Add a foreign key</td>
<td>Adding it validated, which checks every row while holding locks</td>
<td>Add it as NOT VALID, then run VALIDATE CONSTRAINT as a second step</td>
</tr>
<tr>
<td>Change a column type</td>
<td>ALTER COLUMN TYPE, which usually rewrites the whole table</td>
<td>Add a new column, dual write, backfill, switch reads, drop the old one</td>
</tr>
<tr>
<td>Rename a column or table</td>
<td>A direct rename, which breaks running code the instant it applies</td>
<td>Expand and contract, as in the walkthrough below</td>
</tr>
<tr>
<td>Drop a column</td>
<td>Dropping it while any running code still reads or writes it</td>
<td>Ship code that stops using it first, then drop it in a later release</td>
</tr>
</tbody></table>
<p>Two details trip people up. <code>CREATE INDEX CONCURRENTLY</code> cannot run inside a transaction block, and if it fails it leaves an invalid index behind that you must drop before retrying. And the NOT NULL pattern is worth memorizing, because it avoids a long lock on a big table:</p>
<pre><code class="language-sql">ALTER TABLE users ADD CONSTRAINT display_name_not_null
  CHECK (display_name IS NOT NULL) NOT VALID;
ALTER TABLE users VALIDATE CONSTRAINT display_name_not_null;
-- Postgres 12+ uses the validated constraint and skips the full scan
ALTER TABLE users ALTER COLUMN display_name SET NOT NULL;
ALTER TABLE users DROP CONSTRAINT display_name_not_null;
</code></pre>
<h2>Walkthrough: renaming a column</h2>
<p>Say you want to rename <code>users.fullname</code> to <code>users.display_name</code>. A direct rename would break every running instance of the old code, so you do it in seven steps, each its own deploy:</p>
<ol>
<li><p><strong>Expand.</strong> Add a nullable <code>display_name</code> column. Old code ignores it.</p>
</li>
<li><p><strong>Dual write.</strong> Deploy code that writes both columns but still reads <code>fullname</code>.</p>
</li>
<li><p><strong>Backfill.</strong> Copy existing rows into the new column in small batches.</p>
</li>
<li><p><strong>Verify.</strong> Confirm the two columns match for every row.</p>
</li>
<li><p><strong>Switch reads.</strong> Deploy code that reads <code>display_name</code>. Keep writing both, so a rollback is still safe.</p>
</li>
<li><p><strong>Stop using the old column.</strong> Deploy code that no longer touches <code>fullname</code>.</p>
</li>
<li><p><strong>Contract.</strong> Drop <code>fullname</code> in a later migration.</p>
</li>
</ol>
<p>At every step the previous version of the code still works, which is exactly what makes rollbacks boring.</p>
<h2>Backfill carefully</h2>
<p>A single giant UPDATE is one of the most common ways to hurt a live database. It holds locks for a long time, generates a large burst of write-ahead log and replication traffic, and in Postgres leaves a pile of dead rows behind. Backfill in small batches instead:</p>
<pre><code class="language-sql">-- Run for consecutive id ranges, pausing between batches
UPDATE users
SET display_name = fullname
WHERE id &gt;= :start AND id &lt; :start + 5000
  AND display_name IS DISTINCT FROM fullname;
</code></pre>
<p>A few habits make backfills safe:</p>
<ul>
<li><p><strong>Run them outside the schema migration</strong>, as a separate job you can pause and resume.</p>
</li>
<li><p><strong>Make them idempotent</strong>, so a restart never corrupts data. The <code>IS DISTINCT FROM</code> check above skips rows that are already correct.</p>
</li>
<li><p><strong>Throttle.</strong> Sleep between batches and watch replication lag and lock waits, slowing down when either rises.</p>
</li>
<li><p><strong>Verify at the end</strong> with a query that counts rows where the old and new values differ.</p>
</li>
</ul>
<h2>Run migrations as their own step</h2>
<p>Do not run migrations automatically when the application boots. With many instances starting at once, they race each other, slow startups, and can fail health checks. Run migrations as a dedicated pipeline step instead, and order them by direction:</p>
<ul>
<li><p><strong>Additive changes</strong> (new columns, tables, indexes): migrate first, then deploy the code that uses them.</p>
</li>
<li><p><strong>Destructive changes</strong> (dropping columns or tables): deploy code that stops using them first, then migrate.</p>
</li>
</ul>
<p>Prefer forward-only migrations. Because each schema state works with both code versions, you can roll back the application without rolling back the schema, and fix problems with a new migration instead.</p>
<p>For very large MySQL tables, tools such as gh-ost and pt-online-schema-change build a shadow copy of the table and swap it in, which avoids long locks on changes that would otherwise rewrite the table in place.</p>
<h2>Test against production-sized data</h2>
<p>The size of the data is the variable that changes everything. A migration that takes 40 milliseconds against a development database with a thousand rows can take hours against hundreds of millions.</p>
<ul>
<li><p><strong>Rehearse on a production-sized copy</strong> and record how long each statement takes and what locks it holds.</p>
</li>
<li><p><strong>Lint your migrations.</strong> Tools such as squawk for Postgres and strong_migrations for Rails flag unsafe operations before they merge.</p>
</li>
<li><p><strong>Rehearse the rollback</strong>, not just the rollout.</p>
</li>
</ul>
<h2>A migration checklist</h2>
<ol>
<li><p>Can the old and new versions of the code both run against every intermediate schema?</p>
</li>
<li><p>Is <code>lock_timeout</code> set, and does the migration retry safely if it times out?</p>
</li>
<li><p>Does any statement rewrite the table or scan every row under a heavy lock?</p>
</li>
<li><p>Are indexes built concurrently, and constraints added as NOT VALID and validated separately?</p>
</li>
<li><p>Is the backfill batched, throttled, idempotent, and separate from the migration?</p>
</li>
<li><p>Have you verified the data before switching reads, and again before dropping the old structure?</p>
</li>
<li><p>Is destructive cleanup deferred to a later release?</p>
</li>
<li><p>Did you test on production-sized data, including the rollback?</p>
</li>
</ol>
<h2>Conclusion</h2>
<p>Zero-downtime migrations are less about clever tricks than about discipline: never ship a schema state that either version of your code cannot survive, keep locks short, and move data in small steps. The cost is more deploys per change. The payoff is that schema changes stop being an event you schedule a maintenance window for and become routine work you can do any afternoon.</p>
<h2>Frequently Asked Questions</h2>
<h3>What are the three phases of the “expand, migrate, contract” pattern for zero‑downtime database migrations?</h3>
<p>Expand adds new columns, tables, or indexes in a backward‑compatible way; Migrate writes to both old and new structures, backfills existing data, and switches reads to the new structure; Contract removes the old structure once it is no longer used.</p>
<h3>How can you make a PostgreSQL migration fail fast instead of waiting on a lock?</h3>
<p>Set a short lock timeout (e.g., SET lock_timeout = '3s') before the DDL statement; if the lock cannot be acquired, the statement aborts and can be retried with a backoff.</p>
<h3>What is a safe way to add a new column with a default value without rewriting the whole table?</h3>
<p>Add the column as nullable or with a constant default that the engine can apply as a metadata‑only change (PostgreSQL 11+ or MySQL 8 INSTANT), then backfill the desired values in a separate step.</p>
<h3>How do you create an index in PostgreSQL without blocking writes?</h3>
<p>Use CREATE INDEX CONCURRENTLY, which builds the index online; note that it cannot run inside a transaction block and must be retried if it fails.</p>
<h3>What are the steps to rename a column safely using the expand‑contract approach?</h3>
<ol>
<li>Expand: add a new nullable column; 2) Dual write: update code to write both old and new columns; 3) Backfill: copy data in small batches; 4) Verify: ensure values match; 5) Switch reads: code reads the new column; 6) Stop using old column: code stops writing/reading it; 7) Contract: drop the old column in a later migration.</li>
</ol>
<h3>Why should backfills be performed in small, idempotent batches?</h3>
<p>Small batches keep locks short, limit write‑ahead‑log and replication traffic, allow throttling based on lag, and make the process resumable without corrupting data.</p>
<h3>Why should database migrations be run as a dedicated pipeline step rather than at application startup?</h3>
<p>Running migrations separately avoids many instances racing to apply the same change, prevents slow startups, and keeps health checks reliable.</p>
<h3>What key items are included in the migration checklist presented in the article?</h3>
<p>Ensure both code versions work on each intermediate schema, set lock_timeout and retry logic, avoid full‑table rewrites under heavy locks, build indexes concurrently and add constraints NOT VALID, batch and throttle backfills, verify data before switching reads and before dropping old structures, defer destructive cleanup, and test on production‑sized data including rollback.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/zero-downtime-database-migrations-how-to-change-a-live-schema-safely-zehkb?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Password Hashing Explained: Why SHA-256 Is the Wrong Tool and What to Use Instead]]></title><description><![CDATA[A leaked password database can expose accounts if passwords are poorly stored. Learn why SHA-256 is unsuitable, how salts and work factors help, and how to…]]></description><link>https://blog.zyvop.com/password-hashing-explained-why-sha-256-is-the-wrong-tool-and-what-to-use-instead-v36wj</link><guid isPermaLink="true">https://blog.zyvop.com/password-hashing-explained-why-sha-256-is-the-wrong-tool-and-what-to-use-instead-v36wj</guid><category><![CDATA[authentication]]></category><category><![CDATA[#backenddevelopment]]></category><category><![CDATA[Cryptography]]></category><category><![CDATA[cybersecurity]]></category><category><![CDATA[PasswordSecurity]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Wed, 11 Nov 2026 05:38:36 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/ee60a675-6bf9-42fb-916f-b6a17176fbea.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Sooner or later, a database leaks. When it happens to yours, the attacker holds everything your login form checks against. What matters then is how long it takes to turn those stored values back into real passwords.</p>
<p>That time depends mostly on one decision: which function you used to hash them. A plain SHA-256 hash can fall in minutes. A well-configured slow hash makes every guess tens of thousands to millions of times more expensive, depending on the algorithm.</p>
<p>This post shows the difference with measured numbers. Then it covers salts, the four algorithms OWASP recommends, how to pick settings, and how to upgrade hashes you already have.</p>
<h2>What a password hash is for</h2>
<p>Passwords should never be stored in plain text, and they shouldn’t be encrypted either. Encryption is reversible, so whoever steals the key gets every password.</p>
<p>A hash is one-way. To check a login, the server hashes what the user typed and compares it with the stored hash. Nobody can “decrypt” a hash, so an attacker has only one option: guess a password, hash it, and compare.</p>
<p><img src="https://mermaid.ink/img/pako:eNptUcFu1EAM_RXjPXCZSG2FQMoBVEBVORQhFpDQTg_OjLMzyiQTjb1Nl-3-O0pS4IJ8sv3e83vyCV32jDW2KU8uUFH49tEOAACJqdtZvFYl13EBUaYkoIFhJJEpFw-elBoStngPVfUW9gcW2Vn8El0HBI4GHz3pP4bF-1V9QS6cQBJ2Fm9JAkSFKWpYjhyEy0sBoaRAg19mQj2DsGoc9vJXa1ZYpFzuRyp8snhH6gKvbkVzYb-g3lk8r5xn6Ex7svg5W3xaPf1v_ZNl3rf5MPg53Z_4hV1-4MJrrJUpeky8WmpjSvWGX7cXLRnRkjuuN1fNG88Xz201Ra-hvhofjcspl3pzSXOhwZ5LT9FjfUIN3M8_8tzSISmadfKDSqQmscyYNg96Q31MR6yxonFMXMlRlHsD71Mcujty26W_yYMasLjlfWb4_smiga-5yZoN3HJ6YI2ODFyXSMmA0CCVcIktmuXINv6avVy-Gh_xfDbY7D_MzrHGF1OIynj-DUJOyiw?type=png" alt="Mermaid Diagram" /></p>
<p>Everything in this post is about that loop. You can’t stop the attacker from running it, but you can decide how long each turn takes.</p>
<h2>Why SHA-256 is the wrong tool</h2>
<p>SHA-256 was designed to be fast. That is a virtue for checking files and signatures, and a flaw for passwords: the faster the hash, the more guesses an attacker makes per second. <a href="https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html">OWASP</a> says plainly that fast hashes such as SHA-256 are not suitable for password storage.</p>
<p>To see how big the gap is, I timed four algorithms on a single CPU core. The script is at the end of this post. These are medians of 12 runs on one core of a small, shared cloud VM. Single runs varied by as much as 3.5×, and yours will differ, but the gap won’t:</p>
<table>
<thead>
<tr>
<th>Algorithm</th>
<th>Guesses per second</th>
<th>Time per hash</th>
<th>One million guesses</th>
</tr>
</thead>
<tbody><tr>
<td>SHA-256</td>
<td>about 2.3 million</td>
<td>under 0.001 ms</td>
<td>under 1 second</td>
</tr>
<tr>
<td>PBKDF2-HMAC-SHA256 (600,000 iterations)</td>
<td>7.8</td>
<td>127 ms</td>
<td>1.5 days</td>
</tr>
<tr>
<td>scrypt (N=2^17, r=8, p=1)</td>
<td>2.8</td>
<td>357 ms</td>
<td>4.1 days</td>
</tr>
<tr>
<td>Argon2id (19 MiB, 2 iterations, 1 lane)</td>
<td>43.9</td>
<td>23 ms</td>
<td>6.3 hours</td>
</tr>
</tbody></table>
<p>The SHA-256 row includes Python’s own overhead, so a real cracking tool does far better.</p>
<p>Attackers don’t use one CPU core. A single RTX 5090 running Hashcat reaches about 27.7 billion SHA-256 guesses per second, while bcrypt manages about 305,000 per second, according to <a href="https://www.tomshardware.com/pc-components/gpus/tests-show-usd30-000-ai-gpus-are-terrible-password-crackers-h200-and-mi300x-outperformed-by-rtx-5090">Tom’s Hardware’s benchmark</a>.</p>
<p>Hashcat’s bcrypt benchmark uses <a href="https://hashcat.net/forum/printthread.php?tid=8303">cost 5</a>, and each extra cost step doubles the work. Real systems use cost 10 or more, so I scaled the benchmark to cost 10 (about 9,500 guesses per second) and cost 12 (about 2,400):</p>
<table>
<thead>
<tr>
<th>What the attacker tries</th>
<th>SHA-256</th>
<th>bcrypt, cost 10</th>
<th>bcrypt, cost 12</th>
</tr>
</thead>
<tbody><tr>
<td>Every 8-character password of lowercase letters and digits</td>
<td>1.7 minutes</td>
<td>9.4 years</td>
<td>37.6 years</td>
</tr>
<tr>
<td>Every 8-character password of any printable ASCII character</td>
<td>2.8 days</td>
<td>about 22,000 years</td>
<td>about 88,000 years</td>
</tr>
<tr>
<td>A wordlist of 1 million common passwords</td>
<td>36 microseconds</td>
<td>1.7 minutes</td>
<td>7.0 minutes</td>
</tr>
</tbody></table>
<p>These are single-GPU estimates, so an attacker with 100 GPUs divides every time by 100.</p>
<p>Real attackers also don’t try every combination. They try likely passwords first: leaked passwords, wordlists and common patterns. A slow hash won’t save a very weak password from a targeted list, but it makes every guess expensive, and that cost adds up across thousands of accounts.</p>
<h2>Salts: a unique random value for every password</h2>
<p>A salt is a random value stored next to the hash and mixed into the hashing. It isn’t secret. Its job is to make the same password produce a different hash for every user.</p>
<p>Without salts, two users with the same password get the same hash:</p>
<pre><code>Unsalted SHA-256 of 'password123':
  alice ef92b778bafe771e...
  bob   ef92b778bafe771e...

Salted, with a random 16-byte salt per user:
  alice salt 6e1052a4...  hash d2b5b726bdcf6e30...
  bob   salt 7b6c4e96...  hash 175bf39233cc9ef2...
</code></pre>
<p>Salts do two jobs, as OWASP explains:</p>
<ul>
<li><p><strong>They defeat precomputed tables.</strong> Lookup tables of known password hashes (rainbow tables) are useless once every hash has its own random salt.</p>
</li>
<li><p><strong>They force one-at-a-time cracking.</strong> The attacker can’t hash a guess once and compare it against every account. The work grows in direct proportion to the number of accounts.</p>
</li>
</ul>
<p>To put a number on the second point: at bcrypt cost 12, one pass of a 1-million-word list takes about 7 minutes on one GPU. Against 1 million accounts with unique salts, that becomes a million passes, or about 13 years.</p>
<p>You rarely generate salts yourself. Most password-hashing libraries create and store one for you.</p>
<p>Some systems also add a <em>pepper</em>: a secret value shared by all passwords and kept outside the database, for example in a secrets vault. It helps when only the database leaks. It’s optional, and OWASP notes that changing a pepper later forces every affected user to reset their password.</p>
<h2>Three ways to make hashing expensive</h2>
<p>Good password hashes are slow on purpose, and each one gets there differently:</p>
<ul>
<li><p><strong>Many iterations (PBKDF2).</strong> Repeat a hash function hundreds of thousands of times. It costs CPU time only, which GPUs have in abundance.</p>
</li>
<li><p><strong>A tunable CPU cost (bcrypt).</strong> A cost setting controls how many rounds run, and each extra step doubles the time. It uses very little memory.</p>
</li>
<li><p><strong>Memory hardness (scrypt and Argon2id).</strong> Each guess needs a large block of RAM. A GPU with thousands of cores can’t run thousands of guesses at once unless it has memory for every one of them.</p>
</li>
</ul>
<p>The defender pays this price once per login. The attacker pays it for every guess.</p>
<h2>Which algorithm to use</h2>
<p><a href="https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html">OWASP</a> lists four, in order of preference:</p>
<table>
<thead>
<tr>
<th>Algorithm</th>
<th>Cost settings</th>
<th>OWASP minimum</th>
<th>Best for</th>
<th>Watch out for</th>
</tr>
</thead>
<tbody><tr>
<td>Argon2id</td>
<td>Memory, iterations, parallelism</td>
<td>19 MiB, 2 iterations, 1 lane (or 46 MiB with 1 iteration)</td>
<td>New systems</td>
<td>Needs a library, and each concurrent login uses its memory</td>
</tr>
<tr>
<td>scrypt</td>
<td>N, r, p</td>
<td>N=2^17 (128 MiB), r=8, p=1</td>
<td>When Argon2id isn’t available</td>
<td>Each hash needs 128 MiB at this setting</td>
</tr>
<tr>
<td>bcrypt</td>
<td>Cost factor (2^cost rounds)</td>
<td>Cost 10 or higher</td>
<td>Legacy systems where neither above is available</td>
<td>72-byte password limit</td>
</tr>
<tr>
<td>PBKDF2-HMAC-SHA256</td>
<td>Iteration count</td>
<td>600,000 iterations</td>
<td>FIPS-140 compliance</td>
<td>CPU cost only, so GPUs attack it efficiently</td>
</tr>
</tbody></table>
<p>Argon2 won the 2015 Password Hashing Competition, and its Argon2id variant is standardized in <a href="https://www.rfc-editor.org/rfc/rfc9106.html">RFC 9106</a>. PBKDF2 is the one to pick when you need FIPS-140 validated implementations, and it is permitted by NIST SP 800-63B-4.</p>
<p><img src="https://mermaid.ink/img/pako:eNptUV1v2zAM_Csc8yoDifsF-GFF2iBIMLQo6m1AEe2BtulEiCy5ktrUS_PfB8d2txSDXsS7I-8I7jG3BWOCpba7fEMuwPeZNAAAz5O9xJmFxr6AYS5gvnxIo8n5GHJb1VqRyfla4qFXx3uJSw9kYOrW1sSqAK0yR64BeiWlKdP_ys86uc9dU4f_KepsW5TxSuLDzbfZPI4Wd9PbKF1M44tLib86DbVOK4mD4wfRjV1J7D4feDbg2V982Bai6Ou7xCf2Et97-0_cvW2p5wGOT1uOaT5RfcfZsPVpR5_ulOta-oD9PB8azZ0BlErrZMSX5bgk4YOzW05GcXZV8Lgvo50qwiaJ6zeRW21dMppQ-1Bgxa4iVWCyx7Dhqr18wSW96ICiQ36SU-0lfKsprQlzqpRuMMGI6lpz5BsfuBJwo5XZ3lGeHuu5NUGAxJTXluHHUqKAR5vZYAUsWL9yUDkJmDpFWoAn4yPPTpUojiap-t1mmZzXb3g4CMzWt21yTPDLbqMC4-EPJ57a9Q?type=png" alt="Mermaid Diagram" /></p>
<h2>Choosing the cost settings</h2>
<p>OWASP’s numbers are minimums, not targets. In my test, Argon2id at its minimum took only about 23 ms, so there is room to spend more. The <code>argon2-cffi</code> library defaults to 64 MiB, 3 iterations and 4 lanes (the RFC 9106 low-memory profile), which is a stronger starting point.</p>
<p>You can also go too far. If one login costs your server a lot of CPU or memory, an attacker can slow it down just by sending many login attempts. With a memory-hard hash, 100 simultaneous logins at 19 MiB each need about 1.9 GB.</p>
<p>OWASP’s rule of thumb is that a hash should take less than one second. Run the script at the end of this post on your production hardware, raise the settings until you reach a latency you can afford, and revisit them as hardware improves.</p>
<h2>Putting it into practice</h2>
<p>Here is the whole life of a password. The stored record includes the algorithm and its settings, so you can change them later without breaking old accounts:</p>
<p><img src="https://mermaid.ink/img/pako:eNqNUr1u2zAQfpULZ2oo0EmDATtCmgDJUkGZvJyos8SE5LHkKW4a-N0LSnULAy6Q8fj9HskPZXggVatMP2YKhhqLY0K_DwAAOAuH2feU1jliEmtsxCDQAWbo8jWoLVBL6e0a2OwK2qBgj5lWvKs2m7aG1o4B5ghHKxMgRMz5yGlYOe3K-UaBEgoBQsIwsIeMTgDDABPmCWSiK8JmV0MrnAjQjZysTF5DJhEbxqwvLS4qPfIINvy_UTF-ZByW3FwSBkhk_nKaXbUabT-Z-2fN-_MuKEI-ytpgCUFP_4RnrwvxLfuIiRa6HHlxp1z2MByylFcQ6-msqTabroatMRQFOEGiFzKyqB2PNiitPCWPdlD1h5KJfPkwAx1wdqL0evKMyWLvKBfOgYPcobfuXdWqwhgdVfk9C3kNO2fD6xOadpnvOIiGvWppZILuYa80fOeehTXck3sjsQY1bJNFV-4r5CpTsgell5DW_ipdvnyNP9XppFU_3rLjpGp1c5yskDr9Bibz9aI?type=png" alt="Mermaid Diagram" /></p>
<p>With the <code>argon2-cffi</code> package, the library does all of this for you:</p>
<pre><code class="language-python">from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError

ph = PasswordHasher()                       # Argon2id with RFC 9106 low-memory defaults

stored = ph.hash(password)                  # random salt, settings included in the string
try:
    ph.verify(stored, attempt)              # raises if the password is wrong
except VerifyMismatchError:
    ...                                     # reject the login
else:
    if ph.check_needs_rehash(stored):       # are the stored settings out of date?
        stored = ph.hash(attempt)           # re-hash with current settings, then save it
</code></pre>
<p>No extra packages? Python’s standard library includes scrypt. This version shows what a library does under the hood: a fresh salt, settings stored with the hash, a constant-time comparison and a rehash check.</p>
<pre><code class="language-python">import base64, hashlib, hmac, os

LN, R, P = 17, 8, 1   # scrypt N=2^17 (128 MiB), r=8, p=1: OWASP's minimum for scrypt

def _scrypt(password, salt, ln, r, p):
    # maxmem must exceed about 128 * N * r bytes, so raise it if you raise ln
    return hashlib.scrypt(password.encode(), salt=salt, n=2**ln, r=r, p=p,
                          maxmem=2**28, dklen=32)

def hash_password(password):
    salt = os.urandom(16)                        # a fresh random salt for every password
    digest = _scrypt(password, salt, LN, R, P)
    b64 = lambda raw: base64.b64encode(raw).decode()
    return f'$scrypt$ln={LN},r={R},p={P}${b64(salt)}${b64(digest)}'

def verify_password(password, stored):
    _, _, params, salt, digest = stored.split('$')
    ln, r, p = (int(item.split('=')[1]) for item in params.split(','))
    candidate = _scrypt(password, base64.b64decode(salt), ln, r, p)
    return hmac.compare_digest(candidate, base64.b64decode(digest))   # constant-time compare

def needs_rehash(stored):
    return stored.split('$')[2] != f'ln={LN},r={R},p={P}'
</code></pre>
<p>The stored string is modeled on the <a href="https://github.com/P-H-C/phc-string-format/blob/master/phc-sf-spec.md">PHC string format</a>, a common standard for this.</p>
<h2>Upgrading hashes you already have</h2>
<p>Settings that are safe today won’t be forever, and you may have old hashes to replace. The most common approach, per OWASP, is to wait until the user next logs in, then re-hash their password:</p>
<p><img src="https://mermaid.ink/img/pako:eNpVUl1v2zAM_Csc8yoDbTFsgB82bC2KDtgH0KwFhqgPikXbmmUxkJi4WZr_XkjughZ6IY_k3YnSARu2hDW2nqemN1Hg95UOAACeOxdWGu8SxZwkcEHjA1TVJ9hRdO1-pfG-BCA9wcakNHG0YDrjQpICJuFIFnqTeo0PM_E8XHh4OGi85Bipkc8aj3MDD7n4pPEna3yCSH-pkZXG2xIU3mLuxHga-EMpT7C3B42_vAXjO45O-hE4wkRmoAiJRFzo0itBb98oWg600viduyK2zRt4LXdqf9GLlC9YHFY5eruPyUkPzTZGCnISBxMsJLMjcHIinnnKZrIFHWY4yd7T_2LrvK8X9KE9a41KEnmgenGx_mjp7CWtJmelry82j6phz7FenJt8UOFIcTTOYn1A6WnMD2-pNVsvqGbk3kRn1p5S7mk5yLUZnd9jjZXZbDxVaZ-ERgVfvQvDD9MsS37NQRRoXFLHBHffNCq45TULK7ghvyNxjVHwJTrjFSQTUpXyN0BVRJbuX_Zy_n7ziMejwnV3mZ1jje-m3gnh8RlPdeT8?type=png" alt="Mermaid Diagram" /></p>
<p>Accounts that never log in keep their old hashes. OWASP suggests two ways to deal with them:</p>
<ul>
<li><p><strong>Expire them.</strong> Delete the old hashes of long-inactive users and make them reset their password. It’s secure, but it can confuse users, who may read it as a sign of a breach.</p>
</li>
<li><p><strong>Wrap them.</strong> Feed the old hash into the new algorithm, for example <code>bcrypt(md5(password))</code>, and replace it with a direct hash at the next login. This protects dormant accounts at once, though layered hashes can be easier to crack.</p>
</li>
</ul>
<h2>Mistakes to avoid</h2>
<ul>
<li><p><strong>Using a fast hash, even with a salt.</strong> MD5, SHA-1 and SHA-256 are all too fast for passwords.</p>
</li>
<li><p><strong>Comparing hashes with</strong> <code>==</code><strong>.</strong> A normal comparison can leak timing information. Use a constant-time comparison, as in the code above.</p>
</li>
<li><p><strong>Pre-hashing for bcrypt with a plain fast hash.</strong> bcrypt ignores anything past 72 bytes, and OWASP warns that wrapping it around a plain SHA-512 is dangerous. If you must pre-hash, it suggests an HMAC keyed with a pepper.</p>
</li>
<li><p><strong>Restricting what users can type.</strong> Accept all Unicode characters and long passphrases, and don’t reduce the entropy of the input before hashing.</p>
</li>
<li><p><strong>Setting the cost once and forgetting it.</strong> Revisit it regularly, and use the rehash-on-login pattern above to apply the change.</p>
</li>
<li><p><strong>Treating the hash as the only defense.</strong> It protects a leaked database, not a live login form. Rate-limit login attempts and offer multi-factor authentication too.</p>
</li>
</ul>
<h2>Measure it yourself</h2>
<p>This script produced the first table. Run it on your own hardware when you tune settings. It times each algorithm for about a second and prints guesses per second, time per hash and how long one core would need for a million guesses.</p>
<p>The Argon2id line needs a recent version of the <code>cryptography</code> package (I used version 46) and is skipped if it can’t be imported.</p>
<pre><code class="language-python">import hashlib, os, time

password = b'correct horse battery staple'
salt = os.urandom(16)

def per_second(fn, seconds=1.0, min_runs=3):
    count, start = 0, time.perf_counter()
    while count &lt; min_runs or time.perf_counter() - start &lt; seconds:
        fn()
        count += 1
    return count / (time.perf_counter() - start)

def human(secs):
    for limit, unit in ((60, 's'), (3600, 'min'), (86400, 'hours'), (365 * 86400, 'days')):
        if secs &lt; limit:
            div = {'s': 1, 'min': 60, 'hours': 3600, 'days': 86400}[unit]
            return f'{secs / div:,.1f} {unit}'
    return f'{secs / (365 * 86400):,.1f} years'

tests = {
    'SHA-256': lambda: hashlib.sha256(salt + password).digest(),
    'PBKDF2-HMAC-SHA256, 600,000 iterations':
        lambda: hashlib.pbkdf2_hmac('sha256', password, salt, 600_000),
    'scrypt, N=2^17, r=8, p=1':
        lambda: hashlib.scrypt(password, salt=salt, n=2**17, r=8, p=1, maxmem=2**28),
}
try:
    from cryptography.hazmat.primitives.kdf.argon2 import Argon2id
    tests['Argon2id, 19 MiB, t=2, p=1'] = lambda: Argon2id(
        salt=salt, length=32, iterations=2, lanes=1, memory_cost=19456).derive(password)
except ImportError:
    pass

print(f'{"algorithm":42} {"guesses/s":&gt;14} {"per hash":&gt;12} {"1M guesses":&gt;14}')
for name, fn in tests.items():
    rate = per_second(fn)
    print(f'{name:42} {rate:&gt;14,.1f} {1000 / rate:&gt;9,.2f} ms {human(1_000_000 / rate):&gt;14}')
</code></pre>
<h2>Wrapping up</h2>
<p>A password hash has one job: make every guess expensive, so a leaked database stays hard to crack. SHA-256 fails at that because it was built to be fast. Argon2id, scrypt, bcrypt and PBKDF2 succeed because they are slow on purpose, and the memory-hard ones also hold back GPUs.</p>
<p>Use Argon2id if you can, measure the cost on your own hardware, and plan to raise it over time.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/password-hashing-explained-why-sha-256-is-the-wrong-tool-and-what-to-use-instead-v36wj?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[SSHole: A 257-Character Password Can Log You In on Alpine Linux]]></title><description><![CDATA[OpenSSH 10.6p2 fixes an authentication bypass on musl-based Linux without PAM: any password of 257 to 1,024 bytes works for accounts whose shadow entry is…]]></description><link>https://blog.zyvop.com/sshole-a-257-character-password-can-log-you-in-on-alpine-linux-4trgp</link><guid isPermaLink="true">https://blog.zyvop.com/sshole-a-257-character-password-can-log-you-in-on-alpine-linux-4trgp</guid><category><![CDATA[Alpine]]></category><category><![CDATA[Linux]]></category><category><![CDATA[Musl]]></category><category><![CDATA[openssh]]></category><category><![CDATA[Security]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Sun, 08 Nov 2026 11:25:42 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/1fbf6ebd-cd16-44af-8208-b0c5236098a3.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>On November 5, 2026, OpenSSH shipped 10.6p2 to close an authentication bypass in sshd on musl-based Linux. On an affected machine, an account that should never accept a password will accept almost any password between 257 and 1,024 bytes long. It takes one connection and works every time.</p>
<p>The bug was reported by Özgün Kültekin of Wakefield, who wrote it up with a great timeline at <a href="https://sshhole.com">sshhole.com</a>. Here's the short version for developers.</p>
<h2>How it works</h2>
<p>Nothing here is a single mistake. Four old, reasonable decisions line up:</p>
<ol>
<li><p><code>*</code> <strong>means "no password".</strong> In <code>/etc/shadow</code>, a field that is just <code>*</code> can never match any hash. This is how key-only users and many service accounts are marked.</p>
</li>
<li><p><strong>sshd borrows a salt.</strong> Without PAM, sshd checks a password by hashing it with <code>crypt()</code> and comparing the result to the stored field. Since the 2016 fix for a user-enumeration timing leak (CVE-2016-6210), if that field is too short to contain a salt, sshd borrows another account's salt so the check takes as long as a real one.</p>
</li>
<li><p><strong>musl's</strong> <code>crypt()</code> <strong>returns</strong> <code>*</code> <strong>on failure.</strong> Where POSIX says to return a null pointer on error, musl returns <code>*</code>. Its <code>$5$</code> and <code>$6$</code> implementations refuse keys longer than 256 bytes (a DoS limit), so an oversized password makes <code>crypt()</code> fail, and fail with <code>*</code>.</p>
</li>
<li><p><strong>Compare</strong> <code>*</code> <strong>to</strong> <code>*</code><strong>.</strong> sshd sees that the "hash" it computed equals the stored field, and lets you in.</p>
</li>
</ol>
<p><img src="https://mermaid.ink/img/pako:eNplkM1Ow0AMhF_F-ATVRuL_kEOkUITggISI4NSLs3HbVTfZYDuUgPruKLQ9cbS_Gc_IP-hTw5ij8sfAnef7QCuhdtEBAPQkFnzoqTMogRRKM_Iblv-4mrDquvmP5hNqB43gZezt9GwvKbOiqHJ4IdVtkgZiWoVukhLMYFAWB1fn5-DXJOSNRfe2KiuKeX441R_MDuokkrbcgFK0Q8I820fM4JRFkjjY8AhBIX2ywOXNLdSjsZ4dD2dFUeZQWRJuYAb8MVBUmLljNe-5N27QYcvSUmgw_0Fbczs9sOElDdHQ7TfvJIHqyDpplqmzB2pDHDHHjPo-cqajGrcO7mLoNs_kq7_5IXXmYIEVrxLD29MCHbymOlly8Mjxky14clBKoOhAqdNMWcIS3V9IFb6nLhfX_Rfudg7r1TzFJJjjyXYdjHH3C8-Fo5o?type=png" alt="Mermaid Diagram" /></p>
<h2>Who is affected</h2>
<p>All of these must be true, per Alpine's advisory:</p>
<ul>
<li><p>sshd runs on musl <strong>without PAM</strong>: Alpine's <code>openssh-server</code>, or <code>openssh-server-pam</code> with <code>UsePAM no</code></p>
</li>
<li><p><code>PasswordAuthentication</code> is <code>yes</code>, which is the OpenSSH default</p>
</li>
<li><p>An account's password field is exactly <code>*</code>: key-only users (created with <code>adduser</code> and no password, or <code>usermod -p '*'</code>), directory users coming through musl-nscd, root (Alpine ships <code>root:*</code>, but the default <code>PermitRootLogin prohibit-password</code> protects it), and the <code>alpine</code> user in tiny-cloud images</p>
</li>
<li><p>At least one <strong>other</strong> local account has a <code>$5$</code> or <code>$6$</code> hash. One is enough, and it makes every password-less account reachable</p>
</li>
</ul>
<p>Not affected: locked accounts (<code>!</code>), systems with <code>PasswordAuthentication no</code>, <code>openssh-server-pam</code> with <code>UsePAM yes</code>, containers that don't run sshd, glibc systems, and Dropbear.</p>
<p>The cloud images deserve a closer look. Official tiny-cloud images from 3.18.11 through 3.24.3 ship the <code>alpine</code> user with passwordless <code>doas</code>. They are safe on first boot, but become exploitable as soon as any <code>$6$</code> hash exists on the machine, for example after someone sets a console password or user-data adds a user with <code>hashed_passwd</code>. From there it's a password bypass, then <code>doas</code> to root.</p>
<h2>What to do</h2>
<p>Update, then restart. Upgrading the package does not restart a running sshd:</p>
<pre><code class="language-bash">apk upgrade -U -a
rc-service sshd restart
</code></pre>
<p>Fixed <code>openssh</code> packages exist for edge and every branch back to 3.16 (for example <code>10.0_p1-r11</code> on 3.22 and <code>9.9_p2-r1</code> on 3.21). The full table is in the advisory.</p>
<p>If you can't update yet, turn off password logins. Keep your current session open and confirm key login works first:</p>
<pre><code class="language-bash">mkdir -p /etc/ssh/sshd_config.d
echo 'PasswordAuthentication no' &gt; /etc/ssh/sshd_config.d/00-no-passwords.conf
sshd -t &amp;&amp; rc-service sshd restart
sshd -T | grep -i passwordauthentication   # expect: passwordauthentication no
</code></pre>
<p>The default config reads <code>sshd_config.d</code> only from 3.19 onward, and existing <code>PasswordAuthentication</code> lines in <code>sshd_config</code>, including inside <code>Match</code> blocks, can still win. The advisory has a <code>sed</code> variant that covers those cases, and the final <code>sshd -T</code> line tells you whether it worked.</p>
<p>Then check for signs of abuse. On Alpine, look for <code>Accepted password for</code> lines in <code>/var/log/messages</code> that name an account with no password:</p>
<pre><code class="language-bash">grep "Accepted password for" /var/log/messages
</code></pre>
<p>If you find one you can't explain, treat the host as compromised: review <code>doas</code> use, new users, changed files, and <code>authorized_keys</code>.</p>
<h2>The takeaway</h2>
<p>OpenSSH's release notes call 10.6p2 a workaround in sshd and say the root cause sits outside it. The advisory said no CVE had been assigned yet; CVE-2026-123158 has since been widely reported, so check NVD for the current record.</p>
<p>The lesson goes beyond SSH: an error return should never be able to look like valid data.</p>
<h2>Sources</h2>
<ul>
<li><p><a href="https://sshhole.com">sshhole.com</a></p>
</li>
<li><p><a href="https://alpinelinux.org/posts/2026-11-05-openssh-password-authentication-bypass/">Alpine Linux security advisory, 2026-11-05</a></p>
</li>
<li><p><a href="https://www.openssh.org/releasenotes.html">OpenSSH release notes, 10.6p2</a></p>
</li>
</ul>
<hr />
<p><em>Published via <a href="https://zyvop.com/sshole-a-257-character-password-can-log-you-in-on-alpine-linux-4trgp?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Clean Code & Design Patterns: Writing Code Other People (and Future You) Can Read]]></title><description><![CDATA[Clean code is about the reader, not cleverness. This post covers naming, guard clauses, pure functions, refactoring with a safety net, SOLID in plain…]]></description><link>https://blog.zyvop.com/clean-code-design-patterns-writing-code-other-people-and-future-you-can-read-nydd6</link><guid isPermaLink="true">https://blog.zyvop.com/clean-code-design-patterns-writing-code-other-people-and-future-you-can-read-nydd6</guid><category><![CDATA[#aiengineer]]></category><category><![CDATA[cleancode]]></category><category><![CDATA[DesignPatterns]]></category><category><![CDATA[refactoring]]></category><category><![CDATA[solid]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Sun, 08 Nov 2026 10:25:40 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/725d039b-1313-4230-874f-c4744d15340a.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Here's something nobody tells beginners. You'll spend far more time <em>reading</em> code than writing it. You read it to fix a bug, to add a feature, to review a teammate's pull request, or to figure out what you wrote three months ago and why. So the real measure of good code isn't how clever it is, or how few lines it uses. It's how quickly another person can understand it and change it safely.</p>
<p>This post is core, and it matters more than ever for AI engineers. AI apps are glue: prompts, tool calls, retries, parsing messy output, switching between providers. That kind of code turns into spaghetti faster than almost any other. And now that AI assistants write a lot of code for us, the skill of <em>reading it critically</em> is worth even more.</p>
<p>Let me say one thing up front, because it's the theme of this whole post: <strong>"clean code" rules are guides, not laws.</strong> People get dogmatic about them, and dogma produces code that's more complicated than the problem. We'll learn the principles, and we'll also learn when not to use them.</p>
<h2>What "clean" means</h2>
<p>Clean code is code that is:</p>
<ul>
<li><p><strong>Easy to read.</strong> A newcomer can follow what it does.</p>
</li>
<li><p><strong>Easy to change.</strong> A new requirement touches one place, not twelve.</p>
</li>
<li><p><strong>Hard to break.</strong> It's tested, and it doesn't surprise you.</p>
</li>
</ul>
<p>A good test: could a teammate, or you in six months, understand this function in about a minute? If not, the code isn't done yet, even if it works.</p>
<h2>Names carry the meaning</h2>
<p>The cheapest way to improve code is better names. A name should tell the reader what something <em>is</em> or <em>does</em>, so they don't have to work it out from the code around it.</p>
<table>
<thead>
<tr>
<th>Unclear</th>
<th>Clearer</th>
</tr>
</thead>
<tbody><tr>
<td>d, data2, tmp</td>
<td>reviews, cleaned_reviews</td>
</tr>
<tr>
<td>process(x)</td>
<td>classify_review(text)</td>
</tr>
<tr>
<td>flag</td>
<td>lowercase, is_streaming</td>
</tr>
<tr>
<td>timeout</td>
<td>timeout_seconds</td>
</tr>
<tr>
<td>check(user)</td>
<td>is_allowed(user)</td>
</tr>
<tr>
<td>lst</td>
<td>prompts</td>
</tr>
</tbody></table>
<p>A few habits:</p>
<ul>
<li><p><strong>Functions are verbs, and things are nouns.</strong> <code>build_prompt</code>, <code>parse_answer</code>, <code>Conversation</code>.</p>
</li>
<li><p><strong>Booleans read like questions.</strong> <code>is_valid</code>, <code>has_tokens</code>, <code>should_retry</code>.</p>
</li>
<li><p><strong>Put units in names.</strong> <code>timeout_seconds</code>, <code>max_tokens</code>, <code>size_mb</code>. It prevents a lot of silly bugs.</p>
</li>
<li><p><strong>Longer scope, longer name.</strong> A loop counter can be <code>i</code>. A function used across the project deserves a descriptive name.</p>
</li>
<li><p><strong>Follow Python's conventions</strong> so your code looks familiar: <code>snake_case</code> for variables and functions, <code>PascalCase</code> for classes, <code>UPPER_CASE</code> for constants.</p>
</li>
<li><p><strong>Don't lie.</strong> A function called <code>get_user</code> that also deletes expired sessions is a trap.</p>
</li>
</ul>
<h2>Functions: small, focused, honest</h2>
<p>From Post 6, a function should do one job. If you have to use "and" to describe it ("it cleans the text <em>and</em> calls the model <em>and</em> saves the result"), it's probably three functions. A few more habits that pay off quickly.</p>
<p><strong>Use guard clauses instead of deep nesting.</strong> Deeply nested <code>if</code> statements push the real logic far to the right, and make you hold a lot in your head. Compare:</p>
<pre><code class="language-python">def shipping_cost_nested(order):
    if order is not None:
        if order["total"] &gt; 0:
            if order["country"] == "IN":
                if order["total"] &gt;= 500:
                    return 0
                else:
                    return 40
            else:
                return 200
        else:
            raise ValueError("empty order")
    else:
        raise ValueError("no order")
</code></pre>
<p>with this version, which does exactly the same thing:</p>
<pre><code class="language-python">def shipping_cost_flat(order):
    if order is None:
        raise ValueError("no order")
    if order["total"] &lt;= 0:
        raise ValueError("empty order")
    if order["country"] != "IN":
        return 200
    if order["total"] &gt;= 500:
        return 0
    return 40
</code></pre>
<p>Handle the bad cases first and leave early. Then the main path reads straight down the page, with no indentation staircase. (I checked that both versions agree on a range of orders, and both raise on the invalid ones.)</p>
<p><strong>Avoid flag arguments.</strong> A parameter like <code>process(data, True)</code> makes the reader look up what <code>True</code> means, and usually means the function does two different things depending on the flag. Split it into two functions, or at least use a keyword argument with a clear name (<code>lowercase=True</code>).</p>
<p><strong>Keep the number of parameters low.</strong> More than three or four is a smell. When several travel together, bundle them into a dataclass or a Pydantic model (Posts 9 and 10).</p>
<p><strong>Be honest about side effects.</strong> A function that looks like a calculation shouldn't quietly change something elsewhere. Functions that just take inputs and return outputs, with no hidden effects, are called <strong>pure</strong>, and they're by far the easiest to test.</p>
<p><strong>Don't mutate what you were given.</strong> If a function modifies the list you passed in, callers get nasty surprises. That's the copy trap from Post 5. Return a new value instead.</p>
<h2>Comments explain <em>why</em></h2>
<p>A comment that repeats the code is noise:</p>
<pre><code class="language-python"># add one to the counter
counter += 1
</code></pre>
<p>A good comment says something the code <em>can't</em>: the reason behind a decision, a trap to avoid, or a link to the issue:</p>
<pre><code class="language-python"># The API rejects batches over 100 items, so we send them in groups.
BATCH_SIZE = 100
</code></pre>
<p>A rule of thumb: if you need a comment to explain <em>what</em> a block does, try extracting it into a function with a good name instead. Then reserve comments for the <em>why</em>. Two more habits:</p>
<ul>
<li><p><strong>Replace magic numbers with named constants.</strong> <code>if retries &gt; 3</code> becomes <code>if retries &gt; MAX_RETRIES</code>.</p>
</li>
<li><p><strong>Keep comments up to date.</strong> A wrong comment is worse than none. When the code changes, the comment must change with it.</p>
</li>
</ul>
<p>Docstrings (Post 6) are for public functions: a one-line summary of what it does, and anything surprising. Type hints (Post 10) double as documentation, and unlike comments, a type checker can verify them.</p>
<h2>Duplication, and the rule of three</h2>
<p>You've heard "DRY: don't repeat yourself." Copy-pasted code is risky, because when you fix a bug in one copy, you forget the others. But there's an opposite mistake, and it's just as common: merging two things that only <em>look</em> similar. The shared function then grows flags and special cases until it's harder to read than the duplication was.</p>
<p>A useful rule is the <strong>rule of three</strong>: the first time, just write it. The second time, you can copy it, and note the resemblance. By the third time, you understand the pattern, so extract it. A little duplication is cheaper than the wrong abstraction.</p>
<h2>Keep logic pure, push the messy stuff to the edges</h2>
<p>This is the most valuable structural habit for AI apps. Code that talks to the outside world, like a model call, the network, the disk, or the clock, is slow, costs money, and gives different answers every time. Code that just transforms data is fast and predictable.</p>
<p>So separate them. Put the <strong>logic</strong> in pure functions: building prompts, parsing replies, validating, formatting. Then keep the actual <strong>model call</strong> in one thin layer at the edge. Look at how this changes a bot:</p>
<pre><code class="language-python">import json

def build_prompt(question):
    return f"Answer in JSON with key 'answer'. Question: {question}"

def parse_answer(raw):
    try:
        return json.loads(raw)["answer"]
    except (json.JSONDecodeError, KeyError, TypeError):
        return raw

class Bot:
    def __init__(self, model, log):
        self.model = model
        self.log = log

    def run(self, question):
        answer = parse_answer(self.model.generate(build_prompt(question)))
        self.log(question, answer)
        return answer
</code></pre>
<p><code>build_prompt</code> and <code>parse_answer</code> are pure, so you can test them with plain assertions and no model:</p>
<pre><code class="language-python">assert parse_answer('{"answer": "yes"}') == "yes"
assert parse_answer("not json") == "not json"
assert parse_answer('{"other": 1}') == '{"other": 1}'
</code></pre>
<p>And <code>Bot</code> receives its model and its logger as parameters, so in a test you can hand it a fake:</p>
<pre><code class="language-python">class FakeModel:
    def generate(self, prompt):
        return '{"answer": "42"}'

logged = []
bot = Bot(FakeModel(), lambda question, answer: logged.append((question, answer)))
print(bot.run("What is 6 x 7?"))    # 42
print(logged)                       # [('What is 6 x 7?', '42')]
</code></pre>
<p>Compare that with a single class that builds the prompt, calls a client it creates itself, parses JSON, <em>and</em> writes a log file inside one method. Testing it would mean running the real client and cleaning up files. That's the difference between code that's easy to test and code that isn't, and we'll come back to it under "SOLID" below.</p>
<h2>Let tools do the boring part</h2>
<p>Arguing about spacing and quote styles is a waste of human attention. Use tools:</p>
<ul>
<li><p>A <strong>formatter</strong> rewrites your code into a consistent style automatically. Two popular choices are <code>black</code> and the formatter built into <code>ruff</code>.</p>
</li>
<li><p>A <strong>linter</strong> flags likely bugs and bad habits: unused variables, missing imports, risky patterns. <code>ruff</code> is a fast, popular one.</p>
</li>
<li><p>A <strong>type checker</strong>, like <code>mypy</code> or <code>pyright</code>, uses your type hints (Post 10) to find mistakes before you run anything.</p>
</li>
</ul>
<p>Set them up once, run them on every commit with a tool like <code>pre-commit</code> (Post 22), and run them in CI (Post 28). Then code review can focus on the interesting things. (These tools aren't installed in my test environment, so I haven't run them here. Their documentation walks through setup, and it's a ten-minute job.)</p>
<h2>Refactoring: change the structure, not the behavior</h2>
<p><strong>Refactoring</strong> means improving how code is organized <em>without changing what it does</em>. It's not rewriting, and it's not adding features. It's tidying, safely. The safe part is everything.</p>
<p>The rules:</p>
<ol>
<li><p><strong>Have tests first.</strong> Without them, you can't know whether you broke something. If there aren't any, write tests that capture what the code does <em>now</em>, even if it's imperfect. These are called <strong>characterization tests</strong>.</p>
</li>
<li><p><strong>Take small steps.</strong> Rename something. Run the tests. Extract a function. Run the tests.</p>
</li>
<li><p><strong>Commit after each step</strong> (Post 22), so you can always go back.</p>
</li>
<li><p><strong>Never mix refactoring with adding features.</strong> Do one, commit, then do the other.</p>
</li>
</ol>
<h3>A worked example</h3>
<p>Here's a function that works, and that nobody wants to touch. It takes a list of customer reviews and a model that classifies each one:</p>
<pre><code class="language-python">def process(d, m, f=False):
    r = []
    for x in d:
        if x is not None:
            if len(x.strip()) &gt; 0:
                if f:
                    t = x.strip().lower()
                else:
                    t = x.strip()
                try:
                    o = m(t)
                    if o == "pos" or o == "neg" or o == "neu":
                        r.append((t, o))
                    else:
                        r.append((t, "unknown"))
                except Exception:
                    r.append((t, "error"))
    c = {}
    for a, b in r:
        if b in c:
            c[b] = c[b] + 1
        else:
            c[b] = 1
    return r, c
</code></pre>
<p>Let's list what's wrong, the way a reviewer would:</p>
<ul>
<li><p><strong>Names.</strong> <code>d</code>, <code>m</code>, <code>f</code>, <code>r</code>, <code>x</code>, <code>t</code>, <code>o</code>, <code>c</code>, <code>a</code>, <code>b</code>. Nothing explains itself.</p>
</li>
<li><p><strong>A flag argument.</strong> What does <code>f=True</code> mean? You have to read the body to find out.</p>
</li>
<li><p><strong>Deep nesting.</strong> Four levels of <code>if</code> before anything happens.</p>
</li>
<li><p><strong>Three jobs in one function.</strong> Cleaning the input, classifying it, and counting results.</p>
</li>
<li><p><strong>A repeated condition.</strong> <code>o == "pos" or o == "neg" or o == "neu"</code>.</p>
</li>
<li><p><strong>A hand-written counter</strong>, when Python has <code>Counter</code> (Post 16).</p>
</li>
<li><p><strong>A</strong> <code>try</code> <strong>wrapped around more than it needs to</strong> (Post 11). The only thing that can fail is the model call.</p>
</li>
</ul>
<p>First, the safety net. Let me build a fake model to test with, and a set of awkward inputs:</p>
<pre><code class="language-python">def fake_model(text):
    lowered = text.lower()
    if "boom" in lowered:
        raise RuntimeError("model failed")
    if "good" in lowered:
        return "pos"
    if "bad" in lowered:
        return "neg"
    if "ok" in lowered:
        return "neu"
    return "meh"

sample = ["  Good service ", None, "   ", "BAD food", "BOOM", "it was ok", "no idea", "", "Great GOOD stuff"]
</code></pre>
<p>The sample includes a <code>None</code>, blanks, whitespace, a failing input, a valid label, and an invalid label (<code>"meh"</code>). Run the original, and you get:</p>
<pre><code class="language-typescript">([('Good service', 'pos'), ('BAD food', 'neg'), ('BOOM', 'error'), ('it was ok', 'neu'),
  ('no idea', 'unknown'), ('Great GOOD stuff', 'pos')],
 {'pos': 2, 'neg': 1, 'error': 1, 'neu': 1, 'unknown': 1})
</code></pre>
<p>That's our "this is what correct looks like" snapshot. Now refactor in small steps: rename, remove the nesting with early <code>continue</code>, extract the classification, replace the hand-written counter, and split the three jobs. This is where I ended up:</p>
<pre><code class="language-python">from collections import Counter

VALID_LABELS = {"pos", "neg", "neu"}

def clean_reviews(reviews, lowercase=False):
    """Strip whitespace, drop blanks and None, and optionally lowercase."""
    for review in reviews:
        if review is None:
            continue
        text = review.strip()
        if not text:
            continue
        yield text.lower() if lowercase else text

def classify(model, text):
    """Ask the model for a label. Bad answers become 'unknown', failures become 'error'."""
    try:
        label = model(text)
    except Exception:
        return "error"
    return label if label in VALID_LABELS else "unknown"

def classify_reviews(model, reviews, lowercase=False):
    return [(text, classify(model, text)) for text in clean_reviews(reviews, lowercase)]

def count_labels(results):
    return dict(Counter(label for _, label in results))

def process_reviews(model, reviews, lowercase=False):
    results = classify_reviews(model, reviews, lowercase)
    return results, count_labels(results)
</code></pre>
<p>Every function does one thing, with a name that says what. <code>clean_reviews</code> is a generator (Post 7), and uses guard clauses (<code>continue</code>) instead of nesting. <code>classify</code> wraps only the model call in <code>try</code>. The valid labels live in a named constant instead of a repeated condition. <code>Counter</code> does the counting. And the original function's job is now four small ones that are easy to test separately.</p>
<h3>Proving it didn't change</h3>
<p>Here's the key step: <strong>did we change the behavior?</strong> On the sample input, the new version returns exactly the same results, including with the lowercase option on. But a handful of examples isn't proof, so I also ran a <strong>differential test</strong>: generate lots of random inputs, run <em>both</em> versions, and check they match:</p>
<pre><code class="language-python">import random

random.seed(11)
words = ["good", "bad", "ok", "boom", "hmm", "Great", "BAD", " ", "", "x y"]

for _ in range(2000):
    reviews = [
        random.choice(words + [None]) if random.random() &lt; 0.5
        else " ".join(random.choices(words[:6], k=random.randint(1, 3)))
        for _ in range(random.randint(0, 8))
    ]
    flag = random.random() &lt; 0.5
    assert process(reviews, fake_model, flag) == process_reviews(fake_model, reviews, flag)

print("2000 random inputs, identical results")
</code></pre>
<p>It passed on all 2,000. That's the property-based thinking from Post 13, applied to refactoring: the old function <em>is</em> the specification, and the new one has to agree with it. Once you trust the new version, delete the old one and keep a few of the sample cases as permanent tests.</p>
<p>Notice what we still haven't fixed. The <code>lowercase</code> flag remains a flag argument, and the label strings are still plain strings. We could make them an <code>Enum</code> next. Refactoring is never "finished". You stop when the code is clear enough for the people who work on it, and no further.</p>
<h3>Common refactorings</h3>
<p>You've just used most of them, and here's the short menu:</p>
<ul>
<li><p><strong>Rename</strong> a variable, function, or class to say what it is.</p>
</li>
<li><p><strong>Extract function</strong>: pull a chunk of code out and give it a name.</p>
</li>
<li><p><strong>Extract variable</strong>: give a confusing expression a name.</p>
</li>
<li><p><strong>Replace magic numbers</strong> with constants.</p>
</li>
<li><p><strong>Replace nested ifs</strong> with guard clauses.</p>
</li>
<li><p><strong>Replace an</strong> <code>if/elif</code> <strong>chain</strong> with a dictionary or registry (more below).</p>
</li>
<li><p><strong>Introduce a parameter object</strong>: bundle several parameters into a dataclass.</p>
</li>
<li><p><strong>Delete dead code.</strong> Git remembers it (Post 22), so you can always get it back.</p>
</li>
</ul>
<h2>SOLID, in plain English</h2>
<p>You'll hear five principles named <strong>SOLID</strong> whenever people talk about object-oriented design. Here's each in plain terms, with the caveat that they're heuristics, not commandments. In a 50-line script, most of them are overkill. They earn their keep as code grows and changes.</p>
<h3>S: Single Responsibility</h3>
<p><em>A piece of code should have one reason to change.</em> Our <code>Bot</code> above has one job: coordinate. If the prompt wording changes, you edit <code>build_prompt</code>. If the reply format changes, you edit <code>parse_answer</code>. If logging moves to a database, you change the <code>log</code> function you pass in. Nothing else is touched. A class that builds prompts, calls the API, parses JSON, <em>and</em> writes log files has four reasons to change, and every change risks breaking the other three.</p>
<h3>O: Open/Closed</h3>
<p><em>You should be able to add new behavior without editing existing code.</em> The classic violation is an <code>if/elif</code> chain that grows every time you add a case:</p>
<pre><code class="language-python">def tool_ifelse(name, arg):
    if name == "upper":
        return arg.upper()
    elif name == "reverse":
        return arg[::-1]
    else:
        raise ValueError(f"unknown tool {name}")
</code></pre>
<p>Every new tool means editing this function. Instead, use the registry decorator from Post 8:</p>
<pre><code class="language-python">TOOLS = {}

def tool(func):
    TOOLS[func.__name__] = func
    return func

@tool
def upper(arg):
    return arg.upper()

@tool
def reverse(arg):
    return arg[::-1]

@tool
def count_words(arg):    # new behavior, and no existing code was touched
    return len(arg.split())

print(TOOLS["count_words"]("a b c"))    # 3
</code></pre>
<p>Adding <code>count_words</code> meant writing one new function. The dispatch code didn't change. This is exactly how the tool lists in agent frameworks work.</p>
<h3>L: Liskov Substitution</h3>
<p><em>A subclass should work anywhere its parent does, without surprises.</em> If your code accepts a <code>Model</code>, any kind of <code>Model</code> should behave like one:</p>
<pre><code class="language-python">class Model:
    def generate(self, prompt):
        return "answer"

class OfflineModel(Model):
    def generate(self, prompt):
        raise RuntimeError("no network")    # breaks every caller that expects an answer
</code></pre>
<p><code>OfflineModel</code> <em>is a</em> <code>Model</code> on paper, but a function that calls <code>generate</code> and expects an answer will blow up. That's a violation. If a subtype can't honor the contract, it probably shouldn't be a subtype. The fix is usually composition and protocols (Post 9), which describe the behavior a caller needs, not a family tree.</p>
<h3>I: Interface Segregation</h3>
<p><em>Don't force code to depend on things it doesn't use.</em> Suppose you have a giant interface with <code>generate</code>, <code>embed</code>, <code>fine_tune</code>, and <code>moderate</code>. A summarizer that only needs <code>generate</code> would force every model it works with to implement all four. Keep interfaces small and focused:</p>
<pre><code class="language-python">from typing import Protocol

class Generator(Protocol):
    def generate(self, prompt: str) -&gt; str: ...

def summarize(model: Generator, text: str) -&gt; str:
    return model.generate(text)
</code></pre>
<p>Now anything with a <code>generate</code> method works, including a three-line fake in a test.</p>
<h3>D: Dependency Inversion (and injection)</h3>
<p><em>High-level logic shouldn't decide which low-level tools it uses. It should be handed them.</em> Compare:</p>
<pre><code class="language-python">class HardWired:
    def __init__(self):
        self.model = RealClient()    # decides its own dependency: hard to test

    def run(self, q):
        return self.model.generate(q)

class Injected:
    def __init__(self, model):
        self.model = model           # told its dependency: easy to swap

    def run(self, q):
        return self.model.generate(q)
</code></pre>
<p><code>HardWired</code> creates its own real client, so you can't test it without hitting the network. <code>Injected</code> takes whatever model you give it, whether that's the real one, a fake, or a cached wrapper. Passing dependencies in this way is called <strong>dependency injection</strong>, and it's the single most useful idea here. It's what made the fake models in Post 13 possible.</p>
<h2>Design patterns you'll actually use</h2>
<p><strong>Design patterns</strong> are named solutions to problems that keep coming up. Most of the famous list, such as Abstract Factory and Visitor, you'll rarely meet in Python. A handful show up all the time in AI code, and Python's functions-as-values make them lighter than in other languages.</p>
<h3>Strategy: swap the algorithm</h3>
<p>When you can do the same job in several ways, make the "way" something you pass in. In Python, a strategy is often just a function. Here are two ways to chunk text (Post 17):</p>
<pre><code class="language-python">def chunk_by_words(text, size=8):
    words = text.split()
    return [" ".join(words[i:i + size]) for i in range(0, len(words), size)]

def chunk_by_sentences(text, max_chars=60):
    sentences = [s.strip() + "." for s in text.split(".") if s.strip()]
    chunks, current = [], ""
    for sentence in sentences:
        if current and len(current) + 1 + len(sentence) &gt; max_chars:
            chunks.append(current)
            current = sentence
        else:
            current = f"{current} {sentence}".strip()
    if current:
        chunks.append(current)
    return chunks
</code></pre>
<p>A summarizer can take either one as a parameter. Change the strategy, not the summarizer.</p>
<h3>Adapter: make different things look the same</h3>
<p>Different model providers return differently shaped data. One gives you <code>{"text": ...}</code>, another buries the reply inside <code>{"choices": [{"message": {"content": ...}}]}</code>. (Those shapes are made up here, but real providers differ in just this way.) An <strong>adapter</strong> wraps each provider so that your code sees one consistent interface:</p>
<pre><code class="language-python">class ProviderAAdapter:
    def __init__(self, client):
        self.client = client

    def generate(self, prompt):
        return self.client.call(prompt)["text"]

class ProviderBAdapter:
    def __init__(self, client):
        self.client = client

    def generate(self, prompt):
        response = self.client.complete([{"role": "user", "content": prompt}])
        return response["choices"][0]["message"]["content"]
</code></pre>
<p>The messy differences live in two small classes. Everything else just calls <code>generate(prompt)</code>. Switching providers later means writing one new adapter, not hunting through your whole codebase.</p>
<h3>Wrapper (the decorator pattern): add behavior around something</h3>
<p>Remember the <code>@retry</code> decorator from Post 8? There's an object-oriented cousin. A wrapper takes a model and returns something that <em>looks like a model</em>, but adds behavior, such as retries or caching:</p>
<pre><code class="language-python">class CachedModel:
    def __init__(self, model):
        self.model = model
        self.cache = {}
        self.hits = 0

    def generate(self, prompt):
        key = hashlib.sha256(prompt.encode("utf-8")).hexdigest()
        if key in self.cache:
            self.hits += 1
            return self.cache[key]
        answer = self.model.generate(prompt)
        self.cache[key] = answer
        return answer
</code></pre>
<p>Because a <code>CachedModel</code> has the same <code>generate</code> method as the thing it wraps, you can stack wrappers freely: a cache around a retrier around an adapter around a client. Each layer does one thing.</p>
<h3>Registry (a simple factory): build by name</h3>
<p>When config says <code>model: provider_b</code>, you need to turn that string into an object. A registry maps names to builders, and the decorator from Post 8 fills it in, as in the tools example above. Adding a provider means registering one more builder, and nothing else changes.</p>
<h3>Patterns to be wary of</h3>
<p>The <strong>Singleton</strong> (exactly one global instance) is usually a disguise for global mutable state, which makes testing hard. In Python, a module-level object already does the job, and injecting dependencies is cleaner. And remember, patterns are a vocabulary for talking about solutions. They are not a checklist. Don't build an abstract factory for a 50-line script.</p>
<h2>Smells, and the over-engineering trap</h2>
<p>A <strong>code smell</strong> is a sign that something deserves a closer look. It doesn't prove there's a bug. Watch for:</p>
<ul>
<li><p><strong>A function over about 30 lines</strong>, or one that does several jobs.</p>
</li>
<li><p><strong>Deep nesting</strong>, more than two or three levels.</p>
</li>
<li><p><strong>Long parameter lists</strong> and <strong>flag arguments.</strong></p>
</li>
<li><p><strong>Copy-pasted blocks.</strong></p>
</li>
<li><p><strong>Magic numbers and magic strings.</strong> Use constants, an <code>Enum</code>, or <code>Literal</code> (Post 10). With <code>Enum</code>, typos fail loudly: <code>Role("robot")</code> raises <code>'robot' is not a valid Role</code>, where a plain string would sail through.</p>
</li>
<li><p><strong>A "god class"</strong> that knows and does everything.</p>
</li>
<li><p><strong>Swallowed exceptions</strong>, like <code>except: pass</code> (Post 11).</p>
</li>
<li><p><strong>Hidden global state.</strong></p>
</li>
<li><p><strong>Deep inheritance trees</strong> (Post 9). Prefer composition.</p>
</li>
<li><p><strong>Cleverness</strong>: a dense one-liner that takes five minutes to decode.</p>
</li>
</ul>
<p>The opposite failure is just as real: <strong>over-engineering.</strong> It looks like an interface with one implementation, a factory that builds one thing, five layers of indirection, and configuration for flexibility nobody asked for. The principle against it is <strong>YAGNI</strong>, "you aren't gonna need it." Build the simple thing, and refactor when real pain shows up. Abstractions should be earned by actual duplication or actual change, not predicted.</p>
<p>A good guide: <strong>write the simplest code that works, then clean it up when it starts to hurt.</strong> The rule of three, and your tests, tell you when.</p>
<h2>Reading code that an AI wrote</h2>
<p>AI assistants will write a lot of your code, and you're responsible for every line that ships. Hold it to the same standard as anyone else's code:</p>
<ul>
<li><p><strong>Read all of it.</strong> If you can't explain a line, don't ship it. Ask the assistant to explain it, then check the answer.</p>
</li>
<li><p><strong>Run it.</strong> Assistants sometimes invent functions or options that don't exist, or use outdated library versions. Running the code, and reading the official docs, catches that.</p>
</li>
<li><p><strong>Look for over-engineering.</strong> Assistants love adding classes, layers, and "flexibility". Ask whether a ten-line function would do.</p>
</li>
<li><p><strong>Look for swallowed errors.</strong> A blanket <code>try/except</code> that returns a default makes failures invisible (Post 11).</p>
</li>
<li><p><strong>Check the dependencies.</strong> Did it quietly add a library? Is it real, and maintained (Post 12)?</p>
</li>
<li><p><strong>Ask for tests, then read the tests</strong>, and check that they'd fail if the code were wrong.</p>
</li>
<li><p><strong>Refactor it to your own standards</strong>, using the steps in this post.</p>
</li>
</ul>
<p>AI is also good at <em>helping</em> with this post's skills: ask it to review your function for readability, suggest better names, or point out duplication. Treat its suggestions like a colleague's: useful, and still your decision.</p>
<h2>Mini project: a provider-agnostic summarizer</h2>
<p>Let's build one small system that uses most of the ideas: injection, adapters, wrappers, a registry, and strategies. It summarizes long text with a model, and works with <em>any</em> provider through one interface. Everything runs on fake providers, so there's no API key or network. Create <code>llm_client.py</code>:</p>
<pre><code class="language-python">import hashlib
import time
from dataclasses import dataclass
from typing import Callable, Protocol

class TextGenerator(Protocol):
    def generate(self, prompt: str) -&gt; str: ...

# ---- two pretend provider SDKs, with different shapes -------------------

class ProviderA:
    """A pretend SDK that returns {"text": ...}."""

    def __init__(self):
        self.calls = 0

    def call(self, prompt):
        self.calls += 1
        return {"text": f"[A] {prompt[:40]}"}

class ProviderB:
    """A pretend SDK that returns a nested structure."""

    def __init__(self):
        self.calls = 0

    def complete(self, messages):
        self.calls += 1
        content = messages[-1]["content"]
        return {"choices": [{"message": {"content": f"[B] {content[:40]}"}}]}

# ---- adapters: make both look like a TextGenerator -----------------------

class ProviderAAdapter:
    def __init__(self, client):
        self.client = client

    def generate(self, prompt):
        return self.client.call(prompt)["text"]

class ProviderBAdapter:
    def __init__(self, client):
        self.client = client

    def generate(self, prompt):
        response = self.client.complete([{"role": "user", "content": prompt}])
        return response["choices"][0]["message"]["content"]

# ---- wrappers: add behavior without changing the interface ---------------

class RetryingModel:
    def __init__(self, model, attempts=3, delay_seconds=0.0):
        self.model = model
        self.attempts = attempts
        self.delay_seconds = delay_seconds

    def generate(self, prompt):
        for attempt in range(1, self.attempts + 1):
            try:
                return self.model.generate(prompt)
            except TimeoutError:
                if attempt == self.attempts:
                    raise
                time.sleep(self.delay_seconds * attempt)

class CachedModel:
    def __init__(self, model):
        self.model = model
        self.cache = {}
        self.hits = 0

    def generate(self, prompt):
        key = hashlib.sha256(prompt.encode("utf-8")).hexdigest()
        if key in self.cache:
            self.hits += 1
            return self.cache[key]
        answer = self.model.generate(prompt)
        self.cache[key] = answer
        return answer

# ---- a registry: build models by name ------------------------------------

BUILDERS = {}

def register(name):
    def decorator(builder):
        BUILDERS[name] = builder
        return builder
    return decorator

@register("provider_a")
def build_provider_a():
    return ProviderAAdapter(ProviderA())

@register("provider_b")
def build_provider_b():
    return ProviderBAdapter(ProviderB())

def build_model(name, retries=3, cache=True):
    if name not in BUILDERS:
        raise ValueError(f"Unknown model {name!r}. Known models: {sorted(BUILDERS)}")
    model = RetryingModel(BUILDERS[name](), attempts=retries)
    return CachedModel(model) if cache else model

# ---- strategies: interchangeable ways to split text ---------------------

def chunk_by_words(text, size=8):
    words = text.split()
    return [" ".join(words[i:i + size]) for i in range(0, len(words), size)]

def chunk_by_sentences(text, max_chars=60):
    sentences = [s.strip() + "." for s in text.split(".") if s.strip()]
    chunks, current = [], ""
    for sentence in sentences:
        if current and len(current) + 1 + len(sentence) &gt; max_chars:
            chunks.append(current)
            current = sentence
        else:
            current = f"{current} {sentence}".strip()
    if current:
        chunks.append(current)
    return chunks

# ---- the part that does the real work, free of provider details ---------

@dataclass
class Summarizer:
    model: TextGenerator
    chunker: Callable[[str], list[str]]

    def summarize(self, text):
        chunks = self.chunker(text)
        partials = [self.model.generate(f"Summarize: {chunk}") for chunk in chunks]
        if len(partials) == 1:
            return partials[0]
        return self.model.generate("Combine: " + " | ".join(partials))

class FlakyModel:
    """Fails with TimeoutError for the first few calls. Handy for tests."""

    def __init__(self, failures):
        self.failures = failures
        self.calls = 0

    def generate(self, prompt):
        self.calls += 1
        if self.calls &lt;= self.failures:
            raise TimeoutError("slow")
        return "ok: " + prompt

if __name__ == "__main__":
    text = (
        "Cats sleep a lot. They nap in the sun. Dogs need walks every day. "
        "Birds sing in the morning. Fish are quiet pets."
    )

    print("Chunks by words:    ", chunk_by_words(text, size=8))
    print("Chunks by sentences:", chunk_by_sentences(text, max_chars=60))

    client = ProviderA()
    model = CachedModel(RetryingModel(ProviderAAdapter(client)))
    summarizer = Summarizer(model, chunk_by_sentences)

    print("\nFirst run: ", summarizer.summarize(text))
    print("Provider calls:", client.calls, "| cache hits:", model.hits)
    summarizer.summarize(text)
    print("After a second identical run -&gt; provider calls:", client.calls, "| cache hits:", model.hits)

    print("\nSame summarizer, different provider and chunking strategy:")
    other = Summarizer(build_model("provider_b"), lambda t: chunk_by_words(t, size=10))
    print(other.summarize(text))

    flaky = FlakyModel(failures=2)
    print("\nRetrying wrapper:", RetryingModel(flaky, attempts=3).generate("hello"), "| calls:", flaky.calls)
    try:
        RetryingModel(FlakyModel(failures=5), attempts=2).generate("hello")
    except TimeoutError:
        print("Gave up after 2 attempts, as expected")

    try:
        build_model("provider_c")
    except ValueError as error:
        print("\nError:", error)
</code></pre>
<p>Run it, and you should see:</p>
<pre><code class="language-yaml">Chunks by words:     ['Cats sleep a lot. They nap in the', 'sun. Dogs need walks every day. Birds sing', 'in the morning. Fish are quiet pets.']
Chunks by sentences: ['Cats sleep a lot. They nap in the sun.', 'Dogs need walks every day. Birds sing in the morning.', 'Fish are quiet pets.']

First run:  [A] Combine: [A] Summarize: Cats sleep a lot
Provider calls: 4 | cache hits: 0
After a second identical run -&gt; provider calls: 4 | cache hits: 4

Same summarizer, different provider and chunking strategy:
[B] Combine: [B] Summarize: Cats sleep a lot

Retrying wrapper: ok: hello | calls: 3
Gave up after 2 attempts, as expected

Error: Unknown model 'provider_c'. Known models: ['provider_a', 'provider_b']
</code></pre>
<p>(The fake providers just echo the first 40 characters of the prompt with a tag, so the "summaries" aren't clever. That's fine. We're testing structure, not language.)</p>
<p>Look at what the design bought us:</p>
<ul>
<li><p><code>Summarizer</code> <strong>knows nothing about providers.</strong> It needs something with <code>generate</code>, and a function that splits text. That's dependency injection plus the strategy pattern.</p>
</li>
<li><p><strong>Switching provider took one name.</strong> The second summarizer uses <code>provider_b</code> and a different chunking strategy, and no code in <code>Summarizer</code> changed.</p>
</li>
<li><p><strong>The wrappers stack.</strong> The cache sits around the retrier, which sits around the adapter, which sits around the client. On the first run, there were 4 provider calls (three chunk summaries plus the final combine). On the second identical run there were still 4, and 4 cache hits, because every prompt was answered from the cache. In real life that's saved money and time.</p>
</li>
<li><p><strong>Testing is easy.</strong> <code>FlakyModel</code> lets us check the retry logic in milliseconds: it succeeded on the third call, and gave up when the attempts ran out. The same fake-model approach from Post 13 applies to everything here.</p>
</li>
<li><p><strong>Errors help.</strong> Asking for an unknown model tells you which ones exist.</p>
</li>
</ul>
<p>Is this over-engineered for what it does? Honestly, for a single provider and a single script, yes. It earns its structure when you support several providers, want caching and retries, and write tests. That's the "start simple, and refactor when it hurts" balance. When you reach that point, you know the shape to move toward.</p>
<h3>Make it yours</h3>
<ol>
<li><p>Add a <code>LoggingModel</code> wrapper that prints each prompt and response, and stack it with the others.</p>
</li>
<li><p>Write tests for <code>CachedModel</code>, <code>RetryingModel</code>, and both chunkers, using the techniques from Post 13.</p>
</li>
<li><p>Add a third provider with yet another response shape, and register it.</p>
</li>
<li><p>Replace the <code>"user"</code>/<code>"assistant"</code> role strings you've used in earlier posts with an <code>Enum</code>.</p>
</li>
<li><p>Add a <code>max_chars</code> safety check so <code>chunk_by_sentences</code> handles a single sentence that's longer than the limit. Decide for yourself what it should do, and write a test.</p>
</li>
<li><p>Take the retry wrapper and make it only retry the errors you choose, passed in as a parameter.</p>
</li>
</ol>
<h2>Practice</h2>
<p>About 20 minutes of real effort on each before asking for help.</p>
<ol>
<li><p>Find the oldest function you've written in this series and improve its names. Don't change anything else, and check your tests still pass.</p>
</li>
<li><p>Write a function with four nested <code>if</code> statements, then rewrite it with guard clauses. Keep its behavior exactly the same, and prove it with a handful of test inputs.</p>
</li>
<li><p>Take a function over 30 lines from one of your projects. Write characterization tests first, then split it into smaller functions.</p>
</li>
<li><p>Replace an <code>if/elif</code> chain in your own code with a dictionary or a registry.</p>
</li>
<li><p>Write an adapter for an API response shape that you invent. Then write a test that uses it through the <code>generate</code> interface.</p>
</li>
<li><p>Rewrite a class that creates its own dependency so that it receives it instead. Then test it with a fake.</p>
</li>
<li><p>Review this smell list against a project of yours, and pick the three worst offenders.</p>
</li>
<li><p>Ask an AI assistant to write a small function, then review it using the checklist above. What would you change?</p>
</li>
</ol>
<p>Then ask an AI tutor to review your answers, and don't let it rewrite them.</p>
<h2>What to take away</h2>
<ul>
<li><p>Code is read far more than it's written. Optimize for the reader.</p>
</li>
<li><p>Good names do most of the work. Functions are small and honest, and guard clauses beat deep nesting.</p>
</li>
<li><p>Comments explain <em>why</em>. Wait for the third repetition before abstracting.</p>
</li>
<li><p>Keep logic pure, and push model calls and I/O to the edges, so most of your code is easy to test.</p>
</li>
<li><p>Refactor in small steps with tests as a safety net, and prove you didn't change the behavior.</p>
</li>
<li><p>SOLID is a set of heuristics: one reason to change, extend without editing, honor your contracts, keep interfaces small, and inject dependencies.</p>
</li>
<li><p>Strategy, adapter, wrapper, and registry cover a surprising amount of AI code.</p>
</li>
<li><p>Watch for smells, and also for over-engineering. Start simple, and add structure when it hurts.</p>
</li>
<li><p>Hold AI-written code to the same standard as anyone's. You're responsible for what ships.</p>
</li>
</ul>
<hr />
<p><strong>Previous:</strong> Post 23, <em>Linux, Shell &amp; Docker</em></p>
<p><strong>Next in the series:</strong> Post 25, <em>TypeScript for AI Engineers</em>, because most AI products have a web front end, and many agent SDKs are TypeScript-first.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/clean-code-design-patterns-writing-code-other-people-and-future-you-can-read-nydd6?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[How to Secure AI Coding Agents: Sandboxing and Permissions]]></title><description><![CDATA[AI coding agents run commands, install packages, and read untrusted text all day. This guide covers the threat model, layered sandboxing, allowlist-based…]]></description><link>https://blog.zyvop.com/how-to-secure-ai-coding-agents-sandboxing-and-permissions-o0bgz</link><guid isPermaLink="true">https://blog.zyvop.com/how-to-secure-ai-coding-agents-sandboxing-and-permissions-o0bgz</guid><category><![CDATA[#aisecurity]]></category><category><![CDATA[codingagents]]></category><category><![CDATA[permissions]]></category><category><![CDATA[PromptInjection]]></category><category><![CDATA[sandboxing]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Sun, 08 Nov 2026 07:57:46 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/99bd46b5-e654-46bf-a6ea-3870c37bb1e0.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>AI coding agents don't just suggest code anymore. They run shell commands, install packages, edit files across your repo, call APIs, and open pull requests. That's what makes them useful, and it's also what makes them risky.</p>
<p>Think of a coding agent as a very fast junior engineer who has access to your laptop or CI runner, never gets tired, and reads untrusted text all day: READMEs, issue comments, web pages, dependency docs, tool output. Some of that text may be written by someone who wants to hijack it.</p>
<p>The good news is that securing agents is mostly classic security engineering: least privilege, isolation, and defense in depth. The key is applying those ideas at the right layers. This post walks through a threat model, how to build sandboxes, how to design permissions that people won't just click through, and what to do about the parts that can't be fully solved yet.</p>
<h2>What are you defending against?</h2>
<p>Start with a realistic threat model. Most agent incidents fall into five buckets.</p>
<p><strong>Prompt injection.</strong> Malicious instructions hidden in content the agent reads: a GitHub issue, a code comment, a web page, an MCP tool response, even a package README. The agent can't reliably tell "data to analyze" from "instructions to follow."</p>
<p><strong>Honest mistakes.</strong> No attacker needed. An agent that misreads a task can delete the wrong directory, force-push over a branch, run a migration against the wrong database, or "fix" a failing test by deleting it.</p>
<p><strong>Credential and data exfiltration.</strong> Environment variables, SSH keys, cloud credentials, <code>.env</code> files, and source code are all within reach of anything that can run a command and make a network request.</p>
<p><strong>Supply chain attacks.</strong> Agents install dependencies quickly and confidently. Models sometimes suggest package names that don't exist, and attackers register those names (sometimes called slopsquatting). Install scripts can then run arbitrary code.</p>
<p><strong>Escalation and persistence.</strong> An agent that can edit its own config, git hooks, CI workflows, or build scripts can quietly widen its own access or leave something behind that runs later, outside any supervision.</p>
<h3>The lethal trifecta</h3>
<p>A useful mental model, coined by Simon Willison in 2025, is the "lethal trifecta." An agent becomes dangerous to data when it combines three things at once:</p>
<ol>
<li><p>Access to private data (your code, secrets, internal systems)</p>
</li>
<li><p>Exposure to untrusted content (anything an outsider can influence)</p>
</li>
<li><p>A way to communicate externally (network access, sending messages, opening PRs)</p>
</li>
</ol>
<p>If all three are present, a prompt injection can read your secrets and ship them out. You often can't eliminate the second leg, because useful agents read the web and third-party code. So the practical strategy is to break one of the other two legs, most commonly by restricting network egress and limiting what data is reachable in the first place.</p>
<h2>The core principle: permissions are policy, sandboxes are enforcement</h2>
<p>This distinction drives everything else.</p>
<p><strong>Permissions</strong> are rules about what the agent is allowed to do: which tools, which commands, which paths. They live in the agent's harness and often depend on the model behaving correctly or on a human approving prompts.</p>
<p><strong>Sandboxes</strong> are enforcement at the operating system or infrastructure level. They restrict what the process <em>can</em> do regardless of what the model decides, what a prompt injection says, or whether a human got tired of clicking "approve."</p>
<p>You want both. Permissions give you good defaults, a clean user experience, and audit-friendly control. The sandbox is what holds when the permissions fail. A system prompt that says "never exfiltrate secrets" is not a security control. A network that can't reach the attacker's server is.</p>
<h2>Layer 1: Sandboxing</h2>
<p>A good sandbox has several independent walls. Think filesystem, process, network, credentials, and resources.</p>
<h3>Filesystem isolation</h3>
<ul>
<li><p>Scope read-write access to the project directory only. Everything else should be read-only or invisible.</p>
</li>
<li><p>Explicitly block secrets locations: <code>~/.ssh</code>, <code>~/.aws</code>, <code>~/.gnupg</code>, cloud CLI config, browser profiles, password manager data, and <code>.env*</code> files.</p>
</li>
<li><p>Work in a disposable clone or a git worktree rather than your main checkout, so a bad edit is trivially reversible.</p>
</li>
<li><p>Never mount the Docker socket into an agent container. Access to it is effectively root on the host.</p>
</li>
</ul>
<h3>Process isolation</h3>
<p>There's a spectrum, and you should pick based on how much you trust the task and how much supervision there is.</p>
<ul>
<li><p><strong>OS-level sandboxing</strong> (for example Seatbelt on macOS, or bubblewrap, Landlock, and seccomp on Linux) is lightweight and works well for an interactive agent on a developer machine.</p>
</li>
<li><p><strong>Hardened containers</strong> are the common middle ground. Run as a non-root user, drop all capabilities, use a read-only root filesystem, and set resource limits.</p>
</li>
<li><p><strong>Stronger isolation</strong> such as gVisor or microVMs like Firecracker is appropriate for unattended agents, multi-tenant platforms, and anything processing untrusted repositories. Containers share a kernel with the host, and a microVM doesn't.</p>
</li>
</ul>
<p>Here's a reasonable starting point for a hardened container:</p>
<pre><code class="language-bash">docker run --rm -it \
  --user 1000:1000 \
  --read-only \
  --tmpfs /tmp \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --pids-limit 256 \
  --memory 4g --cpus 2 \
  --network none \
  -v "$PWD":/workspace \
  -w /workspace \
  agent-image
</code></pre>
<p><code>--network none</code> is the safest default for tasks that don't need the internet. For tasks that do, move to the next layer.</p>
<h3>Network egress control</h3>
<p>Network access is usually the most important leg of the trifecta to cut. A few rules:</p>
<ul>
<li><p><strong>Default deny outbound traffic</strong>, then allowlist what the task actually needs: your package registries, your git host, and perhaps specific documentation sites.</p>
</li>
<li><p>Route traffic through a <strong>proxy that logs requests</strong> so you can see what the agent talked to.</p>
</li>
<li><p>Remember that allowlisted domains can still be abused if they accept user content (a public gist or pastebin-style site is an exfiltration channel). Allowlist narrowly.</p>
</li>
<li><p>Consider DNS as a side channel and apply the same egress policy to DNS resolution.</p>
</li>
</ul>
<h3>Credential hygiene</h3>
<p>The best secret is the one the agent never sees.</p>
<ul>
<li><p>Don't give the agent your personal credentials. Give it its own identity, such as a dedicated bot account, with the minimum scopes needed.</p>
</li>
<li><p>Prefer <strong>short-lived, scoped tokens</strong> (for example OIDC-issued tokens in CI) over long-lived keys.</p>
</li>
<li><p>Where possible, inject credentials at the network layer through a proxy, so the agent can <em>use</em> an API without ever holding the key in its environment.</p>
</li>
<li><p>Point agents at read-only replicas or staging systems, never production databases.</p>
</li>
<li><p>Scrub environment variables. Start the agent's process with an explicit allowlist rather than inheriting your whole shell.</p>
</li>
</ul>
<h3>Resource limits and ephemerality</h3>
<ul>
<li><p>Cap CPU, memory, disk, process count, and wall-clock time. This stops runaway loops and fork bombs, and limits cryptomining abuse.</p>
</li>
<li><p>Set budgets on tokens and spend, with a hard stop.</p>
</li>
<li><p>Make environments <strong>ephemeral</strong>. Build a fresh sandbox per task and destroy it afterward, so nothing persists between runs.</p>
</li>
</ul>
<h2>Layer 2: Permission design</h2>
<p>With a solid sandbox underneath, permissions become about intent and workflow rather than being your last line of defense.</p>
<h3>Organize by capability</h3>
<p>Treat permissions as categories with different risk levels:</p>
<table>
<thead>
<tr>
<th>Capability</th>
<th>Typical default</th>
</tr>
</thead>
<tbody><tr>
<td>Read files in the project</td>
<td>Allow</td>
</tr>
<tr>
<td>Edit files in the project</td>
<td>Allow, with diffs visible</td>
</tr>
<tr>
<td>Run tests, linters, type checkers</td>
<td>Allow</td>
</tr>
<tr>
<td>Install dependencies</td>
<td>Ask</td>
</tr>
<tr>
<td>Edit CI config, git hooks, agent settings</td>
<td>Ask or deny</td>
</tr>
<tr>
<td>Network access beyond the allowlist</td>
<td>Ask or deny</td>
</tr>
<tr>
<td>Push, merge, deploy</td>
<td>Ask, or deny for the agent identity</td>
</tr>
<tr>
<td>Touch secrets or production systems</td>
<td>Deny</td>
</tr>
</tbody></table>
<h3>Prefer allowlists over denylists</h3>
<p>Blocking dangerous commands sounds sensible but is brittle. If you deny <code>curl</code>, the agent can use a Python one-liner. If you deny <code>rm -rf /</code>, there are a hundred equivalent spellings. Shell is expressive, and command chaining, subshells, and encoding tricks make pattern-based blocking leaky.</p>
<p>A better approach is to <strong>allow specific commands with specific argument patterns</strong> (such as <code>git status</code>, <code>git diff</code>, <code>npm test</code>, <code>pytest</code>) and let everything else fall through to a prompt or a hard block. Then let the sandbox, not string matching, cover what slips through. Keep denylists as defense in depth, not as the primary control.</p>
<p>Here's an illustrative policy. It's not any particular tool's format, but the shape applies broadly:</p>
<pre><code class="language-yaml">filesystem:
  read_write: ["./"]
  read_only: ["/usr", "/opt/toolchain"]
  deny: ["~/.ssh", "~/.aws", "~/.gnupg", "**/.env*"]

network:
  default: deny
  allow: ["registry.npmjs.org", "pypi.org", "github.com"]

commands:
  allow: ["git status", "git diff", "npm test", "pytest *"]
  ask:   ["npm install *", "pip install *", "git push *"]
  deny:  ["sudo *", "git push --force*"]

protected_paths:   # agent may not modify these without approval
  - ".github/workflows/**"
  - ".git/hooks/**"
  - "agent-config/**"
</code></pre>
<h3>Protect the escape hatches</h3>
<p>Some files are special because changing them changes what runs later, or what the agent is allowed to do. Treat these as protected:</p>
<ul>
<li><p>The agent's own configuration and permission files</p>
</li>
<li><p>Git hooks and CI/CD workflow definitions</p>
</li>
<li><p>Package scripts (<code>postinstall</code>, <code>prepare</code>) and build files</p>
</li>
<li><p>Editor task configs and anything that auto-executes</p>
</li>
</ul>
<p>A common failure pattern is an agent (or an injected instruction) adding a script that runs outside the sandbox the next time a human opens the project or CI triggers.</p>
<h3>Design against approval fatigue</h3>
<p>If your agent asks for approval forty times an hour, people will start approving everything without reading. That's worse than having no prompts, because it creates false confidence.</p>
<p>The fix is to shift from "ask constantly" to <strong>"generous inside the sandbox, strict at the boundary."</strong> Let the agent move quickly within a well-fenced environment, and reserve prompts for the moments that cross the boundary: new network destinations, new dependencies, protected files, and irreversible actions. When you do prompt, show the exact command or diff, not a vague summary.</p>
<h3>Be careful with "skip all permissions" modes</h3>
<p>Many agents have a mode that disables approval prompts for fully autonomous operation. That can be reasonable, but only when the sandbox is doing the real work: a disposable environment, no secrets, no production access, and restricted egress. Running it on your main machine with your real credentials is the configuration to avoid.</p>
<h3>Keep humans on irreversible actions</h3>
<p>Use the controls your platform already has:</p>
<ul>
<li><p>Branch protection and required reviews, with the agent unable to approve or merge its own PRs</p>
</li>
<li><p>Push access to feature branches only, no force-push</p>
</li>
<li><p>Deploys and schema migrations gated behind a human</p>
</li>
<li><p>Agent identities that are clearly labeled, so you can audit what it did versus what a person did</p>
</li>
</ul>
<h2>Tools and MCP servers</h2>
<p>Every tool you connect to an agent is an expansion of both its capabilities and its attack surface. An MCP server is code running with whatever access you gave it, <em>and</em> a source of text the model will read and treat as meaningful.</p>
<ul>
<li><p><strong>Vet servers like dependencies.</strong> Review the source, pin versions, and be wary of tools that update silently.</p>
</li>
<li><p><strong>Scope tokens narrowly</strong> and prefer read-only access where the task allows.</p>
</li>
<li><p><strong>Watch for tool poisoning.</strong> Tool descriptions and responses are model-visible text, so they can carry injected instructions.</p>
</li>
<li><p><strong>Don't pair "reads untrusted content" with "can send or publish"</strong> in the same session without a confirmation step. That's the trifecta again: a tool that reads inbound email plus a tool that sends messages is an exfiltration path.</p>
</li>
<li><p>Separate servers for sensitive systems, so a compromise in one doesn't hand over everything.</p>
</li>
</ul>
<h2>Prompt injection: defend the blast radius</h2>
<p>Be honest about this one: there is no known way to fully prevent prompt injection today. Models can be trained to resist it and filters can catch known patterns, but none of that is a guarantee. So the engineering goal is to <strong>assume it will sometimes succeed and limit what that success can do.</strong></p>
<ul>
<li><p>Treat everything the agent reads from outside your control as untrusted: issues, PR descriptions, web pages, third-party docs, dependency source, and tool output.</p>
</li>
<li><p>Break the trifecta. If the agent must read untrusted content, make sure it can't also reach secrets and the open internet.</p>
</li>
<li><p>Require confirmation for sensitive actions that occur <em>after</em> the agent has ingested untrusted content.</p>
</li>
<li><p>Don't rely on "ignore any instructions in the data" lines in the system prompt as a defense. They help a little and fail often.</p>
</li>
<li><p>For high-risk workflows, such as reviewing PRs from strangers or processing untrusted repositories, use read-only access, no network, and the strongest isolation available.</p>
</li>
</ul>
<h2>Supply chain safeguards</h2>
<p>Because agents add dependencies so readily, put guardrails around it:</p>
<ul>
<li><p>Commit and enforce <strong>lockfiles</strong>, and require approval for any new dependency.</p>
</li>
<li><p>Use a <strong>private registry or mirror</strong> with an allowlist where you can.</p>
</li>
<li><p>Verify that a suggested package actually exists, has real history, and is the one you expect, before it's installed.</p>
</li>
<li><p>Consider a waiting period on very new package versions, and disable install scripts where your workflow allows it (for example <code>npm install --ignore-scripts</code>).</p>
</li>
<li><p>Run dependency and vulnerability scanning in CI so problems surface even if an agent introduced them.</p>
</li>
</ul>
<h2>Logging, monitoring, and a kill switch</h2>
<p>You can't secure what you can't see.</p>
<ul>
<li><p><strong>Log everything</strong>: every tool call, command, file diff, and outbound request, with timestamps and the agent identity that made them. Store logs where the agent can't modify them.</p>
</li>
<li><p><strong>Alert on anomalies</strong>: reads of credential paths, connections to unfamiliar domains, unusual volumes of file reads, edits to protected files.</p>
</li>
<li><p><strong>Make it stoppable.</strong> You should be able to kill a session or revoke an agent's tokens instantly.</p>
</li>
<li><p><strong>Review sessions after the fact</strong>, especially for unattended runs. Treat the transcript like a commit history you'd actually look at.</p>
</li>
</ul>
<h2>Matching controls to context</h2>
<p>Not every situation needs maximum lockdown. Match the controls to the risk.</p>
<table>
<thead>
<tr>
<th>Scenario</th>
<th>Suggested posture</th>
</tr>
</thead>
<tbody><tr>
<td>Interactive, on your machine, you're watching</td>
<td>OS-level sandbox, scoped filesystem, prompts at the boundary, no ambient secrets</td>
</tr>
<tr>
<td>Autonomous / background tasks</td>
<td>Ephemeral container or microVM, allowlisted egress, scoped short-lived token, no prod access</td>
</tr>
<tr>
<td>CI agents responding to issues or PRs</td>
<td>Treat all input as hostile: isolated runner, minimal permissions, no secrets exposed to untrusted triggers, human review before merge</td>
</tr>
<tr>
<td>Analyzing untrusted code or repos</td>
<td>Strongest isolation, read-only, no network, throwaway environment</td>
</tr>
</tbody></table>
<h2>A quick checklist</h2>
<ul>
<li><p>The agent runs in a sandbox that enforces limits at the OS or infrastructure level, not just in prompts</p>
</li>
<li><p>Read-write access is limited to the project directory; secrets paths are blocked</p>
</li>
<li><p>Outbound network is deny-by-default with a narrow allowlist and logging</p>
</li>
<li><p>The agent uses its own scoped, short-lived credentials, never yours</p>
</li>
<li><p>Permissions are allowlist-based, tiered by risk, and protect config, hooks, and CI files</p>
</li>
<li><p>Approval prompts are rare, specific, and meaningful</p>
</li>
<li><p>Autonomous modes run only in disposable environments with no secrets</p>
</li>
<li><p>MCP servers and dependencies are vetted, pinned, and least-privileged</p>
</li>
<li><p>Humans gate merges, deploys, and migrations</p>
</li>
<li><p>Everything is logged, and you can revoke access fast</p>
</li>
</ul>
<h2>Closing thoughts</h2>
<p>Securing coding agents isn't about distrusting them so much that they become useless. It's about building an environment where a mistake, or a successful attack, has a small and recoverable blast radius. Permissions shape how the agent behaves day to day; sandboxes guarantee the worst case stays contained.</p>
<p>Start with the highest-leverage moves: take away ambient credentials, cut off unrestricted network egress, and keep the agent inside a disposable, scoped workspace. Then tune permissions so that the agent is fast where it's safe and slow where it matters. Do that, and you can let agents do real work with a lot more confidence.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/how-to-secure-ai-coding-agents-sandboxing-and-permissions-o0bgz?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[How AI Generates Images: From Pure Noise to Finished Art]]></title><description><![CDATA[Type a sentence, get a picture. This guide explains how AI image generators work: how prompts become numbers, how diffusion turns random noise into an…]]></description><link>https://blog.zyvop.com/how-ai-generates-images-from-pure-noise-to-finished-art-8zq0b</link><guid isPermaLink="true">https://blog.zyvop.com/how-ai-generates-images-from-pure-noise-to-finished-art-8zq0b</guid><category><![CDATA[ai image generation]]></category><category><![CDATA[DiffusionModels]]></category><category><![CDATA[#generativeai]]></category><category><![CDATA[MachineLearning]]></category><category><![CDATA[#TextToImage]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Fri, 06 Nov 2026 08:17:18 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/5850a5a0-d22a-4116-8c15-564196e4e708.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<blockquote>
<p><strong>The short version:</strong> An AI image generator turns your words into numbers, then builds a picture from them. Most open tools start with pure random static and clean it up over a few dozen steps, guided by your prompt. Some newer systems instead build the image piece by piece, the way a language model writes a sentence.</p>
<p>Type "a corgi astronaut floating above a neon city, oil painting" into a text box, wait a few seconds, and an image appears that nobody has ever painted or photographed. It feels like magic. It is actually a chain of clever mathematical ideas, and once you see how the pieces fit together, the magic becomes something even better: understanding.</p>
</blockquote>
<hr />
<h2>The Pipeline at a Glance</h2>
<p><img src="https://mermaid.ink/img/pako:eNpVUEtPwzAM_ivG5_SAxKkHpG3dBBJw2AAJNTukrdtGJHGVZC-2_XeUFpA4-vP3ss9Yc0OYY2v4UPfKR3haSwcAMCslfvDOw-DZDlHiFrLsHualxFc6RiCXpF7iduLPx_WilLi0FTWNdl3IwZJy2nWgAridrciHP8FLKXGtXMMWHOtAvwlFKbGgBCWhYR5y8GR5T6DA6BgNTQIYyEOINPxZLpLBRWKIRD4A7cmffhgXKCZOMYYsx5D_FyzHzaqUuNJOh54a0FZ1qRgKtOSt0g3mZ4w92fS0hlq1MxHFhLwrr1VlKCROyy6ulNXmhDlmahgMZeEUIlkBc6Pd57OqN-O8YhcFSNxQxwRvjxIFrLniyAIeyOwp6loJmHmtjICgXMgCed2iGEM2-it1ub0bjni9Cqy6BRv2mOPNodeR8PoNSa2d-w?type=png" alt="Mermaid Diagram" /></p>
<p>The rest of this post takes each box in turn, then looks at what happens when the recipe changes, why images still go wrong, and the legal debates around the whole field.</p>
<hr />
<h2>1. The Big Idea: Learning What Images Look Like</h2>
<p>An AI image generator is not a search engine, and it does not stitch together pieces of existing pictures. It is a model that has learned the <em>statistical patterns</em> of images by studying an enormous collection of them, often billions of image-and-caption pairs scraped or licensed from across the internet.</p>
<p>Nobody hand-codes rules for what a cat looks like. The model sees huge numbers of images paired with captions and gradually absorbs the patterns: pointed ears, whiskers, fur texture, typical poses, how light falls, and how all of that differs from dogs, foxes, or couches. Those patterns are stored in the model's <strong>parameters</strong>, numbers that act like a compressed memory of visual concepts. The count ranges from roughly 860 million (the main network in Stable Diffusion 1.5) to around 12 billion (Flux).</p>
<blockquote>
<p><strong>A caveat:</strong> In rare cases, especially for images that appeared many times in the training data, a model can reproduce something very close to a training image. This is one reason the copyright debates later in this post are so heated.</p>
</blockquote>
<hr />
<h2>2. Turning Words Into Numbers</h2>
<p>Computers cannot understand "corgi astronaut" directly, so the first job is translation. Your prompt goes through a <strong>text encoder</strong>, a neural network that converts your words into a sequence of number lists called <em>embeddings</em>, roughly one for each chunk of text.</p>
<p>The key trick is that embeddings capture <strong>meaning</strong>. Phrases with similar meanings land close together in this numeric space. One famous approach, called <strong>CLIP</strong>, was trained by matching images to their captions, so that a caption and the image it describes end up pointing in similar directions. That is how the system connects "sunset" to warm orange gradients and "melancholy" to muted, moody tones.</p>
<p>Many newer models pair CLIP with a larger, language-model-style encoder called <strong>T5</strong>, which helps them follow longer and more detailed prompts.</p>
<hr />
<h2>3. Diffusion: Learning to Remove Noise</h2>
<p>For several years, the dominant technique for image generation has been <strong>diffusion</strong>, and it still powers open-weight tools like Stable Diffusion and Flux. (Some leading systems now work differently, as we will see in Section 7.) The idea is surprisingly simple once you see it, and it has two phases:</p>
<table>
<thead>
<tr>
<th>Phase</th>
<th>What happens</th>
<th>What the model learns</th>
</tr>
</thead>
<tbody><tr>
<td>Training</td>
<td>Real photos are buried in random static, at randomly chosen noise levels. The network tries to predict the noise so it can be subtracted.</td>
<td>To guess what clean image is probably hiding under any amount of noise</td>
</tr>
<tr>
<td>Generating</td>
<td>The process runs in reverse. Start from pure random noise and remove a little of it at every step.</td>
<td>To turn static into a picture that matches your prompt</td>
</tr>
</tbody></table>
<p>The noise is like the snow on an old TV. In practice, training jumps straight to a randomly chosen noise level rather than adding it bit by bit, but the idea is the same. Repeat this millions of times across every kind of image, and the network gets very good at undoing noise.</p>
<p>When generating, each pass gets a little closer to a finished picture:</p>
<p><img src="https://mermaid.ink/img/pako:eNpFkE9Lw0AQxb_KOOfNoVovOQgqFAUFadRLtodJMskObnbD7sS2ln53aYV6fH_g_XgHbGPHWGLv47Z1lBRe1jYAAFSL2uLbnBiykkprcQNFcQfVdW3xMfqYoPGxyRf_pra4jvPgIDua-D9Y1hbfeadz4gwUOvAyOL3Et7XFlQTJjjuQkQa2uEGDI6eRpMPygOp4PFF23NPsFc2f80lJqPGcT50-Bl3RKH6PJRY0TZ6LvM_Ko4EHL-HrldrqrFcxqAGLFQ-R4ePZooF1bKJGA0_sv1mlJQP3ScgbyBRykTlJj-Y8UsnPiWWxnHZ4PBpshvMbWOLV1okyHn8BLRVwfg?type=png" alt="Mermaid Diagram" /></p>
<blockquote>
<p><strong>Analogy:</strong> It looks like a photograph slowly developing in a darkroom, except that no photograph existed to begin with.</p>
</blockquote>
<p><strong>How many steps?</strong> Typical setups use somewhere between 20 and 50. The original method needed around 1,000, while "turbo" and "schnell" models that have been distilled down can manage as few as one to four.</p>
<p><strong>A newer variation:</strong> Some models, including Flux and Stable Diffusion 3, use a close cousin of this recipe called <em>flow matching</em> (Stable Diffusion 3 uses a version called rectified flow). Instead of predicting the noise, the network learns which direction to move from noise toward a clean image. The intuition stays the same.</p>
<hr />
<h2>4. Steering the Noise With Your Prompt</h2>
<p>If the model only learned to remove noise, it would produce random plausible images. So how does your text prompt control the result?</p>
<p>At every denoising step, the model receives your text embeddings as extra guidance. It is effectively asking, <em>"What clean image, that matches this description, is hiding under this noise?"</em></p>
<p>Many systems also use <strong>classifier-free guidance</strong>. At each step the model makes two predictions, one with your prompt and one without, then pushes the result further in the direction that the prompt adds.</p>
<table>
<thead>
<tr>
<th>Guidance scale</th>
<th>Typical result</th>
</tr>
</thead>
<tbody><tr>
<td>Low</td>
<td>Looser, more creative, may ignore parts of your prompt</td>
</tr>
<tr>
<td>Medium</td>
<td>A good balance of faithfulness and natural look</td>
</tr>
<tr>
<td>Too high</td>
<td>Follows words strictly, but images can look oversaturated or strange</td>
</tr>
</tbody></table>
<hr />
<h2>5. The Efficiency Trick: Working in Latent Space</h2>
<p>Diffusion works on the whole image at once, but running a big network over every pixel of a high-resolution image, dozens of times over, would be very expensive. Models like Stable Diffusion solve this with a compression step.</p>
<p>A component called an <strong>autoencoder</strong> squeezes images into a much smaller representation called <em>latent space</em>, which keeps the essential structure but discards redundant pixel-level detail. The entire diffusion process happens in this compact space. At the end, a decoder expands the result back into a full-size image.</p>
<table>
<thead>
<tr>
<th></th>
<th>Pixel space</th>
<th>Latent space (Stable Diffusion)</th>
</tr>
</thead>
<tbody><tr>
<td>Size</td>
<td>512 × 512 pixels, 3 color values each</td>
<td>64 × 64 cells, 4 values each</td>
</tr>
<tr>
<td>Total numbers</td>
<td>786,432</td>
<td>16,384</td>
</tr>
<tr>
<td>Relative cost</td>
<td>Baseline</td>
<td>Roughly 48 times fewer numbers</td>
</tr>
</tbody></table>
<blockquote>
<p><strong>Analogy:</strong> It is like sketching your composition on a small notepad and only then enlarging it into a finished canvas.</p>
</blockquote>
<hr />
<h2>6. Under the Hood: The Denoising Network</h2>
<p>The network that does the actual denoising has evolved:</p>
<table>
<thead>
<tr>
<th>Generation</th>
<th>Backbone</th>
<th>Training approach</th>
<th>Examples</th>
</tr>
</thead>
<tbody><tr>
<td>Early latent diffusion</td>
<td>U-Net: processes the image at multiple scales to handle both layout and fine detail</td>
<td>Predict the noise</td>
<td>Stable Diffusion 1.5, SDXL</td>
</tr>
<tr>
<td>Diffusion transformers</td>
<td>Transformer working on patches of an image, the same family of architecture behind large language models</td>
<td>Predict the noise, or use flow matching</td>
<td>PixArt-α, Stable Diffusion 3, Flux</td>
</tr>
</tbody></table>
<p>Transformers tend to scale well with more data and compute, which is one reason image quality has improved so quickly.</p>
<hr />
<h2>7. Beyond Diffusion: Other Ways to Make Images</h2>
<p>Diffusion is not the only approach, and it may not stay on top.</p>
<table>
<thead>
<tr>
<th></th>
<th>GANs</th>
<th>Diffusion</th>
<th>Autoregressive</th>
</tr>
</thead>
<tbody><tr>
<td>How it works</td>
<td>A generator makes fake images while a discriminator tries to spot them, and each improves by beating the other</td>
<td>Start from noise and remove it step by step</td>
<td>Treat the image as a sequence of tokens and generate it piece by piece, like writing a sentence</td>
</tr>
<tr>
<td>Strengths</td>
<td>Very fast: one pass</td>
<td>Stable to train, high quality, flexible</td>
<td>Can share a language model's knowledge and even reason about the prompt</td>
</tr>
<tr>
<td>Weaknesses</td>
<td>Hard to train, historically less varied and weaker at following text prompts</td>
<td>Needs many steps (though distilled models cut this)</td>
<td>Early versions trailed diffusion on complex prompts; newer ones have caught up</td>
</tr>
<tr>
<td>Examples</td>
<td>StyleGAN, BigGAN</td>
<td>Stable Diffusion, Flux</td>
<td>The original DALL·E, Infinity (research), GPT Image (reported)</td>
</tr>
</tbody></table>
<p><strong>The OpenAI shift.</strong> The most visible change came from OpenAI, which moved from its diffusion-based DALL·E models to GPT Image models that are described as native to the GPT architecture, generating images inside the same system that handles text. (OpenAI removed DALL·E 2 and DALL·E 3 from its API on May 12, 2026.) OpenAI has not published full technical details, so the exact recipe is not public. Research models like Infinity, which builds an image coarse-to-fine rather than left to right, have shown that sufficiently scaled autoregressive designs can compete with diffusion on benchmarks of prompt accuracy.</p>
<p>In practice, the field now includes pure diffusion, diffusion transformers, autoregressive models, and hybrids. Which works best depends on whether you care most about speed, openness, artistic control, or following complicated instructions.</p>
<hr />
<h2>8. Why AI Images Still Go Wrong</h2>
<p>Glitches used to be everywhere. Newer models have improved a lot, but errors still happen, especially with smaller or older models, and they make sense given how the models work.</p>
<table>
<thead>
<tr>
<th>Glitch</th>
<th>Why it happens</th>
<th>Where things stand</th>
</tr>
</thead>
<tbody><tr>
<td>Extra or twisted fingers</td>
<td>Hands vary enormously, and the model learns visual patterns rather than anatomy</td>
<td>Much better in recent models, not perfect</td>
</tr>
<tr>
<td>Garbled text on signs</td>
<td>Letters demand precise, consistent detail across the image</td>
<td>A big leap in 2026: OpenAI says GPT Image 2 renders text far more accurately, including in Japanese, Korean, Chinese, Hindi, and Bengali. Some reviewers still report weak spots, such as non-English text and inconsistent brand logos</td>
</tr>
<tr>
<td>Swapped colors or objects ("red hat, blue shirt" comes out reversed)</td>
<td>Models can blend concepts rather than assigning each to the right object</td>
<td>Models that reason about a prompt before drawing may handle this better</td>
</tr>
<tr>
<td>Objects melting into each other</td>
<td>Keeping fine structure consistent across a whole image is hard</td>
<td>Improves with scale</td>
</tr>
</tbody></table>
<blockquote>
<p><strong>Do these models "understand" anything?</strong> Research suggests they pick up some internal structure, such as rough depth, so "no understanding at all" would be too strong. But how much real comprehension sits behind the patterns is still an open question.</p>
</blockquote>
<hr />
<h2>9. The Bigger Questions</h2>
<p>Understanding the technology also helps frame the debates around it.</p>
<p><strong>Copyright, consent, and compensation.</strong> Because models learn from vast collections of existing artwork and photographs, artists and rights holders are challenging how that data was used. As of late 2026, the courts have not settled the central questions:</p>
<table>
<thead>
<tr>
<th>Case</th>
<th>What is at stake</th>
<th>Status</th>
</tr>
</thead>
<tbody><tr>
<td>Disney, Universal, and Warner Bros. v. Midjourney (US)</td>
<td>Studios allege Midjourney trained on their works and generates their characters</td>
<td>Pretrial, with disputes over evidence sharing; no ruling on fair use</td>
</tr>
<tr>
<td>Andersen v. Stability AI, Midjourney, DeviantArt, and Runway (US)</td>
<td>Class action by artists over training on their work</td>
<td>Hearing on class certification and summary judgment pushed to September 2027; no trial date</td>
</tr>
<tr>
<td>Getty Images v. Stability AI (UK)</td>
<td>Getty alleged Stable Diffusion infringed its copyrights and trademarks</td>
<td>November 2025: the High Court held the model is not an "infringing copy" because it does not store the images it learned from. It did not decide whether training itself is infringement (Getty had dropped that claim) and found limited trademark infringement where some outputs carried Getty or iStock watermarks</td>
</tr>
</tbody></table>
<p>Different courts may well reach different answers.</p>
<p><strong>Deepfakes and misinformation.</strong> The same realism that makes these tools creatively exciting also makes fakes easier to produce. Many companies now add invisible watermarks (such as Google's SynthID) and provenance labels (such as C2PA content credentials) to generated images.</p>
<p><strong>The creative conversation.</strong> There is a genuine debate about whether these tools replace artists, assist them, or open creativity to people who never had the technical skills to paint or draw. Reasonable people land in different places.</p>
<hr />
<h2>Key Terms at a Glance</h2>
<table>
<thead>
<tr>
<th>Term</th>
<th>Plain-English meaning</th>
</tr>
</thead>
<tbody><tr>
<td>Embedding</td>
<td>Meaning turned into numbers, so a computer can compare ideas</td>
</tr>
<tr>
<td>CLIP / T5</td>
<td>Text encoders that turn prompts into embeddings</td>
</tr>
<tr>
<td>Diffusion</td>
<td>Generating an image by gradually removing noise</td>
</tr>
<tr>
<td>Flow matching</td>
<td>A diffusion cousin that learns the direction from noise to image</td>
</tr>
<tr>
<td>Latent space</td>
<td>A compressed version of the image where the work happens</td>
</tr>
<tr>
<td>Autoencoder</td>
<td>The pair of networks that compress an image and expand it back</td>
</tr>
<tr>
<td>Classifier-free guidance</td>
<td>A dial that controls how strictly the image follows your prompt</td>
</tr>
<tr>
<td>U-Net / Transformer</td>
<td>Two kinds of network that do the denoising</td>
</tr>
<tr>
<td>Autoregressive</td>
<td>Generating an image as an ordered sequence of tokens</td>
</tr>
<tr>
<td>GAN</td>
<td>Two networks competing: one forges images, one spots fakes</td>
</tr>
</tbody></table>
<hr />
<h2>Wrapping Up</h2>
<p><strong>Key takeaways</strong></p>
<ul>
<li><p>Image generators learn <em>patterns</em> from billions of captioned images; they do not copy and paste.</p>
</li>
<li><p>Your prompt becomes embeddings that steer the image at every step.</p>
</li>
<li><p>Diffusion turns static into a picture by repeated cleanup, and working in latent space keeps it affordable.</p>
</li>
<li><p>Autoregressive models, which build images token by token, now compete with diffusion.</p>
</li>
<li><p>Errors are shrinking but not gone, and the legal and ethical questions are still open.</p>
</li>
</ul>
<p>No single step is magic. But stacked together, they let a machine turn a sentence into something you can see. And we are still in the early chapters of that story.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/how-ai-generates-images-from-pure-noise-to-finished-art-8zq0b?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Git & GitHub Workflows: Branches, Pull Requests, and Code Review]]></title><description><![CDATA[Git makes breaking things safe. This post explains commits, branches, merge conflicts, and how to undo almost anything, then covers GitHub pull requests…]]></description><link>https://blog.zyvop.com/git-github-workflows-branches-pull-requests-and-code-review-ntiub</link><guid isPermaLink="true">https://blog.zyvop.com/git-github-workflows-branches-pull-requests-and-code-review-ntiub</guid><category><![CDATA[#aiengineer]]></category><category><![CDATA[codereview]]></category><category><![CDATA[Git]]></category><category><![CDATA[GitHub]]></category><category><![CDATA[versioncontrol]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Fri, 06 Nov 2026 07:47:09 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/961f01d3-620a-41e3-8ad3-af2630415a7d.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Sooner or later, every programmer has the same awful moment. You change something, it breaks, and you can't remember what the code looked like before. Or you work on a project with someone else, and your changes collide with theirs. Or you accidentally delete a file you needed.</p>
<p><strong>Git</strong> exists to make all of that boring. It keeps a complete history of your project, lets you try risky ideas safely, lets you undo almost anything, and lets many people work on the same code without trampling each other. This post is core, and it's also personal for you: back in Post 3, I asked you to start a repository called <code>ai-engineer-journey</code>. By the end of this post, that repository will be a real working project with a proper history, and the habits you build here are the same ones used by every professional team.</p>
<h2>Git and GitHub are not the same thing</h2>
<p>People mix these up all the time.</p>
<ul>
<li><p><strong>Git</strong> is a tool on your computer. It tracks changes to files, in a folder called a <strong>repository</strong> (or repo). It works completely offline.</p>
</li>
<li><p><strong>GitHub</strong> is a website that hosts Git repositories and adds collaboration features on top: pull requests, code review, issue tracking, automated checks, and a public profile. GitLab and Bitbucket are similar services.</p>
</li>
</ul>
<p>So Git is the engine, and GitHub is one place to park the car and invite people to look at it. Almost everything in the first half of this post is plain Git, and you can practice it without an account.</p>
<p>For AI work, Git is more valuable than usual. Prompts, configuration files, evaluation sets, and code all change constantly, and a small tweak to a prompt can quietly make results better or worse. With Git, every change is recorded, and you can always say what changed and when. And a tidy public GitHub profile is one of the best things to show an employer (Post 72).</p>
<h2>The mental model</h2>
<p>Three ideas make everything else click.</p>
<p><strong>1. A commit is a snapshot.</strong> When you commit, Git saves the state of your files at that moment, together with a message, an author, a time, and a pointer to the commit that came before it. Each commit is a permanent, named checkpoint.</p>
<p><strong>2. History is a chain, which makes it a graph.</strong> Every commit points back to its parent, so a project's history is a chain of commits. When work splits into branches and rejoins, it becomes the kind of graph you met in Post 16. Many Git commands are really just walking that graph.</p>
<p><strong>3. A branch is just a movable label.</strong> A branch is a name that points at one commit. When you make a new commit, the label moves forward to it. That's why branches are cheap, and why you should use lots of them. There's also a special pointer called <strong>HEAD</strong>, which means "where I am right now."</p>
<p>And there are three places your files can be:</p>
<pre><code class="language-sql">working directory  --git add--&gt;  staging area  --git commit--&gt;  repository
 (files you edit)                 (what goes in the                (the saved
                                   next commit)                     history)
</code></pre>
<p>You edit files in the <strong>working directory</strong>. You choose which changes to include using <code>git add</code>, which puts them in the <strong>staging area</strong>. Then <code>git commit</code> saves exactly what's staged into the <strong>repository</strong>. The staging area looks like an extra step at first. It's actually what lets you make <em>clean</em> commits: if you changed ten things, you can commit them as three tidy groups.</p>
<h2>One-time setup</h2>
<p>Install Git (from git-scm.com, or your system's package manager), then tell it who you are. This name and email get stamped on every commit you make:</p>
<pre><code class="language-bash">git config --global user.name "Your Name"
git config --global user.email "you@example.com"
git config --global init.defaultBranch main
</code></pre>
<p>(Use the same email as your GitHub account so your commits are linked to your profile.) The last line makes new repositories start with a branch called <code>main</code>.</p>
<h2>The everyday loop</h2>
<p>I ran the Git commands in this post in real repositories on my own machine, so those outputs are genuine. (The parts about the GitHub website I describe from knowledge, and I'll flag them when we get there. Your commit hashes, the short codes like <code>d9fcc09</code>, will be different from mine. That's normal.) Let's make a little project for prompt experiments:</p>
<pre><code class="language-bash">mkdir prompt-notes
cd prompt-notes
git init
</code></pre>
<p><code>git init</code> turns the folder into a repository. Now let's create a few files, and ask Git what it sees:</p>
<pre><code class="language-bash">git status
</code></pre>
<pre><code class="language-rust">On branch main

No commits yet

Untracked files:
  (use "git add &lt;file&gt;..." to include in what will be committed)
	.gitignore
	README.md
	prompts/

nothing added to commit but untracked files present (use "git add" to track)
</code></pre>
<p><code>git status</code> is the command you'll type more than any other. It tells you where you are and what's changed, and it usually suggests what to do next. Make a habit of running it before every <code>add</code> and <code>commit</code>.</p>
<p>The files are "untracked," meaning Git has noticed them but isn't recording them. Stage them, then commit:</p>
<pre><code class="language-bash">git add README.md .gitignore prompts
git commit -m "Add README, gitignore, and first summarize prompt"
</code></pre>
<pre><code class="language-sql">[main (root-commit) d9fcc09] Add README, gitignore, and first summarize prompt
 3 files changed, 8 insertions(+)
</code></pre>
<p>That's your first commit. Now let's change the prompt and look at what changed, using <code>git diff</code>:</p>
<pre><code class="language-bash">git diff
</code></pre>
<pre><code class="language-diff">--- a/prompts/summarize.txt
+++ b/prompts/summarize.txt
@@ -1 +1 @@
-Summarize the text below in two sentences.
+Summarize the text below in two sentences. Use plain language.
</code></pre>
<p>A line starting with <code>-</code> was removed, and one starting with <code>+</code> was added. Then stage it and commit again. <code>git diff</code> shows changes that <em>aren't</em> staged yet, and <code>git diff --staged</code> shows what <em>is</em> staged and about to be committed. Finally, look at the history:</p>
<pre><code class="language-bash">git log --oneline
</code></pre>
<pre><code class="language-sql">9d5cb3e Ask for plain language in summarize prompt
d9fcc09 Add README, gitignore, and first summarize prompt
</code></pre>
<p>The newest commit is on top.</p>
<h3>A word on <code>git add .</code></h3>
<p>You'll see <code>git add .</code> in lots of tutorials. It stages <em>everything</em>, which is fast, and also how people accidentally commit <code>.env</code> files, big data files, and junk. Either add specific paths, as above, or run <code>git status</code> first and read what you're about to add.</p>
<h3>Writing good commit messages</h3>
<p>A commit message is a note to your future self and your teammates. Compare:</p>
<table>
<thead>
<tr>
<th>Bad</th>
<th>Good</th>
</tr>
</thead>
<tbody><tr>
<td>fix</td>
<td>Fix off-by-one in chunk_words overlap</td>
</tr>
<tr>
<td>update stuff</td>
<td>Add retry with backoff to the model client</td>
</tr>
<tr>
<td>final version 2</td>
<td>Ask for plain language in summarize prompt</td>
</tr>
</tbody></table>
<p>A few habits:</p>
<ul>
<li><p><strong>Start with a short summary line</strong>, around 50 characters, written as a command: "Add", "Fix", "Remove".</p>
</li>
<li><p><strong>Explain <em>why</em> in the body when it isn't obvious.</strong> The <code>-m</code> flag can be used twice, once for the summary and once for the body:</p>
</li>
</ul>
<pre><code class="language-bash">git commit -m "Ask for plain language in summarize prompt" \
           -m "Early tests showed jargon-heavy summaries, so the prompt now asks for plain language."
</code></pre>
<ul>
<li><p><strong>Commit small, focused changes.</strong> One logical change per commit. "Add retry logic" is a good commit. "Lots of changes" is a bad one, because you can't undo one piece without undoing everything.</p>
</li>
<li><p><strong>Commit often.</strong> Commits are cheap, and they're your save points.</p>
</li>
</ul>
<h2>What to leave out: .gitignore</h2>
<p>A file called <code>.gitignore</code> lists patterns of files Git should pretend don't exist. Ours contains:</p>
<pre><code class="language-bash">.venv/
__pycache__/
.env
*.db
</code></pre>
<p>Git can tell you which rule matched:</p>
<pre><code class="language-bash">git check-ignore -v .env .venv/junk.txt
</code></pre>
<pre><code class="language-bash">.gitignore:3:.env	.env
.gitignore:1:.venv/	.venv/junk.txt
</code></pre>
<p>Neither of those files showed up in <code>git status</code>, even though they exist. For Python and AI projects, a good starting list of things to ignore includes:</p>
<ul>
<li><p>Virtual environments (<code>.venv/</code>), and <code>__pycache__/</code> (Post 12).</p>
</li>
<li><p><strong>Secrets</strong>: <code>.env</code> files and anything with API keys. Commit a <code>.env.example</code> with fake values instead.</p>
</li>
<li><p>Local databases (<code>*.db</code>) and logs.</p>
</li>
<li><p>Large datasets and <strong>model weights</strong>. Git isn't built for big binary files, and GitHub rejects individual files over 100 MB. Large assets belong in a different place (such as a model hub or object storage), or a tool like Git LFS.</p>
</li>
<li><p>Notebook checkpoints, build output, and editor settings.</p>
</li>
</ul>
<p>GitHub offers ready-made <code>.gitignore</code> templates for Python when you create a repository, which is a fine place to start.</p>
<h2>Undoing things: the safety net</h2>
<p>This is the part that makes people love Git. Almost nothing you do is permanent, as long as you know which tool to reach for:</p>
<table>
<thead>
<tr>
<th>The situation</th>
<th>The command</th>
</tr>
</thead>
<tbody><tr>
<td>I edited a file and want to throw the changes away</td>
<td>git restore &lt;file&gt;</td>
</tr>
<tr>
<td>I staged a file by mistake</td>
<td>git restore --staged &lt;file&gt;</td>
</tr>
<tr>
<td>I just committed and want to fix the message or add a forgotten file</td>
<td>git commit --amend</td>
</tr>
<tr>
<td>I want to undo an old commit, safely</td>
<td>git revert &lt;commit&gt;</td>
</tr>
<tr>
<td>I need to set work aside for a minute</td>
<td>git stash</td>
</tr>
<tr>
<td>I want to move a branch back in time</td>
<td>git reset</td>
</tr>
<tr>
<td>I think I destroyed something</td>
<td>git reflog</td>
</tr>
</tbody></table>
<p>Let me show the ones that matter most.</p>
<p><strong>Discarding a bad edit:</strong></p>
<pre><code class="language-bash">printf "THIS EDIT IS A MISTAKE\n" &gt; prompts/summarize.txt
git restore prompts/summarize.txt
</code></pre>
<p>The file goes back to how it was at the last commit. (This one is final: those uncommitted changes are gone.)</p>
<p><strong>Fixing the last commit.</strong> I committed with a typo:</p>
<pre><code class="language-bash">git commit -m "Add clasify prompt"
git commit --amend -m "Add classify prompt"
</code></pre>
<pre><code class="language-sql">4e915f8 Add clasify prompt      &lt;- before
8e2f1b4 Add classify prompt     &lt;- after
</code></pre>
<p>Notice the hash changed. Amending <em>replaces</em> the commit with a new one. That's perfectly fine for a commit that only exists on your computer, but never amend a commit you've already shared (more on that below).</p>
<p><strong>Undoing an old commit safely with</strong> <code>revert</code><strong>.</strong> I added a translate prompt, then decided I didn't want it:</p>
<pre><code class="language-bash">git revert --no-edit HEAD
</code></pre>
<pre><code class="language-csharp">[main 065b64b] Revert "Add translate prompt (not needed after all)"
 1 file changed, 1 deletion(-)
</code></pre>
<p><code>revert</code> doesn't erase history. It adds a <em>new</em> commit that does the opposite of the old one. That's the right tool for anything that's already been shared, because nobody's history gets rewritten.</p>
<p><strong>Stashing.</strong> You're in the middle of something and need a clean folder for a moment:</p>
<pre><code class="language-bash">git stash          # shelves your uncommitted changes
git stash pop      # brings them back
</code></pre>
<h3>The scary one: <code>reset --hard</code>, and the rescue</h3>
<p><code>git reset --hard</code> moves your branch back to an earlier commit and <strong>throws away</strong> the changes in your working folder. It's the one command that can really hurt. But watch what happens when I make a commit and then wipe it:</p>
<pre><code class="language-bash">git reset --hard HEAD~1
</code></pre>
<pre><code class="language-python">HEAD is now at 065b64b Revert "Add translate prompt (not needed after all)"
</code></pre>
<p>The "Add dates prompt" commit has vanished from the log, and its file is gone from the folder. But Git keeps a private diary of everywhere HEAD has been, called the <strong>reflog</strong>:</p>
<pre><code class="language-bash">git reflog
</code></pre>
<pre><code class="language-yaml">065b64b HEAD@{0}: reset: moving to HEAD~1
ed39bfa HEAD@{1}: commit: Add dates prompt
065b64b HEAD@{2}: revert: Revert "Add translate prompt (not needed after all)"
</code></pre>
<p>There's the lost commit, <code>ed39bfa</code>. Jump back to it:</p>
<pre><code class="language-bash">git reset --hard ed39bfa
</code></pre>
<pre><code class="language-python">HEAD is now at ed39bfa Add dates prompt
</code></pre>
<p>Everything is back, including the file. The lesson: <strong>if you ever committed something, it's almost certainly recoverable</strong>, at least for a while (the reflog keeps entries for weeks by default). What's <em>not</em> recoverable is work you never committed. That's one more reason to commit often.</p>
<h2>Branches and merging</h2>
<p>A <strong>branch</strong> lets you work on something without disturbing the main line of the project. This is how professionals work: <code>main</code> always holds working code, and every new idea gets its own branch.</p>
<pre><code class="language-bash">git switch -c feature/extract-emails
</code></pre>
<p>That creates a new branch and moves you onto it. (You'll see <code>git checkout -b</code> in older tutorials. <code>git switch</code> is the newer, clearer command for the same job.) Make a commit there, and <code>main</code> doesn't change. Then come back to <code>main</code> and <strong>merge</strong>:</p>
<pre><code class="language-bash">git switch main
git merge feature/extract-emails
</code></pre>
<pre><code class="language-bash">Updating ed39bfa..a5ac0a1
Fast-forward
 prompts/emails.txt | 1 +
</code></pre>
<p>A <strong>fast-forward</strong> is the simplest kind: <code>main</code> hadn't moved, so Git just slides its label forward to catch up with the branch. No new commit needed.</p>
<p>When both branches have moved, Git has to combine them with a <strong>merge commit</strong>. I added a tone prompt on one branch, and edited the README on <code>main</code>, then merged:</p>
<pre><code class="language-sql">Merge made by the 'ort' strategy.
</code></pre>
<p>And <code>git log --oneline --graph</code> draws the shape of what happened:</p>
<pre><code class="language-sql">*   4d633d0 Merge branch 'feature/tone-prompt'
|\  
| * 0dc7feb Add tone prompt
* | c1bdedd Describe prompts folder in README
|/  
* a5ac0a1 Add email extraction prompt
</code></pre>
<p>Two lines of work split apart and joined back together. That's the graph from Post 16, drawn in your terminal. Some tips on naming: use short, descriptive names like <code>feature/extract-emails</code>, <code>fix/retry-bug</code>, or <code>experiment/short-summary</code>.</p>
<h3>Merge conflicts</h3>
<p>Here's the moment that scares beginners, so let's meet it on purpose. What if two branches change the <em>same line</em>? That happens in AI projects all the time, because people tweak the same prompt. I changed <code>summarize.txt</code> to "one sentence" on one branch and to "three bullet points" on <code>main</code>, and then merged:</p>
<pre><code class="language-sql">Auto-merging prompts/summarize.txt
CONFLICT (content): Merge conflict in prompts/summarize.txt
Automatic merge failed; fix conflicts and then commit the result.
</code></pre>
<p>It isn't an error, and nothing is broken. Git just can't decide which version you want, so it asks you. If you open the file, you'll see both versions marked up:</p>
<pre><code class="language-markdown">&lt;&lt;&lt;&lt;&lt;&lt;&lt; HEAD
Summarize the text below in three bullet points.
=======
Summarize the text below in one sentence.
&gt;&gt;&gt;&gt;&gt;&gt;&gt; experiment/short-summary
</code></pre>
<p>The part between <code>&lt;&lt;&lt;&lt;&lt;&lt;&lt;</code> and <code>=======</code> is your side (the branch you're on). The part between <code>=======</code> and <code>&gt;&gt;&gt;&gt;&gt;&gt;&gt;</code> is the incoming side. To resolve it:</p>
<ol>
<li><p><strong>Edit the file</strong> to be what you actually want, and delete all three marker lines. You can keep one side, the other, or write something new that combines them. I wrote: "Summarize the text below in three short bullet points."</p>
</li>
<li><p><strong>Stage it</strong>: <code>git add prompts/summarize.txt</code>.</p>
</li>
<li><p><strong>Finish the merge</strong>: <code>git commit</code>.</p>
</li>
</ol>
<p>That's all a conflict is. VS Code highlights conflicts and offers buttons for "accept current" or "accept incoming," which makes it easier still. And if you panic halfway through, <code>git merge --abort</code> puts everything back exactly as it was before you started. (I tested that too.)</p>
<h3>Rebase: a tidier alternative</h3>
<p>There's a second way to combine branches, called <strong>rebase</strong>. Instead of joining two lines with a merge commit, it <em>replays</em> your branch's commits on top of the latest <code>main</code>, as if you'd started your work from there. Before:</p>
<pre><code class="language-markdown">* f92240e F2
* 64d00cc F1
| * 0e3344c M1       &lt;- main moved on
|/
* bbfda66 A
</code></pre>
<p>After <code>git rebase main</code> on the feature branch:</p>
<pre><code class="language-markdown">* 65730f4 F2
* ce26bed F1
* 0e3344c M1
* bbfda66 A
</code></pre>
<p>One straight line, with no merge commit. Many teams like the clean, linear history. But there's an important rule, which is the most important rule in this post:</p>
<blockquote>
<p><strong>Never rebase (or amend, or otherwise rewrite) commits that other people already have.</strong></p>
</blockquote>
<p>Rebasing creates <em>new</em> commits with new hashes. If someone else is building on the old ones, their history and yours no longer match, and untangling it is miserable. Rewriting your own private work is fine. Rewriting shared work is not. When in doubt, use <code>merge</code>.</p>
<h2>Working with a remote</h2>
<p>So far everything has been on your own computer. A <strong>remote</strong> is another copy of the repository, usually on GitHub, and by convention the main one is called <code>origin</code>.</p>
<table>
<thead>
<tr>
<th>Command</th>
<th>What it does</th>
</tr>
</thead>
<tbody><tr>
<td>git clone &lt;url&gt;</td>
<td>Copy a remote repository to your computer</td>
</tr>
<tr>
<td>git push</td>
<td>Send your new commits to the remote</td>
</tr>
<tr>
<td>git fetch</td>
<td>Download new commits from the remote, but don't touch your files</td>
</tr>
<tr>
<td>git pull</td>
<td>Fetch, then bring those commits into your branch</td>
</tr>
</tbody></table>
<p>You can't try GitHub without an account, but you can try the whole workflow offline, because a remote can just be another folder. I created a "bare" repository to stand in for GitHub, and cloned it twice, as two imaginary people, Alice and Bob. After Alice pushed the first commit, Bob cloned it and saw the same history.</p>
<p>Then they both made a commit. Alice pushed first. When Bob tried to push:</p>
<pre><code class="language-sql">! [rejected]        main -&gt; main (fetch first)
error: failed to push some refs to '...'
hint: Updates were rejected because the remote contains work that you do not
hint: have locally. This is usually caused by another repository pushing to
hint: the same ref. If you want to integrate the remote changes, use
hint: 'git pull' before pushing again.
</code></pre>
<p>This is Git protecting Alice's work. Bob's history and the remote's history have <strong>diverged</strong>, and he's not allowed to overwrite hers. He needs to bring her commit in first. After <code>git fetch</code>, you can see the fork:</p>
<pre><code class="language-sql">* 6ad0eae Add translate prompt         &lt;- Bob's commit
| * ca86132 Summaries are two sentences   &lt;- Alice's commit
|/
* 0eacee0 Initial commit
</code></pre>
<p>Now Bob has a choice. Either he merges, or he uses <code>git pull --rebase</code>, which replays his commit on top of Alice's:</p>
<pre><code class="language-sql">Successfully rebased and updated refs/heads/main.
* f73b44b Add translate prompt
* ca86132 Summaries are two sentences
* 0eacee0 Initial commit
</code></pre>
<p>Then his push goes through.</p>
<p>If you do a plain <code>git pull</code> on diverged branches, newer versions of Git stop and ask you to decide how, with a long hint:</p>
<pre><code class="language-yaml">hint: You have divergent branches and need to specify how to reconcile them.
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
fatal: Need to specify how to reconcile divergent branches.
</code></pre>
<p>That's not an error in your repo. It's just asking you to pick a default. Either choice is fine. Pick one and set it once with <code>git config --global pull.rebase true</code> (or <code>false</code>).</p>
<h3>Connecting to GitHub</h3>
<p>With a real GitHub account, you create an empty repository on the website, then connect your local one to it:</p>
<pre><code class="language-bash">git remote add origin https://github.com/YOUR-NAME/ai-engineer-journey.git
git push -u origin main
</code></pre>
<p>The <code>-u</code> flag remembers the connection, so later you can just type <code>git push</code> and <code>git pull</code>. One note on signing in: GitHub no longer accepts your account password for Git over HTTPS. You use either an <strong>SSH key</strong>, a <strong>personal access token</strong>, or the GitHub CLI (<code>gh auth login</code>), which sets it up for you. GitHub's setup pages walk through each, and they change from time to time, so follow their current instructions. (I can't reach GitHub from my test environment, so the GitHub-specific parts of this post are described from knowledge, while the Git commands were all run.)</p>
<h2>Pull requests</h2>
<p>On a team, you don't usually push straight to <code>main</code>. Instead you work on a branch, push that branch, and open a <strong>pull request</strong> (PR): a page on GitHub that says "I'd like to merge these changes into <code>main</code>. Please look at them." It shows exactly what changed, line by line, and gives people a place to discuss it. The usual flow is:</p>
<ol>
<li><p><strong>Branch</strong>: <code>git switch -c fix/retry-bug</code></p>
</li>
<li><p><strong>Commit</strong> your work in small, clear commits.</p>
</li>
<li><p><strong>Push</strong> the branch: <code>git push -u origin fix/retry-bug</code></p>
</li>
<li><p><strong>Open a pull request</strong> on GitHub.</p>
</li>
<li><p><strong>Review</strong>: teammates comment and ask for changes, and you push more commits to the same branch to address them.</p>
</li>
<li><p><strong>Checks pass</strong>: tests and linters run automatically on every PR (we'll set that up in Post 28).</p>
</li>
<li><p><strong>Merge</strong>, then delete the branch and update your local <code>main</code> with <code>git switch main &amp;&amp; git pull</code>.</p>
</li>
</ol>
<p>GitHub offers three ways to merge:</p>
<ul>
<li><p><strong>Merge commit</strong> keeps every commit on the branch, plus a merge commit.</p>
</li>
<li><p><strong>Squash and merge</strong> squashes the whole branch into a single commit on <code>main</code>. It's popular because it keeps <code>main</code>'s history tidy, one commit per PR.</p>
</li>
<li><p><strong>Rebase and merge</strong> replays the branch's commits onto <code>main</code>, with no merge commit.</p>
</li>
</ul>
<h3>A good pull request</h3>
<ul>
<li><p><strong>Small.</strong> A PR that changes 30 lines gets read carefully. A PR that changes 3,000 gets skimmed and approved. Aim for one idea per PR.</p>
</li>
<li><p><strong>Explained.</strong> Say what changed, why, and how you tested it. Many repos put a template in <code>.github/pull_request_template.md</code>, and GitHub fills it in automatically. Here's a good one for an AI project:</p>
</li>
</ul>
<pre><code class="language-markdown">## What changed

## Why

## How I tested it

## Checklist
- [ ] Tests pass
- [ ] No secrets or API keys in the diff
- [ ] Prompt changes are noted, with before/after examples
</code></pre>
<ul>
<li><p><strong>A draft first, if it's not ready.</strong> GitHub lets you open a "draft" PR to get early feedback.</p>
</li>
<li><p><strong>Linked to an issue</strong>, if there is one.</p>
</li>
</ul>
<p>Many teams also turn on <strong>branch protection</strong>, so that nobody can push straight to <code>main</code>, and merging requires a passing build and at least one approval.</p>
<h2>Code review</h2>
<p>Review is one of the best ways to learn, and one of the most underrated skills. You'll be doing it from both sides.</p>
<p><strong>As the author:</strong></p>
<ul>
<li><p>Review your own PR first. Read the "Files changed" tab as if someone else wrote it. You'll catch debugging leftovers and typos before anyone else sees them.</p>
</li>
<li><p>Don't take comments personally. They're about the code, and the goal is a better project.</p>
</li>
<li><p>Answer every comment, even if it's just "done."</p>
</li>
<li><p>Say thanks. It costs nothing.</p>
</li>
</ul>
<p><strong>As the reviewer:</strong></p>
<ul>
<li><p>Read the description first, so you know what the change is trying to do.</p>
</li>
<li><p>Comment on the code, never the person. "This could fail if the list is empty" beats "you forgot to handle empty lists."</p>
</li>
<li><p>Ask questions instead of giving orders when you're not sure: "What happens if the model returns an empty string here?"</p>
</li>
<li><p>Separate <strong>must-fix</strong> from <strong>nitpick.</strong> Prefix small style comments with "nit:", so the author knows what's optional.</p>
</li>
<li><p>Praise good work. A review that's only criticism teaches people to dread them.</p>
</li>
</ul>
<p>For AI projects specifically, here's a checklist worth keeping in your head:</p>
<ul>
<li><p><strong>Secrets.</strong> Any API key, token, or password in the diff? (Even in a "temporary" test.)</p>
</li>
<li><p><strong>Prompt changes.</strong> Was a prompt edited? Is there evidence it's better, such as examples or eval results (Post 60)? A one-word change to a prompt can shift a lot.</p>
</li>
<li><p><strong>Tests.</strong> Does new logic have tests, with fake models rather than real ones (Post 13)?</p>
</li>
<li><p><strong>Errors and retries.</strong> What happens on a timeout, a 429, or a malformed reply (Posts 11 and 20)?</p>
</li>
<li><p><strong>Cost.</strong> Does this make more model calls, or send much longer prompts (Post 18)?</p>
</li>
<li><p><strong>Privacy.</strong> Does it log prompts or user data that shouldn't be stored?</p>
</li>
<li><p><strong>Dependencies.</strong> Was a new library added, and is it a real, maintained one (Post 12)?</p>
</li>
<li><p><strong>Pinned versions.</strong> Is the model name or version fixed, so behavior doesn't change silently?</p>
</li>
</ul>
<p>AI assistants can be a useful extra pair of eyes on a PR, good at catching things you missed. Treat their comments like a quick first pass from a colleague. You're still the reviewer, and responsible for the result.</p>
<h2>Common workflows</h2>
<p>The most widely used pattern is simple, and sometimes called <strong>GitHub Flow</strong>:</p>
<ol>
<li><p><code>main</code> is always in a working state.</p>
</li>
<li><p>Every change happens on a short-lived branch.</p>
</li>
<li><p>Changes get merged through a pull request.</p>
</li>
<li><p>Delete the branch after merging.</p>
</li>
</ol>
<p>Keep branches short-lived, a few days at most. The longer a branch lives, the more it drifts from <code>main</code>, and the nastier the conflicts when you merge.</p>
<p>For open-source projects where you don't have permission to push, you use a <strong>fork</strong>: your own copy of the repository on GitHub. You work there, then open a PR from your fork to the original, which is called the <strong>upstream</strong>. That's exactly the process in Post 76, when we make your first open-source contribution.</p>
<h2>Mistakes you'll make (and how to survive them)</h2>
<p><strong>Committing a secret.</strong> This one deserves a clear warning, because the fix surprises people. Say you commit a file with an API key, notice, add it to <code>.gitignore</code>, and remove it from tracking:</p>
<pre><code class="language-bash">git rm --cached config.env
</code></pre>
<p>The file no longer shows up in the latest commit. But <strong>it's still in the history.</strong> Anyone who looks at the earlier commit can read it:</p>
<pre><code class="language-python">203c7b4 Stop tracking config.env
9f8f4d3 Add config          &lt;- the key is still readable here
</code></pre>
<p>Deleting a secret from the latest version doesn't remove it from the past. If a key was ever committed, and especially if you pushed it, <strong>treat it as stolen: revoke it and create a new one</strong>, as we said in Post 12. There are tools that rewrite history to scrub a file out, but rotating the key is the part that actually protects you. GitHub can also scan for known kinds of secrets and warn you or block a push, which is worth turning on.</p>
<p><strong>Committing something huge or junk.</strong> A <code>.venv/</code> folder, a dataset, a model file. Use <code>.gitignore</code> before you add, and remember GitHub rejects files over 100 MB.</p>
<p><strong>Force-pushing over other people's work.</strong> <code>git push --force</code> overwrites the remote's history with yours. After you rebase <em>your own</em> branch, you may need it, and the safer version is <code>git push --force-with-lease</code>, which refuses if someone else has pushed in the meantime. Never force-push to <code>main</code>.</p>
<p><strong>Working on</strong> <code>main</code> <strong>directly.</strong> It's fine for a personal scratch repo, but get in the habit of branching. It costs nothing, and it makes every experiment disposable.</p>
<h2>Git as a detective</h2>
<p>History isn't just for undoing. It's a database of <em>why</em> the code looks the way it does.</p>
<p><code>git blame</code> shows who last changed each line of a file, and in which commit:</p>
<pre><code class="language-bash">git blame prompts/summarize.txt
</code></pre>
<pre><code class="language-yaml">cf014ae8 (Asha Learner 2026-11-01 06:56:37 +0000 1) Summarize in a limerick.
</code></pre>
<p>It isn't about blaming anyone. It's about finding the commit message that explains <em>why</em> the line changed. <code>git log -- &lt;file&gt;</code> lists every commit that touched a file, and <code>git log -p</code> shows the changes themselves.</p>
<p><code>git bisect</code> is magic, and it's the binary search from Post 17 applied to your history. Suppose something worked a week ago and is broken now, across dozens of commits. Instead of checking each one, you tell Git a "good" commit and a "bad" one, and it checks out the commit in the middle. You say whether that one's good or bad, and it halves the range again, over and over, until it finds the first bad commit.</p>
<p>Better still, if you have a script that exits with 0 when things are fine and 1 when they're not (the exit codes from Post 19), Git can do the whole search for you. In a test repo, I made nine commits, and one of them set <code>temperature=7.0</code> in a settings file, which my check script rejects. Then:</p>
<pre><code class="language-bash">git bisect start HEAD &lt;first-commit&gt;
git bisect run python3 check.py
</code></pre>
<pre><code class="language-sql">running 'python3' 'check.py'
running 'python3' 'check.py'
running 'python3' 'check.py'
6f45376168c02c88eeab0f12ce447976ce49ca97 is the first bad commit
    Tweak settings
</code></pre>
<p>Three runs to search nine commits, and it named the guilty commit. When you're done, <code>git bisect reset</code> returns you to where you were. This is a great way to find which change made your model's output worse, if you have a script that can tell.</p>
<p><strong>Tags</strong> mark special commits, such as releases:</p>
<pre><code class="language-bash">git tag -a v0.2.0 -m "Second release"
git push --tags
</code></pre>
<p>A tag is a permanent name for one exact version, so you can always get back to what you shipped.</p>
<h2>Git for AI projects</h2>
<p>A few habits that are specific to our line of work:</p>
<ul>
<li><p><strong>Treat prompts as code.</strong> Keep them in files, commit changes with messages that say <em>why</em>, and review them in PRs.</p>
</li>
<li><p><strong>Commit your lock file</strong> (<code>uv.lock</code> or pinned requirements, Post 12), so everyone gets the same library versions.</p>
</li>
<li><p><strong>Keep secrets out</strong>, and commit <code>.env.example</code>.</p>
</li>
<li><p><strong>Small eval sets can live in the repo</strong>, but large datasets and model weights shouldn't. Store them elsewhere, and keep a note on where to get them.</p>
</li>
<li><p><strong>Notebooks make noisy diffs</strong>, since their saved outputs change constantly. A tool called <code>nbstripout</code> can remove outputs before committing.</p>
</li>
<li><p><strong>Tag the version that went to production</strong>, so you can reproduce what users saw.</p>
</li>
<li><p><strong>Link results to commits.</strong> When you record an eval result, record the commit hash too. Then "this score came from this exact code and prompt" is always answerable.</p>
</li>
</ul>
<h2>Mini project: put your journey on GitHub</h2>
<p>Time to do it for real. We'll use the repository from Post 3, and practice the full team workflow on your own.</p>
<p><strong>Step 1: create the repo.</strong> On GitHub, create a new repository named <code>ai-engineer-journey</code>. If you already made one in Post 3, use that. If you're making a fresh one locally, run <code>git init -b main</code> in a folder.</p>
<p><strong>Step 2: set up the basics.</strong> Make sure it has a <code>README.md</code> (your goals and start date), a Python <code>.gitignore</code>, and folders called <code>notes/</code> and <code>projects/</code>. Commit them with a clear message, connect the remote, and push:</p>
<pre><code class="language-bash">git add README.md .gitignore notes projects
git commit -m "Set up journey repo with README and folders"
git remote add origin https://github.com/YOUR-NAME/ai-engineer-journey.git
git push -u origin main
</code></pre>
<p><strong>Step 3: add a project on a branch.</strong> Don't commit straight to <code>main</code>. Do it the professional way:</p>
<pre><code class="language-bash">git switch -c add-textkit
</code></pre>
<p>Copy in the <code>textkit</code> package from Post 12 (without its <code>.venv</code> folder), and commit it in <strong>several small, logical commits</strong>: the package code, then the tests (Post 13), then the README for it. Then push the branch:</p>
<pre><code class="language-bash">git push -u origin add-textkit
</code></pre>
<p><strong>Step 4: open a pull request.</strong> Go to your repo on GitHub, and open a PR from <code>add-textkit</code> into <code>main</code>. Write a proper description: what, why, how tested. Then review your own PR on the "Files changed" tab, and leave yourself at least one comment. Finally, <strong>squash and merge</strong> it, delete the branch, and update your local copy:</p>
<pre><code class="language-bash">git switch main
git pull
</code></pre>
<p><strong>Step 5: make a conflict on purpose.</strong> Clone your repo into a second folder, pretending to be a teammate:</p>
<pre><code class="language-bash">cd ..
git clone https://github.com/YOUR-NAME/ai-engineer-journey.git ai-engineer-journey-2
</code></pre>
<p>In each copy, edit the <em>same line</em> of <code>README.md</code> differently, commit, and push one of them. In the other, try to push (it'll be rejected), then <code>git pull</code>, resolve the conflict in the file, <code>git add</code>, commit, and push. After you've done it once, conflicts stop being scary.</p>
<p><strong>Step 6: add a PR template.</strong> Create <code>.github/pull_request_template.md</code> with the template from earlier, commit it on a branch, and open a PR. Open one more PR afterward, and watch GitHub fill in the template.</p>
<p><strong>Step 7: tag a release.</strong> Run <code>git tag -a v0.1.0 -m "First release"</code> and <code>git push --tags</code>.</p>
<p>When you're done, <code>git log --oneline --graph</code> should show a history with real structure, and your GitHub profile should have a repository that shows exactly how you work: small commits, clear messages, PRs, and a tidy README. That's something a hiring manager can read.</p>
<h3>Make it yours</h3>
<ol>
<li><p>Write a <code>CONTRIBUTING.md</code> that explains how to branch, name commits, and open a PR.</p>
</li>
<li><p>Add a <code>.env.example</code> file and make sure <code>.env</code> is ignored. Check with <code>git check-ignore -v .env</code>.</p>
</li>
<li><p>Try <code>git bisect</code> on your own project by deliberately breaking something in the middle of the history.</p>
</li>
<li><p>Install <code>pre-commit</code> and add a hook that blocks commits containing something that looks like an API key.</p>
</li>
<li><p>Ask a friend to review one of your PRs, or review one of theirs.</p>
</li>
</ol>
<h2>Practice</h2>
<p>About 20 minutes of real effort on each before asking for help.</p>
<ol>
<li><p>Make a repo, commit three changes to one file, and use <code>git log -p</code> to read the story they tell.</p>
</li>
<li><p>Break a file on purpose in three different ways, and fix each using <code>restore</code>, <code>revert</code>, and <code>reset --hard</code> with the reflog.</p>
</li>
<li><p>Create two branches that edit the same line of a prompt file. Merge them, resolve the conflict, and read the graph with <code>git log --oneline --graph --all</code>.</p>
</li>
<li><p>Practice the full Alice-and-Bob exercise with a local bare repo (<code>git init --bare</code>) and two clones. Get a rejected push, then fix it with <code>pull --rebase</code>.</p>
</li>
<li><p>Rewrite five bad commit messages into good ones.</p>
</li>
<li><p>Commit a fake secret, remove it, and then find it in the history. Write down exactly what you'd do next in a real case.</p>
</li>
<li><p>Write the review comments you'd leave on a PR that logs full user prompts, and has no tests. Make them kind and specific.</p>
</li>
<li><p>Use <code>git bisect run</code> with a tiny script to find the commit that broke something you planted.</p>
</li>
</ol>
<p>Then ask an AI tutor to review your answers, and don't let it rewrite them.</p>
<h2>What to take away</h2>
<ul>
<li><p>Git tracks your history on your computer. GitHub hosts copies and adds collaboration.</p>
</li>
<li><p>A commit is a snapshot, branches are movable labels, and files move from working directory to staging area to repository.</p>
</li>
<li><p><code>git status</code> before everything. Commit small and often, with clear messages that explain why.</p>
</li>
<li><p>Use <code>.gitignore</code> for secrets, environments, and big files. A committed secret stays in history, so revoke it.</p>
</li>
<li><p>Almost everything is undoable: <code>restore</code>, <code>amend</code>, <code>revert</code>, <code>stash</code>, and <code>reflog</code> as a rescue. Only uncommitted work is truly lost.</p>
</li>
<li><p>Branch for every change. Merge conflicts are just Git asking you to choose. Never rewrite history that others have.</p>
</li>
<li><p>Pull requests with small changes, good descriptions, and kind, specific reviews are how teams stay sane.</p>
</li>
<li><p><code>blame</code>, <code>log</code>, and <code>bisect</code> turn your history into a debugging tool.</p>
</li>
</ul>
<hr />
<p><strong>Previous:</strong> Post 21, <em>Databases in Depth</em></p>
<p><strong>Next in the series:</strong> Post 23, <em>Linux, Shell &amp; Docker</em>, the daily toolkit: working in the terminal, writing shell commands, and packaging your app so it runs the same everywhere.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/git-github-workflows-branches-pull-requests-and-code-review-ntiub?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Your Docker Image Is 1.2 GB. Here's How to Get It Under 100 MB]]></title><description><![CDATA[Most oversized Docker images come from the same few habits. Here's how to measure yours, fix the big ones with a slim base and a multi-stage build, and…]]></description><link>https://blog.zyvop.com/your-docker-image-is-1-2-gb-here-s-how-to-get-it-under-100-mb-9oiu3</link><guid isPermaLink="true">https://blog.zyvop.com/your-docker-image-is-1-2-gb-here-s-how-to-get-it-under-100-mb-9oiu3</guid><category><![CDATA[containers]]></category><category><![CDATA[Devops]]></category><category><![CDATA[Docker]]></category><category><![CDATA[optimization]]></category><category><![CDATA[Python]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Fri, 06 Nov 2026 07:35:08 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/76436d3e-ba67-46cc-9663-339bc960a578.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Your CI spends a minute pulling a 1.2 GB image. Deploys wait on it. A fresh node has to download it before it can serve a single request.</p>
<p>Most bloated images share the same handful of causes, and each one has a cheap fix. This post walks through them in order of payoff, using one realistic Python image.</p>
<p>First, a definition, because "image size" means two different things.</p>
<h2>Two sizes, and how to see both</h2>
<ul>
<li><p><strong>Compressed size</strong> is what the registry stores and what you download. Docker Hub shows this number.</p>
</li>
<li><p><strong>Unpacked size</strong> is what the image takes on disk once it's pulled.</p>
</li>
</ul>
<p>The unpacked number is always bigger. Docker's own docs show a small image at 12.8 MB unpacked and 3.99 MB of compressed content.</p>
<p>On Docker Engine 29 and later, <code>docker image ls</code> shows both, as <strong>DISK USAGE</strong> (unpacked) and <strong>CONTENT SIZE</strong> (compressed). Older engines show a single size, which is the unpacked one.</p>
<pre><code class="language-bash">docker image ls myapp                      # sizes, in two columns on Docker 29 and later
docker history myapp:before                # how much each layer added
docker save myapp:before | gzip | wc -c   # a rough compressed size on older engines
</code></pre>
<p>For a visual tour of every layer and its files, try the open-source tool <a href="https://github.com/wagoodman/dive">dive</a>.</p>
<p>In this post, <strong>"under 100 MB" means compressed</strong>, the size you push and pull. Unpacked will be larger, so always look at both.</p>
<h2>Where 1.2 GB comes from</h2>
<p>Here's a small FastAPI service. Two files:</p>
<pre><code class="language-python"># main.py
from fastapi import FastAPI

app = FastAPI()

@app.get("/health")
def health():
    return {"status": "ok"}
</code></pre>
<pre><code class="language-text"># requirements.txt  (pin your versions in a real project)
fastapi
uvicorn[standard]
</code></pre>
<p>And the Dockerfile that most of us write first:</p>
<pre><code class="language-dockerfile">FROM python:3.14
WORKDIR /app
COPY . .
RUN apt-get update &amp;&amp; apt-get install -y build-essential libpq-dev curl
RUN pip install -r requirements.txt
RUN apt-get clean
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
</code></pre>
<p>It works. It's also carrying a lot of weight:</p>
<ul>
<li><p><strong>A full base image.</strong> <code>python:3.14</code> is a complete Debian system with compilers and headers.</p>
</li>
<li><p><strong>Build tools in the final image.</strong> <code>build-essential</code> and <code>libpq-dev</code> are needed to compile things, not to run them.</p>
</li>
<li><p><code>COPY . .</code> <strong>with no</strong> <code>.dockerignore</code><strong>.</strong> Your <code>.git</code> folder, virtualenv and local files all ride along.</p>
</li>
<li><p><strong>No</strong> <code>--no-install-recommends</code>, so <code>apt</code> adds suggested extras.</p>
</li>
<li><p><strong>The package lists and pip's cache</strong> stay in the image.</p>
</li>
<li><p><strong>A cleanup in its own layer.</strong> <code>RUN apt-get clean</code> on its own line shrinks nothing, as you'll see below.</p>
</li>
</ul>
<p>How big is the base alone? These are the compressed sizes Docker Hub lists for the <code>linux/amd64</code> builds of the 3.14 tags, checked on October 31, 2026:</p>
<table>
<thead>
<tr>
<th>Base image</th>
<th>Compressed size</th>
</tr>
</thead>
<tbody><tr>
<td>python:3.14 (the full image)</td>
<td>about 400 MB</td>
</tr>
<tr>
<td>python:3.14-slim</td>
<td>about 42 MB</td>
</tr>
<tr>
<td>python:3.14-alpine</td>
<td>about 17 MB</td>
</tr>
</tbody></table>
<p>Published comparisons put full Python images at roughly 800 MB to 1 GB once unpacked. Add the toolchain, the caches and your whole project folder, and 1.2 GB arrives quickly.</p>
<blockquote>
<p><strong>Which Python?</strong> The examples use 3.14. Python 3.15 is out now, and <code>latest</code> points to it. Use the newest version your dependencies publish wheels for, because a missing wheel means compiling from source, and a slim image has no compiler.</p>
</blockquote>
<h3>A scoreboard before you start</h3>
<p>Instead of guessing, run a checker over the Dockerfile. The one at the end of this post looks only at the habits that cost size or build time:</p>
<pre><code class="language-bash">python dockerfile_check.py Dockerfile
</code></pre>
<pre><code class="language-text">Dockerfile:0  no-dockerignore  No .dockerignore next to the Dockerfile, so .git, virtualenvs and local files all go into the build context.
Dockerfile:1  heavy-base  python:3.14 is the full image. Use a -slim or -alpine tag for the stage that ships.
Dockerfile:4  apt-lists  The apt package lists stay in the image. End the same RUN with rm -rf /var/lib/apt/lists/*.
Dockerfile:4  apt-recommends  apt-get install without --no-install-recommends pulls in suggested extras.
Dockerfile:4  build-tools-in-final-stage  Build tools in the shipped image: build-essential, libpq-dev. Install them in a builder stage.
Dockerfile:5  copy-before-install  Dependencies install after COPY . (line 3), so any code change reinstalls them. Copy the dependency files first.
Dockerfile:5  pip-cache  pip keeps its download cache in the layer. Add --no-cache-dir, or use a cache mount.
Dockerfile:6  cleanup-in-own-layer  Deleting files in a later layer doesn't shrink the image. Do the cleanup in the same RUN that made the mess.
8 thing(s) to fix
</code></pre>
<p>Eight findings. Each one maps to a fix below, and by the end the output will read <code>Nothing found</code>.</p>
<h2>Fix 1: Start from a smaller base</h2>
<p>This is the biggest single win, and it's one line. The Docker docs say a smaller base brings portability and fast downloads, shrinks your image, and cuts the number of vulnerabilities you inherit through dependencies.</p>
<p>Your options for a Python app:</p>
<ul>
<li><p><strong>Full (</strong><code>python:3.14</code><strong>).</strong> Everything included. Fine for a builder stage, wasteful for the image you ship.</p>
</li>
<li><p><strong>Slim (</strong><code>python:3.14-slim</code><strong>).</strong> Debian with the extras removed. It's the sensible default.</p>
</li>
<li><p><strong>Alpine (</strong><code>python:3.14-alpine</code><strong>).</strong> Smallest, but it uses musl instead of glibc. The official image page recommends it when final size is your primary concern, warns that software can hit libc compatibility issues, and notes that tools such as git or bash usually aren't included.</p>
</li>
<li><p><strong>Distroless or</strong> <code>scratch</code><strong>.</strong> No shell, no package manager. Best for static binaries like Go.</p>
</li>
</ul>
<p><img src="https://mermaid.ink/img/pako:eNpVkMtu20AMRX_lluvRokBXXiSIo7rNIl3EfaDweEFJlEVkpBGGo7iq438v5KAJsiRxD-8BT1THRmhFbYjHuuOU8b30AwDc7Dz96rTuULEJ2piQO4H2fBDkjjOs09GuPe1RFFdYnzzdGTSDYZmz1qh04DQ7BH0UfImICQ-T5WtP55eO9UI-e_ot5ukZtztPjVpOMYjZ_ysxwerEue487d9z3-KClSdPZQSHgDlOCY2MMjQy1Cp2kUQ_WcCxEwnmwEMDNciTpBm9HLias2BMUmuc7E2ufC_3eeeJw6iDvGqUrxoZNiVZYpudJwvae9qTo15Sz9rQ6kS5k375dCMtTyGTe9n85KRcBbEl08Yhb7jXMNOKCh7HIIXNlqV3WAcdHu-53l7mTRyyg6etHKLgx50nh4dYxRwdvkp4kqw1O9wk5eBgPFhhkrQldynZ6t_F5eOn8Q-dz46qw20MMdGKPhw7zULnf1kVsic?type=png" alt="Mermaid Diagram" /></p>
<p>Two caveats from the official sources. The Python image page itself recommends the full image unless you have space constraints, because slim has no compilers, so <code>pip install</code> can fail for packages that need building from source. And Docker's own docs point to Alpine as a small general-purpose base.</p>
<p>For a Python app where size matters, the builder stage in Fix 2 is where you add a compiler, or switch to the full image, if a dependency needs building from source. None of it reaches the image you ship.</p>
<p>And slim avoids the musl issues behind the second caveat. So when in doubt, use <strong>slim</strong>. Alpine saves a few more megabytes, but it's the one most likely to turn a two-minute build into an afternoon of debugging, as the gotchas section shows.</p>
<h2>Fix 2: Build in one stage, ship from another</h2>
<p>Multi-stage builds are the other big lever. You build in a heavy image, then copy only the results into a clean one. Docker's docs say it reduces the size of the final image by separating the build from the output, and note that a production image often doesn't need compilers or other build tools.</p>
<p><img src="https://mermaid.ink/img/pako:eNplUF1Lw0AQ_CvrPl8QxaeghX5QLChKi4Lk-rBJNs3q5S7cXVpr2_8uaaGi7tsMszPD7LBwJWOKlXGboiYf4WGuLQBA6PKVp7aGUSemZJ9pXERaMVylkJ8ojcuTtr9hpnFmQyRjoOSWbcm2EA63ub8ciI0OCNbiY0eG7fr8yrb8kzcVS-acdp1CrBmk6cHWdRBqaX8FjzONQygMk4VgpIGcAmtcQpIMYJJpHLt2C2LhvQvxaPZT41iObNk7e-i3-F9s2BvtNY6fnt8gSSrvmrvzAHuYoMKGfUNSYrrDWHPTD1pyRZ2JqE7MK3mh3HDoNZWzcUqNmC2mmFDbGk7CNkRuFIyM2I9HKhZHPHU2KtC44JVjeJlpVDB3uYtOwT2bNUcpSMHQCxkFgWxIAnupUB1DFvLVd7m6aT_xcFCYr8bOOI8pXmxqiYyHb8T3qO8?type=png" alt="Mermaid Diagram" /></p>
<p>For Python the trick is a virtualenv. Create it in the builder, install into it, then copy the whole folder across. Everything the build needed stays behind: compilers, headers, pip's cache, intermediate files.</p>
<p>Two rules keep it working. Use the same Python version and the same OS family in both stages, so the interpreter and system libraries match. For example, build on <code>python:3.14</code> and ship on <code>python:3.14-slim</code>. And keep the virtualenv at the same path in both.</p>
<h2>Fix 3: Clean up in the same layer</h2>
<p>Every instruction creates a layer, and layers only add. A file deleted in a later layer is hidden from the container, but it's still stored in the earlier layer, and you still ship and pull it.</p>
<p><img src="https://mermaid.ink/img/pako:eNpVj8tqw0AMRX9F1XpMSenKlEAcCC00XeTRTZyF7JFtUXnGzEySpiH_XuJAoUuJq3OPLlh7y5hjo_5UdxQSbIrSAQDMdiW-05kDTHJYbT-AhpS1nEBcTKQK1UHUZhwjuySkL1V4nJK1EQgaPkF3cDawhWVR4h6ybArFH_HpP7FWJjfed2I5QiPK0UDg3h85gvOpE9eWuL-bFSNtvitx0zFITy1DTKIKtXeJxEXQuziIg-agWuIeDfYcehKL-QVTx_3tbcsNHTShuW8-KQhVyvGWabxLC-pFz5hjRsOgnMVzTNwbKFTc15Lq9TgvvEsGSlxz6xm2byUaWPnKJ2_glfXISWoyMAtCaiCSi1nkIA2asWQtPzeXyfPwjderwaqde_UBc3w4dZIYr78iFY0v?type=png" alt="Mermaid Diagram" /></p>
<p>That's what the <code>cleanup-in-own-layer</code> finding is about. The fix is to clean up in the same <code>RUN</code> that made the mess. The Docker docs show this pattern for <code>apt</code>:</p>
<pre><code class="language-dockerfile">RUN apt-get update &amp;&amp; apt-get install -y --no-install-recommends curl \
    &amp;&amp; rm -rf /var/lib/apt/lists/*
</code></pre>
<p><code>--no-install-recommends</code> skips suggested packages. The <code>rm</code> runs before the layer is saved, so the lists never reach it.</p>
<p>For pip, either pass <code>--no-cache-dir</code>, or use a BuildKit cache mount, which keeps pip's cache outside the image and makes rebuilds faster too:</p>
<pre><code class="language-dockerfile">RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt
</code></pre>
<h2>Fix 4: Add a .dockerignore</h2>
<p>Without one, <code>COPY . .</code> sends your entire folder to the builder and copies it into the image. That includes <code>.git</code>, <code>.venv</code>, test data, and a <code>.env</code> file with secrets you'd rather not bake into a layer.</p>
<p>A <code>.dockerignore</code> uses patterns much like <code>.gitignore</code>. Here's a starting point:</p>
<pre><code class="language-text">.git
.gitignore
.venv/
venv/
__pycache__/
*.pyc
.env
.pytest_cache/
tests/
docs/
*.md
Dockerfile*
.dockerignore
</code></pre>
<p>It also makes the build faster, because there's less to send to the builder before anything starts.</p>
<h2>Fix 5: Copy dependency files first</h2>
<p>This one doesn't shrink the image, but it saves your afternoon. Docker caches each layer, and a layer is rebuilt when anything above it changes.</p>
<p>If you <code>COPY . .</code> first and install dependencies second, every code edit invalidates the install step. Copy only <code>requirements.txt</code>, install, and copy the code last. That's the <code>copy-before-install</code> finding.</p>
<h2>Fix 6: Ship only what you run</h2>
<p>Dev dependencies, test frameworks and docs belong in CI, not production. Most tools have a switch:</p>
<table>
<thead>
<tr>
<th>Tool</th>
<th>In the image</th>
</tr>
</thead>
<tbody><tr>
<td>pip</td>
<td>Keep a runtime-only requirements.txt</td>
</tr>
<tr>
<td>uv</td>
<td>uv sync --frozen --no-dev</td>
</tr>
<tr>
<td>npm</td>
<td>npm ci --omit=dev</td>
</tr>
</tbody></table>
<p>If you use uv, its Docker guide covers the pieces below. <code>UV_COMPILE_BYTECODE=1</code> compiles bytecode, which tends to improve startup time. A cache mount keeps uv's cache out of the image. And <code>--no-install-project</code> installs dependencies in their own layer.</p>
<p>Pin the uv version in real projects, as the guide advises. The two-stage layout is the same idea as the Python example above:</p>
<pre><code class="language-dockerfile">FROM python:3.14-slim AS builder
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/   # pin a specific uv version in real projects
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --no-install-project --no-dev
COPY . .
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --no-dev

FROM python:3.14-slim
COPY --from=builder /app /app
ENV PATH="/app/.venv/bin:$PATH"
WORKDIR /app
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
</code></pre>
<h2>Fix 7: Static binaries can go much smaller</h2>
<p>Compiled languages like Go can produce a single static binary, and then the base image barely matters. Build in the full toolchain image, ship from distroless:</p>
<pre><code class="language-dockerfile"># syntax=docker/dockerfile:1

FROM golang:1.27 AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /server .

FROM gcr.io/distroless/static-debian13:nonroot
COPY --from=builder /server /server
EXPOSE 8080
ENTRYPOINT ["/server"]
</code></pre>
<p>Three details do the work:</p>
<ul>
<li><p><code>CGO_ENABLED=0</code> makes the binary fully static, with no C library needed.</p>
</li>
<li><p><code>-ldflags="-s -w"</code> strips the symbol table and debug information.</p>
</li>
<li><p><code>gcr.io/distroless/static</code> holds the bare minimum to run a static binary: CA certificates, timezone data and a <code>/tmp</code> directory. The <code>:nonroot</code> tag runs as an unprivileged user.</p>
</li>
</ul>
<p>Check the distroless project for the current tag names. Docker's docs also describe pinning a base image by digest, which makes builds reproducible but opts you out of automatic security updates, unless a tool like Dependabot bumps the digest for you.</p>
<p>The trade-off is debugging. Distroless images have no shell and no package manager. For local debugging there's a <code>:debug</code> variant that includes a BusyBox shell, which you should never ship.</p>
<h2>Put it together</h2>
<p>Here's the Python image after the fixes:</p>
<pre><code class="language-dockerfile"># syntax=docker/dockerfile:1

FROM python:3.14-slim AS builder
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

FROM python:3.14-slim
RUN useradd --create-home --uid 10001 app
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH" \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1
WORKDIR /app
COPY --chown=app:app main.py .
USER app
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
</code></pre>
<p>Compare it with where we started:</p>
<ul>
<li><p><strong>The base is slim</strong>, in both stages.</p>
</li>
<li><p><strong>The builder holds the toolchain.</strong> Only a copy of the virtualenv crosses over.</p>
</li>
<li><p><strong>There's no</strong> <code>COPY . .</code><strong>.</strong> Just <code>main.py</code>, owned by a non-root user.</p>
</li>
<li><p><strong>Dependencies install from</strong> <code>requirements.txt</code> <strong>alone</strong>, so code changes don't reinstall them.</p>
</li>
<li><p><strong>pip's cache lives in a cache mount</strong>, not a layer.</p>
</li>
</ul>
<p>Add the <code>.dockerignore</code> from Fix 4 and run the checker again:</p>
<pre><code class="language-text">Nothing found
</code></pre>
<p>Then build both versions and compare:</p>
<pre><code class="language-bash">docker build -f Dockerfile.before -t myapp:before .
docker build -f Dockerfile.after  -t myapp:after  .
docker image ls myapp
</code></pre>
<p>The slim base is about 42 MB compressed, and FastAPI with its dependencies adds to that. For a service like this, the total should land comfortably under 100 MB compressed, and well under the full-image version. Your exact numbers depend on your dependencies and platform, so use the two columns from <code>docker image ls</code> to find out.</p>
<h2>Gotchas</h2>
<table>
<thead>
<tr>
<th>Gotcha</th>
<th>What goes wrong</th>
<th>What to do</th>
</tr>
</thead>
<tbody><tr>
<td>Alpine and wheels</td>
<td>Alpine uses musl, not glibc. Python wheels built for it follow PEP 656, and a package without one is compiled from source, which needs compilers and slows the build</td>
<td>Check your dependencies publish musllinux wheels, or stay on slim</td>
</tr>
<tr>
<td>Missing runtime libraries</td>
<td>A slim image has fewer system libraries, so an import can fail at startup</td>
<td>Install the runtime package (libpq5), not the -dev one, in the final stage</td>
</tr>
<tr>
<td>Copying a virtualenv</td>
<td>It breaks if the Python version or OS family differs between stages, or if the path changes</td>
<td>Same Python and OS family in both stages, same path</td>
</tr>
<tr>
<td>No shell</td>
<td>docker exec ... sh and a curl-based HEALTHCHECK don't work in distroless or scratch</td>
<td>Use the :debug tag locally, and a health endpoint that the orchestrator probes</td>
</tr>
<tr>
<td>scratch has nothing</td>
<td>No CA certificates, no timezone data, no user database</td>
<td>Use distroless static, or copy those files in</td>
</tr>
<tr>
<td>File ownership</td>
<td>Files copied as root can't be written by a non-root user</td>
<td>COPY --chown=app:app</td>
</tr>
<tr>
<td>Smaller isn't patched</td>
<td>A small image can still be out of date</td>
<td>Pin versions, rebuild on a schedule, and use docker build --pull to fetch fresh base images</td>
</tr>
</tbody></table>
<h2>Run a real linter too</h2>
<p>My checker is a teaching tool. It's small enough to read in one sitting, and it covers size and caching only. For CI, use <a href="https://github.com/hadolint/hadolint">hadolint</a>, which parses the Dockerfile properly and runs ShellCheck on your <code>RUN</code> lines:</p>
<pre><code class="language-bash">hadolint Dockerfile
</code></pre>
<p>Several of the rules overlap with hadolint's:</p>
<table>
<thead>
<tr>
<th>This post's rule</th>
<th>hadolint rule</th>
</tr>
</thead>
<tbody><tr>
<td>apt-recommends</td>
<td>DL3015: avoid additional packages by specifying --no-install-recommends</td>
</tr>
<tr>
<td>apt-lists</td>
<td>DL3009: delete the apt-get lists after installing something</td>
</tr>
<tr>
<td>pip-cache</td>
<td>DL3042: avoid the cache directory with pip install --no-cache-dir</td>
</tr>
<tr>
<td>unpinned-tag</td>
<td>DL3006 and DL3007: always tag the image version, and don't use latest</td>
</tr>
</tbody></table>
<h2>The checker and its tests</h2>
<p>Here's the whole checker. It splits a Dockerfile into stages and applies size rules only to what ships. A mess in a builder stage is fine, because it never reaches the final image.</p>
<pre><code class="language-python">import re
import sys
from dataclasses import dataclass, field
from pathlib import Path

HEAVY_IMAGES = {"python", "node", "ruby", "php", "golang", "rust", "openjdk"}
LIGHT_MARKERS = ("slim", "alpine", "minimal")
BUILD_PACKAGES = {"build-essential", "gcc", "g++", "make", "cmake", "pkg-config"}
INSTALL_STEP = re.compile(r"\b(pip3?\s+install|npm\s+(ci|install)|yarn\s+install|uv\s+sync|poetry\s+install|go\s+mod\s+download|bundle\s+install)\b")
APT_INSTALL = re.compile(r"\bapt(?:-get)?\s+(?:-\S+\s+)*install\s+([^&amp;;|]*)")
CLEANUP = re.compile(r"^(rm\s+-\S+\s+\S+.*|apt-get\s+(clean|autoremove)\b.*|pip\s+cache\s+purge)$")

@dataclass(frozen=True)
class Finding:
    line: int
    rule: str
    message: str

@dataclass
class Stage:
    name: str | None
    line: int
    image: str
    steps: list = field(default_factory=list)  # (line, KEYWORD, arguments)

def parse(text):
    """Instructions as (line, KEYWORD, arguments), with continuation lines joined.
    Heredocs (RUN &lt;&lt;EOF) aren't supported."""
    steps, buffer, start = [], "", 0
    for number, raw in enumerate(text.splitlines(), start=1):
        line = raw.rstrip()
        if not buffer and (not line.strip() or line.lstrip().startswith("#")):
            continue
        if not buffer:
            start = number
        if line.endswith("\\"):
            buffer += line[:-1] + " "
            continue
        buffer += line
        keyword, _, args = buffer.strip().partition(" ")
        steps.append((start, keyword.upper(), args.strip()))
        buffer = ""
    return steps

def split_stages(text):
    stages = []
    for line, keyword, args in parse(text):
        if keyword == "FROM":
            parts = [p for p in args.split() if not p.startswith("--")]
            name = parts[2].lower() if len(parts) &gt;= 3 and parts[1].upper() == "AS" else None
            stages.append(Stage(name, line, parts[0]))
        elif stages:
            stages[-1].steps.append((line, keyword, args))
    return stages

def parse_image(ref):
    """'ghcr.io/org/app:1.2' -&gt; ('app', '1.2', False). The last flag says whether a digest pins it."""
    pinned = "@" in ref
    ref = ref.split("@", 1)[0]
    name, colon, tag = ref.rpartition(":")
    if not colon or "/" in tag:  # no tag, or that colon was a registry port
        name, tag = ref, ""
    return name.rsplit("/", 1)[-1], tag, pinned

def image_chain(stages):
    """The final stage, then every earlier stage it builds on (FROM builder ...)."""
    by_name = {s.name: s for s in stages if s.name}
    chain = [stages[-1]]
    while chain[-1].image.lower() in by_name and by_name[chain[-1].image.lower()] not in chain:
        chain.append(by_name[chain[-1].image.lower()])
    return chain

def is_copy_all(args):
    sources = [a for a in args.split() if not a.startswith("--")]
    return bool(sources) and "--from" not in args and sources[0] in (".", "./")

def check(text, has_dockerignore=True):
    stages = split_stages(text)
    if not stages:
        return []
    findings, aliases = [], {s.name for s in stages if s.name}

    if not has_dockerignore:
        findings.append(Finding(0, "no-dockerignore", "No .dockerignore next to the Dockerfile, so .git, virtualenvs and local files all go into the build context."))

    for stage in stages:
        repo, tag, pinned = parse_image(stage.image)
        if stage.image.lower() in aliases or stage.image.lower() == "scratch" or "$" in stage.image:
            continue
        if not pinned and tag in ("", "latest"):
            findings.append(Finding(stage.line, "unpinned-tag", f"{stage.image} has no version tag, so the base can change under you."))

    chain = image_chain(stages)  # only the final image ships: leftovers in a builder stage don't count
    repo, tag, _ = parse_image(chain[-1].image)
    if repo in HEAVY_IMAGES and "$" not in chain[-1].image and not any(marker in tag for marker in LIGHT_MARKERS):
        findings.append(Finding(chain[-1].line, "heavy-base", f"{chain[-1].image} is the full image. Use a -slim or -alpine tag for the stage that ships."))

    for stage in stages:  # build-cache order matters in every stage
        copied_all = None
        for line, keyword, args in stage.steps:
            if keyword == "COPY" and is_copy_all(args):
                copied_all = line
            elif keyword == "RUN" and copied_all and INSTALL_STEP.search(args):
                findings.append(Finding(line, "copy-before-install", f"Dependencies install after COPY . (line {copied_all}), so any code change reinstalls them. Copy the dependency files first."))
                copied_all = None

    pip_cache_off, previous = False, None
    for stage in reversed(chain):
        for line, keyword, args in stage.steps:
            if keyword == "ENV" and "PIP_NO_CACHE_DIR" in args:
                pip_cache_off = True
            if keyword == "RUN":
                install = APT_INSTALL.search(args)
                if install:
                    if "--no-install-recommends" not in args and "Install-Recommends=false" not in args:
                        findings.append(Finding(line, "apt-recommends", "apt-get install without --no-install-recommends pulls in suggested extras."))
                    if "/var/lib/apt/lists" not in args:
                        findings.append(Finding(line, "apt-lists", "The apt package lists stay in the image. End the same RUN with rm -rf /var/lib/apt/lists/*."))
                    tools = [p.split("=")[0] for p in install.group(1).split() if not p.startswith("-")]
                    tools = [p for p in tools if p in BUILD_PACKAGES or p.endswith("-dev")]
                    if tools:
                        findings.append(Finding(line, "build-tools-in-final-stage", f"Build tools in the shipped image: {', '.join(tools)}. Install them in a builder stage."))
                if re.search(r"(?&lt;!uv )pip3?\s+install", args) and not (pip_cache_off or "--no-cache-dir" in args or "--mount=type=cache" in args):
                    findings.append(Finding(line, "pip-cache", "pip keeps its download cache in the layer. Add --no-cache-dir, or use a cache mount."))
                if previous in ("RUN", "COPY", "ADD") and all(CLEANUP.match(part.strip()) for part in args.split("&amp;&amp;")):
                    findings.append(Finding(line, "cleanup-in-own-layer", "Deleting files in a later layer doesn't shrink the image. Do the cleanup in the same RUN that made the mess."))
            previous = keyword
    return sorted(findings, key=lambda f: (f.line, f.rule))

def main(argv):
    path = Path(argv[1] if len(argv) &gt; 1 else "Dockerfile")
    findings = check(path.read_text(), (path.parent / ".dockerignore").exists())
    for f in findings:
        print(f"{path}:{f.line}  {f.rule}  {f.message}")
    print(f"{len(findings)} thing(s) to fix" if findings else "Nothing found")
    return 1 if findings else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv))
</code></pre>
<p>Its limits are worth knowing. It doesn't understand heredocs (<code>RUN &lt;&lt;EOF</code>), and it skips <code>FROM</code> lines that use build arguments instead of expanding them. It's a set of heuristics, so treat a finding as a prompt to look, not a verdict.</p>
<p>The tests use the Dockerfiles from this post as fixtures. The <code>before</code> file must trigger every rule, and the <code>after</code> and Go files must come out clean:</p>
<pre><code class="language-python">import contextlib
import io
import tempfile
from pathlib import Path

from dockerfile_check import check, main, parse, parse_image

HERE = Path(__file__).parent / "examples"

def rules(text, dockerignore=True):
    return sorted({f.rule for f in check(text, dockerignore)})

def test_parse_joins_continued_lines_and_skips_comments():
    text = "# a comment\nFROM python:3.14-slim\n\nRUN apt-get update &amp;&amp; \\\n    apt-get install -y curl\nCMD [\"x\"]\n"
    assert parse(text) == [
        (2, "FROM", "python:3.14-slim"),
        (4, "RUN", "apt-get update &amp;&amp;      apt-get install -y curl"),
        (6, "CMD", '["x"]'),
    ]

def test_parse_image():
    assert parse_image("python:3.14-slim") == ("python", "3.14-slim", False)
    assert parse_image("python") == ("python", "", False)
    assert parse_image("registry:5000/team/app") == ("app", "", False)  # that colon is a port, not a tag
    assert parse_image("ghcr.io/astral-sh/uv:latest") == ("uv", "latest", False)
    assert parse_image("python:3.14-slim@sha256:abc") == ("python", "3.14-slim", True)

# ---- the examples used in the post ---------------------------------------------------------------

def test_the_before_dockerfile_trips_every_size_rule():
    found = check((HERE / "Dockerfile.before").read_text())
    by_rule = {f.rule: f.line for f in found}
    assert by_rule == {
        "heavy-base": 1,
        "apt-lists": 4,
        "apt-recommends": 4,
        "build-tools-in-final-stage": 4,
        "copy-before-install": 5,
        "pip-cache": 5,
        "cleanup-in-own-layer": 6,
    }

def test_the_after_and_go_dockerfiles_are_clean():
    assert check((HERE / "Dockerfile.after").read_text()) == []
    assert check((HERE / "Dockerfile.go").read_text()) == []

def test_a_missing_dockerignore_is_flagged():
    assert rules((HERE / "Dockerfile.after").read_text(), dockerignore=False) == ["no-dockerignore"]

# ---- only what ships counts ------------------------------------------------------------------------

BUILDER_MESS = """FROM python:3.14 AS builder
RUN apt-get update &amp;&amp; apt-get install -y build-essential libpq-dev
RUN pip install -r requirements.txt
RUN apt-get clean
FROM python:3.14-slim
COPY --from=builder /opt/venv /opt/venv
"""

def test_a_messy_builder_stage_is_fine_because_it_does_not_ship():
    assert rules(BUILDER_MESS) == []

def test_building_on_top_of_a_stage_inherits_its_mess():
    text = BUILDER_MESS.replace("FROM python:3.14-slim\n", "FROM builder\n")
    assert rules(text) == ["apt-lists", "apt-recommends", "build-tools-in-final-stage", "cleanup-in-own-layer", "heavy-base", "pip-cache"]

# ---- individual rules ------------------------------------------------------------------------------------

def test_unpinned_tags():
    assert rules("FROM python:3.14-slim\n") == []
    assert rules("FROM python\n") == ["heavy-base", "unpinned-tag"]
    assert rules("FROM python:latest\n") == ["heavy-base", "unpinned-tag"]
    assert rules("FROM python:3.14-slim@sha256:abc\n") == []
    assert rules("FROM debian@sha256:abc\n") == []  # a digest pins it, even without a tag
    assert rules("FROM debian:trixie-slim AS a\nFROM scratch\nFROM a\n") == []  # scratch and stage names are fine
    assert rules("FROM debian AS a\nFROM a\n") == ["unpinned-tag"]

def test_which_base_images_count_as_heavy():
    assert rules("FROM golang:1.25\n") == ["heavy-base"]
    assert rules("FROM golang:1.25-alpine\n") == []
    assert rules("FROM node:24-bookworm-slim\n") == []
    assert rules("FROM gcr.io/distroless/static-debian13:nonroot\n") == []
    assert rules("FROM golang:1.25 AS builder\nFROM gcr.io/distroless/static-debian13:nonroot\n") == []  # a fat builder is the point

def test_copying_from_another_stage_is_not_copying_the_whole_project():
    text = "FROM node:24-slim AS b\nFROM node:24-slim\nCOPY --from=b . .\nRUN npm ci\n"
    assert rules(text) == []

def test_plain_apt_is_checked_like_apt_get():
    assert rules("FROM debian:trixie-slim\nRUN apt update &amp;&amp; apt install -y curl\n") == ["apt-lists", "apt-recommends"]

def test_uv_pip_is_not_told_to_use_a_pip_flag():
    assert rules("FROM python:3.14-slim\nRUN uv pip install --system fastapi\n") == []

def test_a_build_argument_in_from_is_left_alone():
    assert rules("ARG BASE=python:3.14-slim\nFROM ${BASE}\n") == []
    assert rules("ARG PY=3.14\nFROM python:${PY}\n") == []  # can't tell whether that tag is slim

def test_apt_done_properly_is_clean():
    ok = ("FROM debian:trixie-slim\nRUN apt-get update &amp;&amp; apt-get install -y --no-install-recommends curl "
          "&amp;&amp; rm -rf /var/lib/apt/lists/*\n")
    assert rules(ok) == []
    by_option = ok.replace("--no-install-recommends", "-o APT::Install-Recommends=false")
    assert rules(by_option) == []

def test_dev_packages_and_compilers_are_flagged_in_the_shipped_image():
    text = "FROM debian:trixie-slim\nRUN apt-get install -y --no-install-recommends gcc libpq-dev curl &amp;&amp; rm -rf /var/lib/apt/lists/*\n"
    found = check(text)
    assert [f.rule for f in found] == ["build-tools-in-final-stage"]
    assert "gcc, libpq-dev" in found[0].message and "curl" not in found[0].message

def test_three_ways_to_keep_pip_quiet():
    base = "FROM python:3.14-slim\n"
    assert rules(base + "RUN pip install flask\n") == ["pip-cache"]
    assert rules(base + "RUN pip install --no-cache-dir flask\n") == []
    assert rules(base + "RUN --mount=type=cache,target=/root/.cache/pip pip install flask\n") == []
    assert rules(base + "ENV PIP_NO_CACHE_DIR=1\nRUN pip install flask\n") == []

def test_cleanup_only_helps_in_the_same_layer():
    base = "FROM debian:trixie-slim\n"
    assert rules(base + "RUN make\nRUN rm -rf /tmp/build\n") == ["cleanup-in-own-layer"]
    assert rules(base + "RUN make\nRUN make install &amp;&amp; rm -rf /tmp/build\n") == []

def test_copying_everything_before_installing_dependencies():
    base = "FROM node:24-slim\n"
    assert rules(base + "COPY . .\nRUN npm ci\n") == ["copy-before-install"]
    assert rules(base + "COPY package*.json ./\nRUN npm ci\nCOPY . .\n") == []

# ---- the command line ------------------------------------------------------------------------------------------

def run_cli(dockerfile_text, dockerignore):
    folder = Path(tempfile.mkdtemp())
    (folder / "Dockerfile").write_text(dockerfile_text)
    if dockerignore:
        (folder / ".dockerignore").write_text(".git\n")
    out = io.StringIO()
    with contextlib.redirect_stdout(out):
        code = main(["dockerfile_check.py", str(folder / "Dockerfile")])
    return code, out.getvalue()

def test_cli_exit_codes_and_output():
    code, output = run_cli((HERE / "Dockerfile.before").read_text(), dockerignore=True)
    assert code == 1 and "heavy-base" in output and output.strip().endswith("7 thing(s) to fix")
    code, output = run_cli((HERE / "Dockerfile.after").read_text(), dockerignore=True)
    assert code == 0 and output.strip() == "Nothing found"
</code></pre>
<p>Save the three Dockerfiles from this post in an <code>examples/</code> folder next to the tests, with these names: <code>Dockerfile.before</code>, <code>Dockerfile.after</code> and <code>Dockerfile.go</code>. Then run <code>pytest -q</code>.</p>
<h2>Checklist</h2>
<ul>
<li><p><strong>Measure first.</strong> Look at compressed and unpacked size, and <code>docker history</code> for the heavy layers.</p>
</li>
<li><p><strong>Use a slim base</strong> for the stage that ships.</p>
</li>
<li><p><strong>Build in one stage, ship from another.</strong></p>
</li>
<li><p><strong>Clean up in the same</strong> <code>RUN</code> that creates the mess.</p>
</li>
<li><p><strong>Add a</strong> <code>.dockerignore</code><strong>.</strong></p>
</li>
<li><p><strong>Copy dependency files before code.</strong></p>
</li>
<li><p><strong>Leave dev dependencies and build tools out of the final image.</strong></p>
</li>
<li><p><strong>Lint the Dockerfile in CI,</strong> and rebuild images regularly.</p>
</li>
</ul>
<hr />
<p><em>Published via <a href="https://zyvop.com/your-docker-image-is-1-2-gb-here-s-how-to-get-it-under-100-mb-9oiu3?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Build a Private Document Organizer in Python: Offline OCR and a Rules File]]></title><description><![CDATA[Cloud OCR means uploading your ID cards and bills to someone else's server. This post builds a local alternative in Python: offline OCR with Tesseract…]]></description><link>https://blog.zyvop.com/build-a-private-document-organizer-in-python-offline-ocr-and-a-rules-file-2zrrq</link><guid isPermaLink="true">https://blog.zyvop.com/build-a-private-document-organizer-in-python-offline-ocr-and-a-rules-file-2zrrq</guid><category><![CDATA[automation]]></category><category><![CDATA[OCR ]]></category><category><![CDATA[privacy]]></category><category><![CDATA[Python]]></category><category><![CDATA[tesseract]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Fri, 06 Nov 2026 03:04:01 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/9ae3ead9-a50a-4ae2-82d2-516cbf5b4c4d.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Your Downloads folder holds <code>scan0007.pdf</code>, <code>IMG_2041.jpg</code> and <code>Download (3).pdf</code>. One is an electricity bill, one is an ID card, and one is a trip itinerary. Finding any of them later means opening each file.</p>
<p>Online OCR and document services can sort this for you, but they do it by uploading your bills, bank statements and ID cards to someone else's server. For documents like these, keeping everything on your own machine is the safer default.</p>
<p>This post builds a small tool that does it locally: it reads each scanned document with offline OCR, matches it against a rules file you write, then renames and files it. It has a dry run, duplicate detection and undo, because a tool that moves your documents should be easy to trust. The complete code is at the end.</p>
<h2>What we're building</h2>
<p>You drop files into an <code>Inbox</code> folder. The tool reads the text of each one, finds the first rule that matches, and moves the file into a <code>Library</code> folder with a clean name:</p>
<pre><code class="language-text">scan0007.pdf  -&gt;  Bills/Electricity/2026/2026-11-03_state-electricity-board-bill.pdf
</code></pre>
<p>The design rests on five decisions, and they are the interesting part:</p>
<ol>
<li><p>Text extraction is a pipeline with fallbacks, not one OCR call.</p>
</li>
<li><p>Rules are data in a TOML file, not code.</p>
</li>
<li><p>The first matching rule wins, so order is priority.</p>
</li>
<li><p>Sensitive documents are classified, never copied into a filename.</p>
</li>
<li><p>Nothing is moved until you ask, and everything can be undone.</p>
</li>
</ol>
<h2>How it works</h2>
<p><img src="https://mermaid.ink/img/pako:eNpdklFv2jAQx7_K7Z4dTZP2xMOmAkKtNKADOnWKq-qIL-DNsSP7aMkQ331KSCnl8ezf_f3z2QcsgmEcYOnCa7GlKLAaaw8AcJNrnFjHYD3c-XXYa3yCLPsGw0O_IU3N3zUeT_wQsgw0yl4gRKiMxo5e5RoXTAZkyyC8F41PHxpsRRvu4XmuccUpcaRCYD5aXMP340mP3h803lIC6kLBUcPxXeb-hP_m1OOPucbalBI-KPTYLCiwArZNSwX5vmfx1lPXlWr9PZzlzhGrDp3mGqckxRbiznFSUNqYBKpu6dX6dObnJ_5UPF4Wi8ti2hWzg8abLvIUxeb9irPrK45zjcOddQbK4AxHIG_AU8VQxlD1869qR8LvOrO3CfQhD7nGH3YdKTafnx98ClHYnOlxx4wOGpdtbhG8sBcgF5lMA6V1l4aja8Ofl-HjXe1s8UFmdCUzaacaXrhzb8NV-7X-7JJAHa2X7tE8EJjYQNx5jU-osOJYkTU4OKBsuWp_t-GSdk5QnVZ-UbS0dpxapgxeJlRZ1-AAM6prx1lqknClYOis_zulYtnVk-BFgcYlbwLDw51GBYuwDhIU3LJ7YbEFKbiJlpyCRD5liaMtUXWHLO2_1uXL13qPx6PC9WYUXIg4wE-vWyuMx_8-cxWo?type=png" alt="Mermaid Diagram" /></p>
<h2>Set up the project</h2>
<p>Create a folder and a virtual environment:</p>
<pre><code class="language-bash">mkdir doc-organizer &amp;&amp; cd doc-organizer
python3 -m venv .venv
source .venv/bin/activate        # Windows PowerShell: .venv\Scripts\Activate.ps1
python --version                 # 3.11 or newer
</code></pre>
<p>The script itself uses only the standard library. Python 3.11 added <code>tomllib</code>, which reads the rules file. On an older Python, run <code>pip install tomli</code> and the script picks it up automatically.</p>
<p>The OCR is done by two command-line tools. <a href="https://github.com/tesseract-ocr/tesseract">Tesseract</a> is an open-source OCR engine that runs fully offline, and poppler provides <code>pdftotext</code> and <code>pdftoppm</code>:</p>
<pre><code class="language-bash"># macOS
brew install tesseract poppler

# Debian / Ubuntu
sudo apt install tesseract-ocr poppler-utils
</code></pre>
<p>On Windows, the easiest route is WSL with the Ubuntu commands above. Native installs work too: install Tesseract and poppler, then add both to your <code>PATH</code>.</p>
<p>For Hindi documents, install the language data (<code>sudo apt install tesseract-ocr-hin</code>, or <code>brew install tesseract-lang</code> on macOS), check it with <code>tesseract --list-langs</code>, and set <code>lang = "eng+hin"</code> in the rules file.</p>
<p>Your folder will look like this:</p>
<pre><code class="language-text">doc-organizer/
  .venv/
  docsort.py
  rules.toml
</code></pre>
<h2>Decision 1: extraction is a pipeline</h2>
<p>Documents arrive in three shapes: plain text, images (phone photos), and PDFs. PDFs come in two kinds, and telling them apart is the useful trick. A PDF exported from a bank's website has a text layer you can read instantly with <code>pdftotext</code>. A scanned PDF is just pictures of pages, so it needs OCR.</p>
<p>The function tries the cheap path first and falls back to OCR only when it has to:</p>
<pre><code class="language-python">def extract_text(path, lang="eng"):
    """Return the document's text, or None for an unsupported file type."""
    ext = path.suffix.lower()
    if ext in TEXT_EXT:
        return path.read_text(errors="ignore")
    if ext in IMAGE_EXT:
        return ocr_image(path, lang)
    if ext == ".pdf":
        text = ""
        if shutil.which("pdftotext"):
            text = run(["pdftotext", "-l", str(MAX_PAGES), "-layout", str(path), "-"])
        if len(text.strip()) &gt;= MIN_TEXT:
            return text  # the PDF already has a text layer: no OCR needed
        with tempfile.TemporaryDirectory() as tmp:
            run(["pdftoppm", "-r", str(OCR_DPI), "-l", str(MAX_PAGES), "-png", str(path), f"{tmp}/page"])
            return "\n".join(ocr_image(p, lang) for p in sorted(Path(tmp).glob("page*.png")))
    return None
</code></pre>
<p>Two details keep it fast. <code>MIN_TEXT</code> is the threshold for "this PDF has no real text layer", and <code>MAX_PAGES = 3</code> limits OCR to the first pages, because the header of a document is what identifies it. Reading 40 pages of a statement would only make you wait.</p>
<h2>Decision 2 and 3: rules are data, and the first match wins</h2>
<p>Here is the rules file:</p>
<pre><code class="language-toml"># docsort rules. The FIRST matching rule wins, so put specific rules above general ones.
inbox = "~/Documents/Inbox"
library = "~/Documents/Library"
lang = "eng"                 # "eng+hin" reads English and Hindi (needs the Hindi language data)

[[rule]]
name = "Electricity bill"
any = ["electricity", "units consumed"]     # at least one of these
none = ["gas bill"]                         # and none of these
folder = "Bills/Electricity/{year}"
name_template = "{date}_{provider}-bill"
[rule.fields]
provider = "^(.{3,40}?) Consumer No"        # first capture group becomes {provider}

[[rule]]
name = "Bank statement"
all = ["statement of account"]              # every one of these...
any = ["savings account", "account number"] # ...plus at least one of these
folder = "Finance/Statements/{year}"
name_template = "{date}_bank-statement"

[[rule]]
name = "Aadhaar card"
any = ["unique identification authority of india"]
sensitive = true                            # classify only: no text is copied into the filename
folder = "Identity"
name_template = "aadhaar-card"

[[rule]]
name = "PAN card"
all = ["income tax department", "permanent account number"]
sensitive = true
folder = "Identity"
name_template = "pan-card"
</code></pre>
<p>A rule matches when:</p>
<ul>
<li><p>at least one word from <code>any</code> is present (if you gave an <code>any</code> list),</p>
</li>
<li><p>every word from <code>all</code> is present, and</p>
</li>
<li><p>no word from <code>none</code> is present.</p>
</li>
</ul>
<pre><code class="language-python">def check(rule, text_l):
    """A rule matches if any 'any' word is present, all 'all' words are, and no 'none' word is."""
    any_hits, all_hits = hits(text_l, rule.get("any", [])), hits(text_l, rule.get("all", []))
    blocked = hits(text_l, rule.get("none", []))
    ok = (bool(any_hits) or not rule.get("any")) and len(all_hits) == len(rule.get("all", [])) and not blocked
    return ok, any_hits + all_hits, blocked

def classify(text, rules):
    """First matching rule wins, so the order of rules in the file is the priority."""
    text_l = normalise(text).lower()
    results = [(r, *check(r, text_l)) for r in rules]
    first = next((r for r, ok, _, _ in results if ok), None)
    return first, results
</code></pre>
<p>Matching is case-insensitive, and the text is first collapsed so that all runs of whitespace and line breaks become single spaces. OCR often breaks a phrase across lines or adds extra spaces, and this step means <code>unique identification authority of india</code> still matches when the scan reads "Unique Identification" on one line and "Authority of India" on the next.</p>
<p>Why first match wins instead of a scoring system? Because you can predict it. When a document lands in the wrong folder you read the file top to bottom and see why. Put specific rules above general ones, and that's the whole model.</p>
<p>Choose keywords that survive OCR errors: short, distinctive phrases rather than whole sentences. A longer phrase has more chances for one misread character to break the match. The <code>explain</code> command, shown below, tells you exactly which words hit.</p>
<h2>Naming files: dates and fields</h2>
<p>A good filename starts with a date so that a folder sorts chronologically. The tool takes the earliest plausible date in the text:</p>
<pre><code class="language-python">def find_date(text):
    """Earliest plausible date in the text. Numeric dates are read day-first (Indian style)."""
    found = []
    for m in re.finditer(r"\b(\d{4})-(\d{2})-(\d{2})\b", text):
        found.append((m.start(), m.group(1), m.group(2), m.group(3)))
    for m in re.finditer(r"\b(\d{1,2})[/.-](\d{1,2})[/.-](\d{4})\b", text):
        found.append((m.start(), m.group(3), m.group(2), m.group(1)))
    for m in re.finditer(r"\b(\d{1,2})[ -]([A-Za-z]{3})[a-z]*[ ,.-]+(\d{4})\b", text):
        found.append((m.start(), m.group(3), MONTHS.get(m.group(2).lower()), m.group(1)))
    valid = []
    for pos, y, mo, d in found:
        try:
            day = datetime(int(y), int(mo), int(d)).date()
        except (TypeError, ValueError):
            continue
        if 2000 &lt;= day.year &lt;= 2100:
            valid.append((pos, day))
    return min(valid)[1] if valid else None
</code></pre>
<p>Numeric dates are read day first (<code>03/11/2026</code> is 3 November), which is the Indian convention. If you work with US-format documents, that assumption is the line to change.</p>
<p>"Earliest date in the text" is a heuristic. For a bill it is usually the bill date, and for a statement it is the start of the period. If a document needs an exact field, capture it with a regex in the rule's <code>[rule.fields]</code> table. If no date is found, the file's modified time is used.</p>
<p>Templates then build the folder and filename. <code>{date}</code>, <code>{year}</code>, <code>{month}</code> and <code>{original}</code> (the old filename) are always available, and every key in <code>[rule.fields]</code> becomes a placeholder too. The first capture group of its regex is turned into a safe lowercase slug. A missing field becomes <code>unknown</code> instead of crashing the run.</p>
<h2>Decision 4: classify sensitive documents, don't extract from them</h2>
<p>Look at the two identity rules in the file: they have <code>sensitive = true</code> and templates with no fields. This is deliberate.</p>
<p>For an ID card, the tool needs to know only what kind of document it is. It has no reason to read the number off the card and put it in a filename.</p>
<p>Filenames leak far more easily than file contents: they show up in cloud sync logs, backup listings, search indexes, shell history and screenshots. A name like <code>pan-card.jpg</code> is useful and harmless; a name with the number in it is a liability.</p>
<pre><code class="language-python">def values_for(rule, text, src):
    day = None if rule.get("sensitive") else find_date(text)  # an ID card's earliest date is usually a birth date
    from_file = day is None
    if from_file:
        day = datetime.fromtimestamp(src.stat().st_mtime).date()
    vals = Safe(date=day.isoformat(), year=str(day.year), month=f"{day.month:02d}", original=slug(src.stem))
    if not rule.get("sensitive"):  # sensitive documents are classified, never copied into a filename
        flat = normalise(text)
        for key, pattern in rule.get("fields", {}).items():
            m = re.search(pattern, flat)
            vals[key] = slug(m.group(1)) if m and m.groups() else "unknown"
    return vals, from_file
</code></pre>
<p>The rest of the tool follows the same rule:</p>
<ul>
<li><p><code>explain</code> prints your own rule keywords and a character count, never the document's text. It shows the date it found only for non-sensitive documents.</p>
</li>
<li><p>Dates for sensitive rules come from the file's modified time rather than the text, because the earliest date on an ID card is usually a date of birth.</p>
</li>
<li><p>The log stores file paths, not contents.</p>
</li>
</ul>
<p>Two honest limits. While a scanned PDF is being OCR'd, its page images sit briefly in a temporary folder that is deleted afterwards. And "local" does not mean "encrypted": turn on full-disk encryption (FileVault, BitLocker or LUKS) for the machine that holds these files.</p>
<h2>Decision 5: dry run, collisions, duplicates and undo</h2>
<p>By default, <code>scan</code> only prints a plan. Nothing moves until you add <code>--apply</code>. A tool that renames your documents should earn trust before it gets write access.</p>
<p>Three situations need handling when you do apply:</p>
<ul>
<li><p><strong>Name collision.</strong> Two different documents map to the same name, for example two bills on the same date. The second becomes <code>...-2.pdf</code>. The tool tracks the names it has planned in the current run too, so two files in one batch can't claim the same destination.</p>
</li>
<li><p><strong>Duplicate.</strong> If the target already holds a file with the same SHA-256 hash, the new file isn't filed again. It goes to <code>_Duplicates</code>, where you can check and delete it.</p>
</li>
<li><p><strong>Undo.</strong> Every move is appended to <code>.docsort-log.jsonl</code> in the library, tagged with a run id. <code>undo</code> reverses the most recent run only. The run id includes microseconds: with one-second resolution, two quick runs could share an id and one undo would reverse both.</p>
</li>
</ul>
<p>Rules can also go wrong. A <code>folder</code> template that resolves outside the library (for example one containing <code>..</code>) is rejected, and that file is skipped with a message.</p>
<h2>Using it</h2>
<p>Put some files in the inbox and look at the plan. The sample documents below were generated with placeholder values (no real IDs): a photo, a scan, a text PDF and a duplicate.</p>
<pre><code class="language-bash">python docsort.py scan
</code></pre>
<pre><code class="language-text">Download (3).pdf
   -&gt; _Unsorted/Download (3).pdf   [no rule matched]
IMG_2041.jpg
   -&gt; Identity/pan-card.jpg   [PAN card]
IMG_2057.png
   -&gt; Identity/aadhaar-card.png   [Aadhaar card]
Statement_Oct.pdf
   -&gt; Finance/Statements/2026/2026-10-01_bank-statement.pdf   [Bank statement]
scan0007 (1).pdf
   -&gt; Bills/Electricity/2026/2026-11-03_state-electricity-board-bill.pdf   [Electricity bill]
scan0007.pdf
   -&gt; _Duplicates/scan0007.pdf   [Electricity bill]  duplicate of scan0007 (1).pdf in this batch

Dry run: 6 file(s) would move. Nothing was changed. Re-run with --apply.
</code></pre>
<p>If the plan looks right, apply it:</p>
<pre><code class="language-bash">python docsort.py scan --apply
</code></pre>
<p>The library now looks like this:</p>
<pre><code class="language-text">Bills/Electricity/2026/2026-11-03_state-electricity-board-bill.pdf
Finance/Statements/2026/2026-10-01_bank-statement.pdf
Identity/aadhaar-card.png
Identity/pan-card.jpg
_Duplicates/scan0007.pdf
_Unsorted/Download (3).pdf
</code></pre>
<p>Notice that both identity documents were filed without any number in the name, the duplicate scan went to <code>_Duplicates</code>, and the itinerary that matched no rule is waiting in <code>_Unsorted</code>. That folder is your to-do list for new rules.</p>
<p>When a file lands in the wrong place, ask why:</p>
<pre><code class="language-bash">python docsort.py explain scan0007.pdf
</code></pre>
<pre><code class="language-text">105 characters of text. First date found: 2026-11-03
  MATCH Electricity bill   hit: ['electricity', 'units consumed']
  no    Bank statement
  no    Aadhaar card
  no    PAN card

Winner: Electricity bill
</code></pre>
<p>And if you don't like what happened, undo the last run:</p>
<pre><code class="language-bash">python docsort.py undo
</code></pre>
<pre><code class="language-text">restored scan0007.pdf
restored scan0007 (1).pdf
restored Statement_Oct.pdf
restored IMG_2057.png
restored IMG_2041.jpg
restored Download (3).pdf
</code></pre>
<p>A password-protected PDF can't be read, so it's skipped with the reason instead of crashing the run:</p>
<pre><code class="language-text">SKIP  encrypted.pdf: pdftotext failed: Command Line Error: Incorrect password
</code></pre>
<h2>Limits worth knowing</h2>
<ul>
<li><p>OCR is not perfect. Blurry photos, handwriting and unusual fonts will fail to match, and those files end up in <code>_Unsorted</code>.</p>
</li>
<li><p>Rules match on keywords, so a document that mentions "electricity" in passing could match the wrong rule. Use <code>none</code> and <code>all</code> to tighten it, and keep specific rules first.</p>
</li>
<li><p>Only the first three pages are read.</p>
</li>
<li><p>Numeric dates are day first.</p>
</li>
</ul>
<h2>The complete code</h2>
<p>Save these two files in your project folder. <code>docsort.py</code> is about 300 lines.</p>
<h3>docsort.py</h3>
<pre><code class="language-python">#!/usr/bin/env python3
"""
docsort - file scanned documents using offline OCR and a TOML rules file.

    python docsort.py scan                 # dry run: show what would happen
    python docsort.py scan --apply         # actually move the files
    python docsort.py undo                 # reverse the last --apply run
    python docsort.py explain some.pdf     # show why a file matched (or didn't)

Needs Python 3.11+ (or `pip install tomli`). Scanned images and PDFs also need the
`tesseract` and `pdftoppm`/`pdftotext` (poppler) command-line tools. Nothing is uploaded.
"""
import argparse
import hashlib
import json
import re
import shutil
import subprocess
import sys
import tempfile
from datetime import datetime
from pathlib import Path

try:
    import tomllib
except ImportError:  # Python &lt; 3.11: pip install tomli
    import tomli as tomllib

IMAGE_EXT = {".png", ".jpg", ".jpeg", ".tif", ".tiff", ".bmp"}
TEXT_EXT = {".txt", ".md"}
OCR_DPI = 300      # higher is more accurate and slower
MAX_PAGES = 3      # the first pages identify a document; no need to read 40 of them
MIN_TEXT = 40      # a PDF yielding fewer characters than this is a scan and needs OCR
LOG_NAME = ".docsort-log.jsonl"
MONTHS = {m: i for i, m in enumerate("jan feb mar apr may jun jul aug sep oct nov dec".split(), 1)}

class ToolMissing(RuntimeError):
    pass

# ------------------------------------------------------------ text extraction
def run(cmd):
    if not shutil.which(cmd[0]):
        raise ToolMissing(f"'{cmd[0]}' is not installed or not on PATH (see the setup notes).")
    done = subprocess.run(cmd, capture_output=True, text=True)
    if done.returncode:
        raise OSError(f"{cmd[0]} failed: {done.stderr.strip() or 'unknown error'}")
    return done.stdout

def ocr_image(path, lang):
    return run(["tesseract", str(path), "stdout", "-l", lang])

def extract_text(path, lang="eng"):
    """Return the document's text, or None for an unsupported file type."""
    ext = path.suffix.lower()
    if ext in TEXT_EXT:
        return path.read_text(errors="ignore")
    if ext in IMAGE_EXT:
        return ocr_image(path, lang)
    if ext == ".pdf":
        text = ""
        if shutil.which("pdftotext"):
            text = run(["pdftotext", "-l", str(MAX_PAGES), "-layout", str(path), "-"])
        if len(text.strip()) &gt;= MIN_TEXT:
            return text  # the PDF already has a text layer: no OCR needed
        with tempfile.TemporaryDirectory() as tmp:
            run(["pdftoppm", "-r", str(OCR_DPI), "-l", str(MAX_PAGES), "-png", str(path), f"{tmp}/page"])
            return "\n".join(ocr_image(p, lang) for p in sorted(Path(tmp).glob("page*.png")))
    return None

# ------------------------------------------------------------ dates and rules
def find_date(text):
    """Earliest plausible date in the text. Numeric dates are read day-first (Indian style)."""
    found = []
    for m in re.finditer(r"\b(\d{4})-(\d{2})-(\d{2})\b", text):
        found.append((m.start(), m.group(1), m.group(2), m.group(3)))
    for m in re.finditer(r"\b(\d{1,2})[/.-](\d{1,2})[/.-](\d{4})\b", text):
        found.append((m.start(), m.group(3), m.group(2), m.group(1)))
    for m in re.finditer(r"\b(\d{1,2})[ -]([A-Za-z]{3})[a-z]*[ ,.-]+(\d{4})\b", text):
        found.append((m.start(), m.group(3), MONTHS.get(m.group(2).lower()), m.group(1)))
    valid = []
    for pos, y, mo, d in found:
        try:
            day = datetime(int(y), int(mo), int(d)).date()
        except (TypeError, ValueError):
            continue
        if 2000 &lt;= day.year &lt;= 2100:
            valid.append((pos, day))
    return min(valid)[1] if valid else None

def normalise(text):
    return re.sub(r"\s+", " ", text).strip()

def hits(text_l, words):
    return [w for w in words if w.lower() in text_l]

def check(rule, text_l):
    """A rule matches if any 'any' word is present, all 'all' words are, and no 'none' word is."""
    any_hits, all_hits = hits(text_l, rule.get("any", [])), hits(text_l, rule.get("all", []))
    blocked = hits(text_l, rule.get("none", []))
    ok = (bool(any_hits) or not rule.get("any")) and len(all_hits) == len(rule.get("all", [])) and not blocked
    return ok, any_hits + all_hits, blocked

def classify(text, rules):
    """First matching rule wins, so the order of rules in the file is the priority."""
    text_l = normalise(text).lower()
    results = [(r, *check(r, text_l)) for r in rules]
    first = next((r for r, ok, _, _ in results if ok), None)
    return first, results

def load_config(path):
    path = Path(path).expanduser()
    if not path.exists():
        sys.exit(f"Config file not found: {path}")
    cfg = tomllib.loads(path.read_text(encoding="utf-8"))
    for key in ("inbox", "library"):
        if key not in cfg:
            sys.exit(f"{path}: missing '{key}'")
    rules = cfg.get("rule", [])
    for r in rules:
        if not (r.get("name") and r.get("folder") and (r.get("any") or r.get("all"))):
            sys.exit(f"Rule {r.get('name', '?')!r} needs name, folder and at least one of any / all.")
    return {"inbox": Path(cfg["inbox"]).expanduser(), "library": Path(cfg["library"]).expanduser(),
            "lang": cfg.get("lang", "eng"), "rules": rules}

# ------------------------------------------------------------ naming and planning
class Safe(dict):
    def __missing__(self, key):
        return "unknown"

def slug(s, limit=40):
    return re.sub(r"[^a-z0-9]+", "-", s.lower()).strip("-")[:limit] or "unknown"

def values_for(rule, text, src):
    day = None if rule.get("sensitive") else find_date(text)  # an ID card's earliest date is usually a birth date
    from_file = day is None
    if from_file:
        day = datetime.fromtimestamp(src.stat().st_mtime).date()
    vals = Safe(date=day.isoformat(), year=str(day.year), month=f"{day.month:02d}", original=slug(src.stem))
    if not rule.get("sensitive"):  # sensitive documents are classified, never copied into a filename
        flat = normalise(text)
        for key, pattern in rule.get("fields", {}).items():
            m = re.search(pattern, flat)
            vals[key] = slug(m.group(1)) if m and m.groups() else "unknown"
    return vals, from_file

def sha256(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 &lt;&lt; 20), b""):
            h.update(chunk)
    return h.hexdigest()

def unique(dest, taken):
    cand, n = dest, 2
    while cand.exists() or cand in taken:
        cand = dest.with_name(f"{dest.stem}-{n}{dest.suffix}")
        n += 1
    return cand

def plan_one(src, cfg, taken):
    """Return (destination, rule name, note), or None for an unsupported file type."""
    text = extract_text(src, cfg["lang"])
    if text is None:
        return None
    lib = cfg["library"]
    rule, _ = classify(text, cfg["rules"])
    if rule is None:
        dest, name, note = lib / "_Unsorted" / src.name, "no rule matched", ""
    else:
        vals, from_file = values_for(rule, text, src)
        template = rule.get("name_template", "{date}_{original}")
        stem = template.format_map(vals)
        dest = lib / rule["folder"].format_map(vals) / (stem + src.suffix.lower())
        try:
            dest.resolve().relative_to(lib.resolve())
        except ValueError:
            raise ValueError(f"rule {rule['name']!r} would write outside the library; "
                             "check its folder and name_template") from None
        uses_date = any(k in template + rule["folder"] for k in ("{date}", "{year}", "{month}"))
        name, note = rule["name"], "date taken from file time" if from_file and uses_date else ""
        existing = dest if dest.exists() else taken.get(dest)
        if existing and sha256(existing) == sha256(src):
            where = f"the library file {dest.name}" if existing == dest else f"{existing.name} in this batch"
            dest, note = lib / "_Duplicates" / src.name, f"duplicate of {where}"
    dest = unique(dest, taken)
    taken[dest] = src
    return dest, name, note

# ------------------------------------------------------------ commands
def cmd_scan(args, cfg):
    files = sorted(p for p in cfg["inbox"].rglob("*") if p.is_file() and not p.name.startswith("."))
    if not files:
        print("Inbox is empty.")
        return 0
    taken, moves = {}, []
    for src in files:
        try:
            result = plan_one(src, cfg, taken)
        except (ValueError, OSError) as err:
            print(f"SKIP  {src.name}: {err}")
            continue
        if result is None:
            print(f"SKIP  {src.name}: unsupported file type")
            continue
        dest, rule, note = result
        moves.append((src, dest))
        print(f"{src.name}\n   -&gt; {dest.relative_to(cfg['library'])}   [{rule}]" + (f"  {note}" if note else ""))
    if not args.apply:
        print(f"\nDry run: {len(moves)} file(s) would move. Nothing was changed. Re-run with --apply.")
        return 0
    run_id = datetime.now().strftime("%Y%m%dT%H%M%S%f")  # microseconds: two runs never share an id
    cfg["library"].mkdir(parents=True, exist_ok=True)
    with open(cfg["library"] / LOG_NAME, "a", encoding="utf-8") as log:
        for src, dest in moves:
            dest.parent.mkdir(parents=True, exist_ok=True)
            shutil.move(str(src), str(dest))
            log.write(json.dumps({"run": run_id, "from": str(src), "to": str(dest)}) + "\n")
            log.flush()
    print(f"\nMoved {len(moves)} file(s). Undo with: python docsort.py undo")
    return 0

def cmd_undo(args, cfg):
    log = cfg["library"] / LOG_NAME
    entries = [json.loads(line) for line in log.read_text(encoding="utf-8").splitlines() if line.strip()] \
        if log.exists() else []
    if not entries:
        print("Nothing to undo.")
        return 0
    last = entries[-1]["run"]
    for e in reversed([e for e in entries if e["run"] == last]):
        src, dest = Path(e["from"]), Path(e["to"])
        if dest.exists() and not src.exists():
            src.parent.mkdir(parents=True, exist_ok=True)
            shutil.move(str(dest), str(src))
            print(f"restored {src.name}")
        else:
            print(f"left alone (missing, or the original spot is taken): {dest}")
    keep = [e for e in entries if e["run"] != last]
    log.write_text("".join(json.dumps(e) + "\n" for e in keep), encoding="utf-8")
    return 0

def cmd_explain(args, cfg):
    src = Path(args.file).expanduser()
    text = extract_text(src, cfg["lang"])
    if text is None:
        sys.exit("Unsupported file type.")
    first, results = classify(text, cfg["rules"])
    shown = find_date(text) or "none" if first and not first.get("sensitive") else "not shown"
    print(f"{len(text.strip())} characters of text. First date found: {shown}")
    for rule, ok, found, blocked in results:  # prints rule keywords only, never the document's text
        line = f"  {'MATCH' if ok else 'no':&lt;6}{rule['name']}"
        line += f"   hit: {found}" if found else ""
        line += f"   blocked by: {blocked}" if blocked else ""
        print(line)
    print(f"\nWinner: {first['name'] if first else 'none (the file would go to _Unsorted)'}")
    return 0

def main():
    common = argparse.ArgumentParser(add_help=False)
    common.add_argument("--config", default="rules.toml", help="rules file (default: ./rules.toml)")
    p = argparse.ArgumentParser(prog="docsort", description=__doc__,
                                formatter_class=argparse.RawDescriptionHelpFormatter)
    sub = p.add_subparsers(dest="cmd", required=True)
    s = sub.add_parser("scan", parents=[common], help="plan (or perform) moves for everything in the inbox")
    s.add_argument("--apply", action="store_true", help="really move the files")
    s.set_defaults(fn=cmd_scan)
    sub.add_parser("undo", parents=[common], help="reverse the last --apply run").set_defaults(fn=cmd_undo)
    e = sub.add_parser("explain", parents=[common], help="show which rules match one file")
    e.add_argument("file")
    e.set_defaults(fn=cmd_explain)
    args = p.parse_args()
    try:
        return args.fn(args, load_config(args.config))
    except ToolMissing as err:
        print(f"Error: {err}", file=sys.stderr)
        return 1

if __name__ == "__main__":
    sys.exit(main())
</code></pre>
<h3>rules.toml</h3>
<p>Change <code>inbox</code> and <code>library</code> to your own folders. Keep the rules order in mind: specific first.</p>
<pre><code class="language-toml"># docsort rules. The FIRST matching rule wins, so put specific rules above general ones.
inbox = "~/Documents/Inbox"
library = "~/Documents/Library"
lang = "eng"                 # "eng+hin" reads English and Hindi (needs the Hindi language data)

[[rule]]
name = "Electricity bill"
any = ["electricity", "units consumed"]     # at least one of these
none = ["gas bill"]                         # and none of these
folder = "Bills/Electricity/{year}"
name_template = "{date}_{provider}-bill"
[rule.fields]
provider = "^(.{3,40}?) Consumer No"        # first capture group becomes {provider}

[[rule]]
name = "Bank statement"
all = ["statement of account"]              # every one of these...
any = ["savings account", "account number"] # ...plus at least one of these
folder = "Finance/Statements/{year}"
name_template = "{date}_bank-statement"

[[rule]]
name = "Aadhaar card"
any = ["unique identification authority of india"]
sensitive = true                            # classify only: no text is copied into the filename
folder = "Identity"
name_template = "aadhaar-card"

[[rule]]
name = "PAN card"
all = ["income tax department", "permanent account number"]
sensitive = true
folder = "Identity"
name_template = "pan-card"
</code></pre>
<h2>Where to take it next</h2>
<ul>
<li><p>Run <code>scan --apply</code> on a schedule with cron or a systemd timer, so new scans are filed on their own. Check the first runs by hand and use dry runs until you trust your rules.</p>
</li>
<li><p>Write the fields you extract (amounts, provider) into a CSV or SQLite table.</p>
</li>
<li><p>Add full-text search over the OCR text with SQLite's FTS5, so you can find "the bill from March" without remembering where it went.</p>
</li>
<li><p>Add a rule-test mode: a folder of sample documents with the folder each one should end up in, run whenever you edit the rules.</p>
</li>
</ul>
<p>Everything here runs on your own machine, and the only thing the tool ever leaves behind is a log of file paths.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/build-a-private-document-organizer-in-python-offline-ocr-and-a-rules-file-2zrrq?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Node.js Security Checklist for 2026]]></title><description><![CDATA[Many Node.js security incidents start with missing basics. This checklist covers a supported runtime, Helmet headers, rate limiting, Argon2id password…]]></description><link>https://blog.zyvop.com/node-js-security-checklist-for-2026-fym1k</link><guid isPermaLink="true">https://blog.zyvop.com/node-js-security-checklist-for-2026-fym1k</guid><category><![CDATA[Node.js]]></category><category><![CDATA[npm]]></category><category><![CDATA[Security]]></category><category><![CDATA[supplychain]]></category><category><![CDATA[websecurity]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Thu, 05 Nov 2026 11:09:20 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/3da63968-1dd3-4c51-8641-c5f2629640ad.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Many Node.js security incidents don't start with an exotic zero-day. They start with missing basics: an unpatched runtime, no limits on a login endpoint, passwords stored with a fast hash, or a dependency that was compromised a few hours before your CI pulled it.</p>
<p>This checklist starts with your runtime, then walks through the five areas that give you the most protection for the least effort. Each section explains the why, shows the code or config, and ends with items you can tick off. The examples use Express, but the ideas apply to Fastify, Hono, NestJS, and anything else.</p>
<h2>1. Start with a Supported Node.js Version</h2>
<p>Everything else is built on the runtime, so check it first. Node.js 20 reached end of life on April 30, 2026, which means it receives no fixes at all, and end-of-life versions are affected whenever a security release ships.</p>
<p>As of late 2026, the supported lines are:</p>
<ul>
<li><p><strong>Node.js 22:</strong> maintenance LTS, supported until April 30, 2027</p>
</li>
<li><p><strong>Node.js 24:</strong> maintenance LTS since October 2026, supported until April 30, 2028</p>
</li>
<li><p><strong>Node.js 26:</strong> Active LTS since October 28, 2026, supported until April 30, 2029</p>
</li>
</ul>
<p>Odd-numbered releases such as Node.js 25 were short-lived and not meant for production; Node.js 25 reached end of life on June 1, 2026. From Node.js 27 on, the project moves to one major release a year, and every release becomes LTS.</p>
<p>Security releases land regularly across every supported line (October 2026 brought one that covered the 22, 24, and 26 lines), so a pinned version is only as safe as your last rebuild. Rebuild and redeploy your images on a schedule instead of only when you ship features.</p>
<ul>
<li><p>Run a supported LTS line in production: Node.js 26 for new projects, or Node.js 24 until you're ready to move</p>
</li>
<li><p>If you're on Node.js 22, plan your upgrade before April 30, 2027</p>
</li>
<li><p>Pin your Docker base image to a major line (for example <code>node:24-alpine</code>), never <code>latest</code>, so a rebuild can't silently jump a major version</p>
</li>
<li><p>Put the Node.js end-of-life date for your version in your team calendar</p>
</li>
<li><p>Subscribe to the Node.js security announcements and patch promptly when a security release lands</p>
</li>
<li><p>Rebuild and redeploy images regularly so security releases actually reach production</p>
</li>
</ul>
<h2>2. Security Headers with Helmet</h2>
<p>Browsers have many built-in protections, but they only switch on when your server asks for them through HTTP response headers. <strong>Helmet</strong> is middleware that sets a sensible group of those headers in one line.</p>
<pre><code class="language-javascript">import express from "express";
import helmet from "helmet";

const app = express();

app.use(helmet());
</code></pre>
<p>With the defaults, Helmet adds headers such as <code>Content-Security-Policy</code>, <code>Strict-Transport-Security</code>, <code>X-Content-Type-Options</code>, <code>Referrer-Policy</code>, and <code>Cross-Origin-Opener-Policy</code>, and it removes the <code>X-Powered-By</code> header that advertises Express.</p>
<p>The part that needs your attention is the <strong>Content Security Policy (CSP)</strong>, the header that controls which scripts, styles, and images a page may load. The default policy is strict: it only allows scripts and most other resources from your own origin and blocks inline event handlers, so it may break inline scripts or third-party assets. Tune it rather than switching it off:</p>
<pre><code class="language-javascript">app.use(
  helmet({
    contentSecurityPolicy: {
      directives: {
        defaultSrc: ["'self'"],
        scriptSrc: ["'self'"],
        imgSrc: ["'self'", "data:", "https://cdn.example.com"],
        objectSrc: ["'none'"],
      },
    },
    referrerPolicy: { policy: "strict-origin-when-cross-origin" },
  })
);
</code></pre>
<p>A few things to keep in mind:</p>
<ul>
<li><p><strong>Roll out CSP in report-only mode first.</strong> Set <code>reportOnly: true</code> inside <code>contentSecurityPolicy</code>, watch the browser console and your violation reports, then switch to enforcing once nothing legitimate is blocked.</p>
</li>
<li><p><strong>Prefer nonces over</strong> <code>'unsafe-inline'</code><strong>.</strong> If you really need an inline script, generate a per-request nonce and add it to <code>scriptSrc</code>, rather than loosening the policy for every script on the page.</p>
</li>
<li><p><strong>Helmet only sets headers.</strong> It doesn't fix injection bugs, broken authorization, or bad cookies.</p>
</li>
<li><p><strong>Cross-origin assets may break.</strong> Helmet's default <code>Cross-Origin-Resource-Policy</code> is <code>same-origin</code>. If other sites legitimately load your images or files, relax it for those routes.</p>
</li>
<li><p><strong>Mind HSTS.</strong> Helmet sends <code>Strict-Transport-Security</code> by default, and it covers subdomains too, so make sure the site and its subdomains work over HTTPS before you deploy it. If a proxy or CDN already sets the header, avoid sending conflicting values from the app.</p>
</li>
<li><p><strong>Fastify users</strong> can get the same result with the <code>@fastify/helmet</code> plugin.</p>
</li>
</ul>
<p>While you're touching headers and cookies, two related basics belong on the same list:</p>
<ul>
<li><p>Configure <strong>CORS with an explicit allowlist of origins</strong>. Never combine a wildcard origin with credentials.</p>
</li>
<li><p>Set session cookies with <code>httpOnly</code>, <code>secure</code>, and <code>sameSite</code> so scripts can't read them and browsers don't send them where they shouldn't.</p>
</li>
<li><p><code>app.use(helmet())</code> is the first middleware</p>
</li>
<li><p>A CSP is defined, rolled out in report-only mode first, and tested in a real browser with the console open for violations</p>
</li>
<li><p>CORS uses an allowlist, not <code>*</code></p>
</li>
<li><p>Cookies use <code>httpOnly</code>, <code>secure</code>, and <code>sameSite</code></p>
</li>
<li><p>You've scanned your site with a header checker after deploying</p>
</li>
</ul>
<h2>3. Rate Limiting</h2>
<p>Without limits, every endpoint is an invitation: credential stuffing on <code>/login</code>, SMS and email abuse on <code>/forgot-password</code>, scraping on <code>/search</code>, or plain old resource exhaustion. A rate limiter caps how often a client can hit a route.</p>
<pre><code class="language-javascript">import { rateLimit } from "express-rate-limit";

// Tell Express how many proxies sit in front of the app
app.set("trust proxy", 1);

const apiLimiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  limit: 100,
  standardHeaders: "draft-8",
  legacyHeaders: false,
});

const loginLimiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  limit: 5,
  skipSuccessfulRequests: true,
});

// Second limiter keyed on the account being attacked, not the client IP.
// It must run after the body parser so req.body.email is available.
const accountLimiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  limit: 10,
  skipSuccessfulRequests: true,
  keyGenerator: (req) =&gt; `login:${String(req.body.email).trim().toLowerCase()}`,
  skip: (req) =&gt; typeof req.body?.email !== "string",
});

app.use("/api", apiLimiter); // check limits before doing any work
app.use(express.json()); // then parse request bodies
app.post("/api/login", loginLimiter, accountLimiter, loginHandler);
</code></pre>
<p>(This matches <code>express-rate-limit</code> v8. Older versions call the <code>limit</code> option <code>max</code> and may not support <code>draft-8</code> headers.)</p>
<p>The code is the easy part. These details decide whether the limiter actually works:</p>
<ul>
<li><p><strong>Get the client IP right.</strong> Behind a reverse proxy, load balancer, or tunnel, every request appears to come from the proxy unless <code>trust proxy</code> is configured. The result is one shared bucket for all users. Set it to match the number of proxies in front of your app, then confirm it with a temporary test route that returns <code>req.ip</code>: it should show your own address, not the proxy's. Setting it too loosely lets attackers spoof their IP with a forged <code>X-Forwarded-For</code> header.</p>
</li>
<li><p><strong>Use a shared store when you run more than one instance.</strong> The default store lives in each process's memory, so ten instances means ten separate limits. Use a Redis-backed store (for example <code>rate-limit-redis</code>) so the count is shared.</p>
</li>
<li><p><strong>Limit by account, not just by IP.</strong> An attacker with many IPs can still hammer one user's login, which is what the <code>accountLimiter</code> above is for. Hard lockouts let an attacker lock a victim out on purpose, so prefer increasing delays or extra verification (such as a CAPTCHA) over permanent blocks.</p>
</li>
<li><p><strong>Watch out for IPv6.</strong> One client can control a huge range of IPv6 addresses, so limiting a single address is easy to dodge. Recent <code>express-rate-limit</code> versions group IPv6 addresses by subnet by default; keep that behavior unless you have a reason to change it.</p>
</li>
<li><p><strong>Be stricter on expensive or sensitive routes.</strong> Login, signup, password reset, OTP verification, and anything that sends email or SMS deserve tighter limits than a read-only endpoint.</p>
</li>
<li><p><strong>Limit more than request counts.</strong> Cap request body size (Express's default of <code>100kb</code> is a reasonable ceiling, so lower it where you can rather than raising it), keep server timeouts sensible, and paginate list endpoints.</p>
</li>
<li><p><strong>Add a layer at the edge.</strong> A CDN, WAF, or reverse proxy can absorb abusive traffic before it reaches Node.js. Treat the in-app limiter as the last line of defense, not the only one.</p>
</li>
</ul>
<p>Return <code>429 Too Many Requests</code> with a <code>Retry-After</code> header so well-behaved clients know when to come back.</p>
<ul>
<li><p>A global limiter protects the whole API</p>
</li>
<li><p>Login, password reset, signup, and OTP routes have stricter limits</p>
</li>
<li><p>Login attempts are also limited per account</p>
</li>
<li><p><code>trust proxy</code> is set correctly and <code>req.ip</code> is verified in production</p>
</li>
<li><p>A shared store is used if you run multiple instances</p>
</li>
<li><p>Request body size and server timeouts are capped</p>
</li>
</ul>
<h2>4. Password Hashing</h2>
<p>If your database leaks, the way you hashed passwords decides whether attackers get a list of usable passwords or a pile of useless strings. The rule is simple: use a <strong>slow, memory-hard, salted</strong> algorithm designed for passwords, never a general-purpose hash.</p>
<p><strong>Never</strong> use plain text, MD5, SHA-1, or a single round of SHA-256. They're built to be fast, which is exactly what an attacker with a GPU wants.</p>
<p><strong>Recommended order of preference:</strong></p>
<ol>
<li><p><strong>Argon2id</strong>, the current recommendation from OWASP</p>
</li>
<li><p><strong>scrypt</strong>, built into Node.js. Its defaults are weaker than OWASP's minimum (N=2^17, r=8, p=1), so set the cost explicitly and raise <code>maxmem</code> to fit it</p>
</li>
<li><p><strong>bcrypt</strong>, still acceptable for existing systems, with a cost factor of at least 10</p>
</li>
</ol>
<p>Here's Argon2id using the popular <code>argon2</code> package, which handles salting and encoding for you:</p>
<pre><code class="language-typescript">import * as argon2 from "argon2";

const OPTIONS = {
  type: argon2.argon2id,
  memoryCost: 19456, // KiB, about 19 MiB
  timeCost: 2,
  parallelism: 1,
};

// Used to keep response time similar when the user doesn't exist
// (top-level await works in ES modules; otherwise create it in an async setup function)
const DUMMY_HASH = await argon2.hash("not-a-real-password", OPTIONS);

export async function hashPassword(password: string) {
  return argon2.hash(password, OPTIONS);
}

export async function login(email: string, password: string) {
  const user = await findUserByEmail(email);

  const ok = await argon2.verify(user?.passwordHash ?? DUMMY_HASH, password);
  if (!user || !ok) {
    throw new Error("Invalid email or password"); // same message either way
  }

  // Upgrade old hashes as your settings improve
  if (argon2.needsRehash(user.passwordHash, OPTIONS)) {
    await updatePasswordHash(user.id, await argon2.hash(password, OPTIONS));
  }

  return user;
}
</code></pre>
<p>The memory, time, and parallelism numbers above match OWASP's minimum guidance for Argon2id at the time of writing. Treat them as a floor: benchmark on your own hardware and raise them as far as your login latency allows, and check the OWASP Password Storage Cheat Sheet for the latest values. OWASP also lists equivalent trade-offs, such as 46 MiB of memory with a single iteration, if you'd rather spend memory than CPU time. As a rule of thumb, one hash should take well under a second on your production hardware.</p>
<p>A few practical notes:</p>
<ul>
<li><p><strong>Node.js now has Argon2 built in.</strong> Since v24.7.0, <code>crypto.argon2()</code> and <code>crypto.argon2Sync()</code> are part of <code>node:crypto</code>. That's attractive if you want zero native dependencies, but you're responsible for generating the salt, storing the parameters alongside the hash, and comparing results in constant time. For most apps, a well-maintained library is less error-prone.</p>
</li>
<li><p><code>verify()</code> <strong>throws on a malformed hash.</strong> It returns <code>false</code> for a wrong password, but an exception means the stored value is broken or something internal failed. Treat that as a server error, not a failed login.</p>
</li>
<li><p><code>argon2</code> <strong>and</strong> <code>bcrypt</code> <strong>are native modules.</strong> If one fails to load because install scripts are blocked (see the supply-chain section), approve or rebuild just that package.</p>
</li>
<li><p><strong>bcrypt has a 72-byte limit.</strong> Anything longer is silently truncated, so cap password length or pre-hash carefully if you stay on bcrypt.</p>
</li>
<li><p><strong>Cap the maximum password length</strong> (say, 128 characters) so nobody can send a megabyte-long password to burn your CPU.</p>
</li>
<li><p><strong>Favor length over complexity rules.</strong> Long passphrases and screening against known-breached passwords beat forcing symbols and digits.</p>
</li>
<li><p><strong>Don't leak whether an account exists.</strong> Use the same error message and similar timing for "unknown email" and "wrong password".</p>
</li>
<li><p><strong>Offer more than a password.</strong> Add MFA, and consider passkeys (WebAuthn) for new projects.</p>
</li>
<li><p><strong>Never log passwords</strong>, even failed attempts, and redact them from error reports.</p>
</li>
<li><p>Passwords are hashed with Argon2id, scrypt, or bcrypt, never a fast hash</p>
</li>
<li><p>Parameters are benchmarked and stored with the hash so they can be upgraded</p>
</li>
<li><p>Old hashes are re-hashed on successful login when settings change</p>
</li>
<li><p>Maximum password length is capped</p>
</li>
<li><p>Login errors are generic and timing is similar for unknown users</p>
</li>
<li><p>MFA is available, at least for admin accounts</p>
</li>
</ul>
<h2>5. Dependency Audits</h2>
<p>A typical Node.js app pulls in hundreds of transitive packages, and some of them will have known vulnerabilities at any given moment. Audits turn that risk into a list you can act on.</p>
<pre><code class="language-bash"># Check production dependencies and fail on high severity or above
npm audit --omit=dev --audit-level=high

# Verify registry signatures and provenance attestations
npm audit signatures
</code></pre>
<p><code>npm audit</code> compares your lockfile against a database of known advisories. <code>npm audit signatures</code> goes a step further and checks that the packages you installed carry valid registry signatures, and verifies provenance attestations where publishers provide them. Both belong in CI:</p>
<pre><code class="language-yaml"># .github/workflows/security.yml (excerpt)
permissions:
  contents: read

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      # Pin actions to a full commit SHA and note the version in a comment
      - uses: actions/checkout@&lt;full-commit-sha&gt; # vX.Y.Z
      - uses: actions/setup-node@&lt;full-commit-sha&gt; # vX.Y.Z
        with:
          node-version: 24
      - run: npm ci
      - run: npm audit --omit=dev --audit-level=high
      - run: npm audit signatures
</code></pre>
<p>Replace each <code>&lt;full-commit-sha&gt;</code> with the commit SHA of the release you've reviewed; Dependabot can keep pinned SHAs up to date for you.</p>
<p>Some habits make audits far more useful:</p>
<ul>
<li><p><strong>Install from the lockfile.</strong> Use <code>npm ci</code> in CI and Docker builds so you get exactly what was reviewed, and commit your lockfile.</p>
</li>
<li><p><strong>Triage instead of panicking.</strong> Many advisories affect code paths you never call, or dev-only tools. Focus on high and critical issues in production dependencies that your app actually reaches.</p>
</li>
<li><p><strong>Fix at the right level.</strong> Try <code>npm audit fix</code> first, review what it changed, and use <code>overrides</code> in <code>package.json</code> when a deep dependency is stuck on a vulnerable version and the parent hasn't caught up.</p>
</li>
<li><p><strong>Automate updates, but review them.</strong> Dependabot or Renovate keep you current. Group low-risk updates, and read the changelog for anything touching authentication, crypto, or HTTP handling.</p>
</li>
<li><p><strong>Delete what you don't use.</strong> Every unused dependency is attack surface with no benefit. Tools like <code>depcheck</code> or <code>knip</code> find them.</p>
</li>
<li><p><strong>Use a second opinion.</strong> Scanners such as OSV-Scanner, Snyk, or Socket catch things from different data sources, including suspicious package behavior that has no CVE yet.</p>
</li>
</ul>
<p>One important limit: audits only know about <strong>disclosed</strong> problems. A freshly poisoned package has no advisory yet, which brings us to the next section.</p>
<ul>
<li><p>Lockfile is committed and CI uses <code>npm ci</code></p>
</li>
<li><p><code>npm audit</code> runs on every pull request and on a weekly schedule</p>
</li>
<li><p><code>npm audit signatures</code> runs in CI</p>
</li>
<li><p>Automated update PRs are reviewed, not blindly merged</p>
</li>
<li><p>Unused dependencies are removed regularly</p>
</li>
</ul>
<h2>6. Supply-Chain Safety</h2>
<p>Supply-chain attacks target you through the code you trust. The Shai-Hulud worm in late 2025 showed how quickly a compromise can spread from package to package. In March 2026, attackers took over an <code>axios</code> maintainer's account through social engineering and published malicious versions of a package downloaded millions of times a week; those versions pulled in an extra dependency that downloaded remote access trojans.</p>
<p>Two assumptions make these attacks work: that you install the bad version quickly, and that your package manager runs install scripts (npm 12 now blocks them by default, but older versions and other tools may not). Breaking both assumptions stops many of these attacks, though not all of them (more on the limits below).</p>
<h3>Add a release cooldown</h3>
<p>Compromised versions are usually spotted and pulled within hours or days. If your tooling refuses to install anything younger than a week, you let everyone else find the problem first.</p>
<pre><code class="language-toml"># .npmrc (min-release-age needs npm 11.10.0 or newer)

# Don't install versions published in the last 7 days
min-release-age=7
# Save exact versions instead of ^ranges
save-exact=true
# Treat "engines" mismatches as errors instead of warnings
engine-strict=true
</code></pre>
<pre><code class="language-yaml"># pnpm-workspace.yaml
minimumReleaseAge: 10080   # minutes (7 days)
</code></pre>
<p>The units differ between tools: npm uses days, pnpm uses minutes, and Bun uses seconds. The npm 12 defaults GitHub announced cover install scripts, git dependencies, and remote URLs, so a cooldown is still something you opt into yourself. Option names and defaults keep evolving, so confirm them against the documentation for the version you run.</p>
<p>Cooldowns of a few days to a week are common. Longer is safer but delays updates, and a cooldown can also delay a genuine security fix. When you need a patch right away, review that release yourself and add a per-package exception (pnpm has <code>minimumReleaseAgeExclude</code>) instead of switching the cooldown off.</p>
<h3>Control install scripts</h3>
<p>Lifecycle scripts like <code>postinstall</code> run code on your laptop or CI machine the moment you install, before you've looked at anything. That's why attackers love them, and it's how the Shai-Hulud worm spread.</p>
<p><strong>npm 12 and later (released in July 2026) blocks them by default.</strong> Install scripts from dependencies, native <code>node-gyp</code> builds, and <code>prepare</code> scripts from git dependencies only run if you allow them. You review packages once, and the allowlist is saved in <code>package.json</code> so the whole team and CI share it:</p>
<pre><code class="language-bash"># List dependencies that have install scripts
npm approve-scripts --allow-scripts-pending

# Allow the ones you trust, block the rest
npm approve-scripts sharp esbuild
npm deny-scripts some-package
</code></pre>
<p>Commit the updated <code>package.json</code>, and treat a pull request that adds a package to the allowlist as a real review, not a rubber stamp. To prepare, upgrade to npm 11.16 or later, where the same checks appear as warnings. Pin the npm version in CI too, so a surprise upgrade doesn't break your pipeline.</p>
<p><strong>On npm 11 and older</strong>, turn scripts off globally and allow them case by case:</p>
<pre><code class="language-toml"># .npmrc on npm 11 and older
ignore-scripts=true
# Block dependencies fetched straight from git (npm 11.10+, and the default in npm 12)
allow-git=none
</code></pre>
<pre><code class="language-bash">npm ci --ignore-scripts
npm rebuild sharp esbuild --ignore-scripts=false
</code></pre>
<p>pnpm doesn't run dependency scripts unless you allow them. List the trusted packages in <code>onlyBuiltDependencies</code> (or let <code>pnpm approve-builds</code> update the list for you):</p>
<pre><code class="language-yaml"># pnpm-workspace.yaml
onlyBuiltDependencies:
  - sharp
  - esbuild
</code></pre>
<p>One caution: <code>ignore-scripts=true</code> is a blanket switch that also skips scripts you've approved, so on npm 12 rely on the allowlist instead of combining the two.</p>
<p><strong>These settings are strong, but they aren't a complete defense.</strong> Blocking install scripts closes one execution path, but a malicious package can still run when your app imports it or when you execute its command (for example through <code>npx</code>). A cooldown won't help if a compromise stays hidden for weeks, and an allowlist is only as good as the review behind it. Treat each of these as one layer among several.</p>
<h3>Check where packages come from</h3>
<ul>
<li><p>Run <code>npm audit signatures</code> to confirm what you installed matches what the registry signed.</p>
</li>
<li><p>Prefer packages that publish with provenance, which links a release to the source repository and build workflow that produced it.</p>
</li>
<li><p>Watch out for look-alike names (typosquats) and packages with a sudden ownership change.</p>
</li>
<li><p>Avoid dependencies pulled straight from git URLs or remote tarballs. npm 12 blocks both by default, and on npm 11.10 and later you can set <code>allow-git=none</code> yourself. Git dependencies skip registry review, and a git dependency's own <code>.npmrc</code> can override the Git executable even when <code>ignore-scripts</code> is on.</p>
</li>
<li><p>Before adding a package, check how it's maintained, how many dependencies it drags in, and whether it really does something you can't write in a few lines.</p>
</li>
</ul>
<h3>Limit the blast radius</h3>
<p>Assume a bad package will eventually get through, and make sure it finds very little to steal.</p>
<ul>
<li><p><strong>Keep secrets away from install steps.</strong> Don't run <code>npm install</code> in an environment that holds production credentials. Build in one stage without secrets, and inject runtime secrets only into the final container.</p>
</li>
<li><p><strong>Scope CI tokens tightly.</strong> Use short-lived, least-privilege credentials, and avoid long-lived tokens with publish rights.</p>
</li>
<li><p><strong>Pin GitHub Actions to commit SHAs</strong> and give workflows read-only <code>permissions</code> by default. Tags can be moved, and in 2025 the popular <code>tj-actions/changed-files</code> action was compromised exactly that way. Dependabot's <code>github-actions</code> ecosystem can keep pinned SHAs up to date.</p>
</li>
<li><p><strong>Run containers as a non-root user</strong> with a read-only filesystem where you can.</p>
</li>
<li><p><strong>Consider Node's permission model</strong> (<code>--permission</code>) as an extra layer to restrict file and network access. Treat it as defense in depth, not a sandbox for untrusted code.</p>
</li>
<li><p><strong>Know your exposure.</strong> When news of a compromised package breaks, you should be able to answer "do we use it, and which version?" in minutes. <code>npm ls &lt;package&gt;</code> and your lockfile give you that answer.</p>
</li>
</ul>
<h3>If you publish packages</h3>
<ul>
<li><p>Turn on 2FA, preferably with passkeys or hardware keys, for every maintainer.</p>
</li>
<li><p>Publish from CI using <strong>npm trusted publishing</strong> (OIDC) instead of a long-lived token, so releases carry provenance and there's no token to steal. Once it's set up, turn off token-based publishing for the package.</p>
</li>
<li><p>Plan for long-lived tokens going away. GitHub has announced that 2FA-bypass access tokens will lose the ability to publish directly (expected around January 2027), so move automated publishing to trusted publishing, or to staged publishing with a human approval step.</p>
</li>
<li><p>Keep your own dependency list small and your install scripts empty.</p>
</li>
</ul>
<h3>Be careful with auto-updates</h3>
<p>Bots that open pull requests for every new release are convenient, but they can also hand you a poisoned version within minutes of its publication. Configure Renovate or Dependabot to wait before proposing brand-new releases, and keep a human in the loop for merges.</p>
<ul>
<li><p>A release cooldown is configured for your package manager</p>
</li>
<li><p>Install scripts are blocked by default (the npm 12 allowlist, or <code>ignore-scripts</code> on older versions), with a short, reviewed allowlist</p>
</li>
<li><p>Git-sourced and remote-URL dependencies are blocked or reviewed</p>
</li>
<li><p>The npm version is pinned in CI</p>
</li>
<li><p>Builds run without production secrets</p>
</li>
<li><p>CI tokens are scoped and short-lived; publishing uses trusted publishing</p>
</li>
<li><p>GitHub Actions are pinned to commit SHAs with read-only default permissions</p>
</li>
<li><p>You can list every project that uses a given package within minutes</p>
</li>
</ul>
<h2>A Few More Basics Worth Checking</h2>
<p>These didn't get their own section, but they belong on every checklist:</p>
<ul>
<li><p><strong>Validate all input</strong> at the boundary with a schema library such as Zod.</p>
</li>
<li><p><strong>Use parameterized queries</strong> or an ORM; never build SQL or NoSQL queries from raw strings.</p>
</li>
<li><p><strong>Keep secrets out of images and logs.</strong> Prefer file-based secrets over environment variables for real credentials.</p>
</li>
<li><p><strong>Serve everything over HTTPS</strong> and redirect HTTP.</p>
</li>
<li><p><strong>Return generic error messages</strong> to clients, and keep stack traces in your logs only.</p>
</li>
<li><p><strong>Log security events</strong> such as failed logins and permission errors, and alert on spikes.</p>
</li>
</ul>
<h2>The Full Checklist</h2>
<p><strong>Runtime</strong></p>
<ul>
<li><p>Production runs a supported Node.js LTS line (not Node.js 20)</p>
</li>
<li><p>Docker images are pinned to a major version and rebuilt regularly</p>
</li>
</ul>
<p><strong>Headers and browser protections</strong></p>
<ul>
<li><p>Helmet is enabled and the CSP is tested</p>
</li>
<li><p>CORS uses an allowlist; cookies are <code>httpOnly</code>, <code>secure</code>, and <code>sameSite</code></p>
</li>
</ul>
<p><strong>Rate limiting</strong></p>
<ul>
<li><p>Global and per-route limits are in place, with stricter limits on auth routes</p>
</li>
<li><p><code>trust proxy</code> is correct and a shared store is used across instances</p>
</li>
</ul>
<p><strong>Passwords</strong></p>
<ul>
<li><p>Argon2id (or scrypt/bcrypt) with benchmarked parameters</p>
</li>
<li><p>Generic login errors, capped password length, MFA available</p>
</li>
</ul>
<p><strong>Dependency audits</strong></p>
<ul>
<li><p><code>npm ci</code>, <code>npm audit</code>, and <code>npm audit signatures</code> run in CI</p>
</li>
<li><p>Update PRs are reviewed; unused packages are removed</p>
</li>
</ul>
<p><strong>Supply chain</strong></p>
<ul>
<li><p>Release cooldown configured; install scripts blocked except for a reviewed allowlist</p>
</li>
<li><p>Secrets are kept out of install and build steps</p>
</li>
<li><p>Trusted publishing and 2FA for anything you publish</p>
</li>
<li><p>CI actions pinned to commit SHAs</p>
</li>
</ul>
<h2>Wrapping Up</h2>
<p>You don't need to do everything in a single afternoon. A good order is: update your runtime, add Helmet and rate limiting (an hour of work), confirm your password hashing, then wire audits and a release cooldown into CI and your <code>.npmrc</code>. Together those steps close some of the most common gaps attackers rely on.</p>
<p>Security is also never finished. Dependencies change, new advisories appear, and your own code grows. Put a recurring reminder in your calendar, run through this list every quarter, and fix whatever has drifted.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/node-js-security-checklist-for-2026-fym1k?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Databases in Depth: SQL, Indexes, Transactions, and When to Use What]]></title><description><![CDATA[A real AI app has to remember users, conversations, and token costs. This post teaches SQL, joins, indexes, transactions, and SQL injection safety using…]]></description><link>https://blog.zyvop.com/databases-in-depth-sql-indexes-transactions-and-when-to-use-what-pklyn</link><guid isPermaLink="true">https://blog.zyvop.com/databases-in-depth-sql-indexes-transactions-and-when-to-use-what-pklyn</guid><category><![CDATA[#aiengineer]]></category><category><![CDATA[Databases]]></category><category><![CDATA[PostgreSQL]]></category><category><![CDATA[SQL]]></category><category><![CDATA[SQLite]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Thu, 05 Nov 2026 07:53:46 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/7dc9b482-3026-4a8c-94ea-c0fc920c5cb5.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>So far, everything our programs remember disappears when they stop. A real AI app has to remember a lot: users, conversations, documents, which prompts were sent, what the model answered, how many tokens it cost, and whether the user clicked thumbs up. That's the job of a <strong>database</strong>, and it's the next step in becoming a real engineer. This post is core.</p>
<p>We'll learn it by doing. Python comes with a small database called SQLite built in, so there's nothing to install, and almost everything we learn carries straight over to Postgres, the database most production AI apps use. It also links back to earlier posts: indexes are the trees from Post 16 and the speedups from Post 18, a good schema looks like the Pydantic models from Post 10, and tests can use a throwaway database like the fixtures from Post 13.</p>
<h2>Why not just use files?</h2>
<p>You could store everything in a JSON file, or in a Python dictionary that you save now and then. For small things, that works. But it breaks down quickly:</p>
<ul>
<li><p><strong>Searching</strong> means loading and scanning everything.</p>
</li>
<li><p><strong>Two users at once</strong> can overwrite each other's changes.</p>
</li>
<li><p><strong>A crash halfway through writing</strong> can leave a corrupted file.</p>
</li>
<li><p><strong>Nothing checks your data.</strong> A typo in a field name just silently creates a new field.</p>
</li>
</ul>
<p>A database solves all four. It stores data in an organized way, finds things fast, lets many users work at once, survives crashes, and enforces rules about what's allowed in.</p>
<h2>Tables, rows, and keys</h2>
<p>Most databases you'll use are <strong>relational</strong>. Data lives in <strong>tables</strong>, like spreadsheets with strict rules. Each <strong>row</strong> is one record, and each <strong>column</strong> is one kind of fact about it.</p>
<p>Tables connect through keys. A <strong>primary key</strong> is a column that uniquely identifies each row, usually a number called <code>id</code>. A <strong>foreign key</strong> is a column that points at a row in another table. Here's the structure for a small chat app:</p>
<ul>
<li><p>A <strong>user</strong> has an <code>id</code> and an email.</p>
</li>
<li><p>A <strong>conversation</strong> belongs to a user, so it holds a <code>user_id</code> that points at a user.</p>
</li>
<li><p>A <strong>message</strong> belongs to a conversation, so it holds a <code>conversation_id</code>.</p>
</li>
</ul>
<p>Notice that a message doesn't copy the user's email. It points at the conversation, which points at the user. Each fact lives in exactly one place, so if the email changes, you change it once.</p>
<p>The language for talking to relational databases is <strong>SQL</strong>, which is short for Structured Query Language. It's been around for decades, it's the same idea everywhere, and it's among the most valuable skills you can add.</p>
<h2>SQLite: a database with nothing to install</h2>
<p>SQLite keeps a whole database in a single file (or entirely in memory), and Python includes it:</p>
<pre><code class="language-python">import sqlite3

conn = sqlite3.connect(":memory:")    # a throwaway database that lives in RAM
</code></pre>
<p>Use <code>":memory:"</code> for experiments and tests, or a filename like <code>"app.db"</code> to keep the data on disk. SQLite is a great fit for learning, testing, and small apps. For a real service with many users writing at once, you'd move to Postgres, which we'll cover below.</p>
<h2>Designing a schema</h2>
<p>A <strong>schema</strong> is the structure of your tables. Here's one for our chat app:</p>
<pre><code class="language-sql">CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    email TEXT NOT NULL UNIQUE,
    created_at TEXT NOT NULL
);

CREATE TABLE conversations (
    id INTEGER PRIMARY KEY,
    user_id INTEGER NOT NULL REFERENCES users(id),
    title TEXT,
    created_at TEXT NOT NULL
);

CREATE TABLE messages (
    id INTEGER PRIMARY KEY,
    conversation_id INTEGER NOT NULL REFERENCES conversations(id),
    role TEXT NOT NULL CHECK (role IN ('user', 'assistant', 'system')),
    content TEXT NOT NULL,
    tokens INTEGER,
    created_at TEXT NOT NULL
);
</code></pre>
<p>By convention, SQL keywords are written in capitals, though it isn't required. The pieces on each column are <strong>constraints</strong>, rules the database enforces so bad data can't get in:</p>
<ul>
<li><p><code>PRIMARY KEY</code> marks the unique identifier.</p>
</li>
<li><p><code>NOT NULL</code> means the value is required.</p>
</li>
<li><p><code>UNIQUE</code> means no two rows can share the value. Here, no two users with the same email.</p>
</li>
<li><p><code>REFERENCES users(id)</code> makes it a foreign key.</p>
</li>
<li><p><code>CHECK (...)</code> lets you write your own rule. Here, a message's role must be one of three values.</p>
</li>
</ul>
<p>Constraints are the database version of the validation that Pydantic did in Post 10, and they have a big advantage: they protect your data no matter <em>which</em> program writes to it. Let's load some data and watch them work. Here's a connection with a schema, and some rows:</p>
<pre><code class="language-python">conn = sqlite3.connect(":memory:")
conn.execute("PRAGMA foreign_keys = ON")
conn.executescript(SCHEMA)    # the three CREATE TABLE statements above

conn.executemany(
    "INSERT INTO users (id, email, created_at) VALUES (?, ?, ?)",
    [
        (1, "asha@example.com", "2026-10-01T09:00:00Z"),
        (2, "ravi@example.com", "2026-10-02T10:30:00Z"),
        (3, "meera@example.com", "2026-10-03T08:15:00Z"),
    ],
)
</code></pre>
<p>(The conversations and messages are inserted the same way. The full runnable file is in the mini project at the end.) Now try to break the rules:</p>
<pre><code class="language-python">conn.execute("INSERT INTO messages (conversation_id, role, content, created_at) VALUES (1, 'robot', 'x', 'x')")
# IntegrityError: CHECK constraint failed: role IN ('user', 'assistant', 'system')

conn.execute("INSERT INTO users (email, created_at) VALUES (NULL, 'x')")
# IntegrityError: NOT NULL constraint failed: users.email

conn.execute("INSERT INTO users (email, created_at) VALUES ('asha@example.com', 'x')")
# IntegrityError: UNIQUE constraint failed: users.email
</code></pre>
<p>Every one of those is rejected, with a clear message.</p>
<h3>Two SQLite surprises</h3>
<p>SQLite has a couple of quirks that catch people out, and it's better to meet them now.</p>
<p><strong>Foreign keys are off by default.</strong> See the <code>PRAGMA foreign_keys = ON</code> line? SQLite doesn't enforce foreign keys unless you turn them on, <em>for each new connection</em>. With it on, inserting a message for a conversation that doesn't exist fails:</p>
<pre><code class="language-sql">IntegrityError: FOREIGN KEY constraint failed
</code></pre>
<p>With it off, the "orphan" row is accepted quietly. When I tried it, the database happily stored a message pointing at conversation 999, which doesn't exist. Postgres always enforces them, so this is just an SQLite habit to build: turn them on.</p>
<p><strong>Types are loose.</strong> SQLite will store text in an <code>INTEGER</code> column without complaint:</p>
<pre><code class="language-python">conn.execute("CREATE TABLE t (n INTEGER)")
conn.execute("INSERT INTO t VALUES ('abc')")
conn.execute("INSERT INTO t VALUES ('42')")
print(conn.execute("SELECT n, typeof(n) FROM t").fetchall())
# [('abc', 'text'), (42, 'integer')]
</code></pre>
<p>The <code>'42'</code> was converted to a number, but <code>'abc'</code> stayed text, in a column declared as an integer. If you want strict types, add the <code>STRICT</code> keyword to the end of the table definition (<code>CREATE TABLE t (n INTEGER) STRICT</code>), and the same insert fails with a clear error. Postgres is strict by default.</p>
<h2>Getting data in and out safely</h2>
<p>You've seen <code>?</code> in the code above. Those are <strong>placeholders</strong>, and they're one of the most important habits in this whole post.</p>
<h3>SQL injection</h3>
<p>It's tempting to build a query with an f-string, since you know f-strings well by now. <strong>Never do this with data that came from outside your program.</strong> Here's why:</p>
<pre><code class="language-python">evil = "nobody' OR '1'='1"

rows = conn.execute(f"SELECT email FROM users WHERE email = '{evil}'").fetchall()
print(rows)
# [('asha@example.com',), ('meera@example.com',), ('ravi@example.com',)]
</code></pre>
<p>The query that actually ran was:</p>
<pre><code class="language-sql">SELECT email FROM users WHERE email = 'nobody' OR '1'='1'
</code></pre>
<p>The text we pasted in broke out of the quotes and added its own condition, <code>'1'='1'</code>, which is always true. It returned <em>every user</em>. A malicious person can use this trick to read data they shouldn't, change it, or delete it. It's called <strong>SQL injection</strong>, and it's one of the oldest and most damaging security bugs in software. This matters even more now: if your app lets a <em>model</em> write or influence a query, the model's output is untrusted input too.</p>
<p>The fix is to pass values separately, so the database treats them as data and never as code:</p>
<pre><code class="language-python">rows = conn.execute("SELECT email FROM users WHERE email = ?", (evil,)).fetchall()
print(rows)    # []
</code></pre>
<p>The database looks for a user whose email is literally <code>nobody' OR '1'='1</code>, finds none, and returns nothing. Always use placeholders. (The symbol varies by database. SQLite uses <code>?</code>, and the common Postgres library, psycopg, uses <code>%s</code>. The principle is the same.)</p>
<h3>Reading rows</h3>
<p>By default, rows come back as plain tuples. Setting a <code>row_factory</code> lets you access columns by name, which is much more readable:</p>
<pre><code class="language-python">conn.row_factory = sqlite3.Row
row = conn.execute("SELECT id, email FROM users WHERE id = 1").fetchone()
print(row["email"], dict(row))    # asha@example.com {'id': 1, 'email': 'asha@example.com'}
</code></pre>
<h2>Asking better questions</h2>
<p>Here's the data we'll query. Three users, three conversations, and eight messages:</p>
<table>
<thead>
<tr>
<th>User</th>
<th>Conversation</th>
<th>Messages</th>
</tr>
</thead>
<tbody><tr>
<td>asha</td>
<td>"Python help"</td>
<td>4 (about generators)</td>
</tr>
<tr>
<td>asha</td>
<td>"Trip ideas"</td>
<td>2</td>
</tr>
<tr>
<td>ravi</td>
<td>"Resume review"</td>
<td>2 (one has no token count recorded)</td>
</tr>
<tr>
<td>meera</td>
<td>(none)</td>
<td></td>
</tr>
</tbody></table>
<h3>The basics: WHERE, ORDER BY, LIMIT</h3>
<pre><code class="language-sql">SELECT id, role, content
FROM messages
WHERE conversation_id = 1 AND role = 'user'
ORDER BY created_at DESC
LIMIT 5;
</code></pre>
<pre><code class="language-rust">(3, 'user', 'Show an example')
(1, 'user', 'How do generators work?')
</code></pre>
<p><code>WHERE</code> filters rows, <code>ORDER BY</code> sorts them (<code>DESC</code> for newest first), and <code>LIMIT</code> caps how many come back.</p>
<h3>Aggregates and GROUP BY</h3>
<p>Functions like <code>COUNT</code>, <code>SUM</code>, <code>AVG</code>, <code>MIN</code>, and <code>MAX</code> summarize rows. <code>GROUP BY</code> does it per group:</p>
<pre><code class="language-sql">SELECT role, COUNT(*) AS n, SUM(tokens) AS total_tokens, AVG(tokens) AS avg_tokens
FROM messages
GROUP BY role;
</code></pre>
<pre><code class="language-typescript">('assistant', 4, 24, 8.0)
('user', 4, 20, 5.0)
</code></pre>
<p><code>HAVING</code> is like <code>WHERE</code>, but for groups, after they've been counted. Which users have more than one conversation?</p>
<pre><code class="language-sql">SELECT user_id, COUNT(*) AS conversations
FROM conversations
GROUP BY user_id
HAVING COUNT(*) &gt; 1;
</code></pre>
<pre><code>(1, 2)
</code></pre>
<h3>Joins: combining tables</h3>
<p>Because each fact lives in one table, answering real questions means combining tables with a <strong>JOIN</strong>. "How many tokens has each user used?" needs all three:</p>
<pre><code class="language-sql">SELECT u.email, SUM(m.tokens) AS total_tokens
FROM users u
JOIN conversations c ON c.user_id = u.id
JOIN messages m ON m.conversation_id = c.id
GROUP BY u.email
ORDER BY total_tokens DESC;
</code></pre>
<pre><code class="language-typescript">('asha@example.com', 40)
('ravi@example.com', 4)
</code></pre>
<p>(<code>u</code>, <code>c</code>, and <code>m</code> are short aliases for the table names.) Each <code>ON</code> says how two tables line up.</p>
<p>Notice something. <strong>meera is missing.</strong> She has no conversations, so a plain <code>JOIN</code> (called an inner join) drops her. If you want to keep every user, even those with nothing to match, use a <code>LEFT JOIN</code>:</p>
<pre><code class="language-sql">SELECT u.email, c.title
FROM users u
LEFT JOIN conversations c ON c.user_id = u.id;
</code></pre>
<pre><code class="language-python">('asha@example.com', 'Python help')
('asha@example.com', 'Trip ideas')
('ravi@example.com', 'Resume review')
('meera@example.com', None)
</code></pre>
<p>Meera shows up, with <code>None</code> (SQL's <code>NULL</code>) where the conversation would be. And that gives you a handy trick for finding things that <em>don't</em> have a match, such as users who've never started a chat:</p>
<pre><code class="language-sql">SELECT u.email
FROM users u
LEFT JOIN conversations c ON c.user_id = u.id
WHERE c.id IS NULL;
</code></pre>
<h3>Making queries readable: CTEs</h3>
<p>A <strong>CTE</strong> (common table expression), written with <code>WITH</code>, gives a name to an intermediate result, like a variable. It makes a long query much easier to read:</p>
<pre><code class="language-sql">WITH counts AS (
    SELECT conversation_id, COUNT(*) AS n
    FROM messages
    GROUP BY conversation_id
)
SELECT c.title, counts.n
FROM counts
JOIN conversations c ON c.id = counts.conversation_id
WHERE counts.n &gt;= 2
ORDER BY counts.n DESC;
</code></pre>
<h3>Window functions</h3>
<p>Here's a question that comes up constantly: what's the <em>latest</em> message in each conversation? <code>GROUP BY</code> can't easily give you the whole row. A <strong>window function</strong> can, by numbering the rows within each group:</p>
<pre><code class="language-sql">SELECT conversation_id, role, content
FROM (
    SELECT m.*,
           ROW_NUMBER() OVER (PARTITION BY conversation_id ORDER BY created_at DESC) AS rn
    FROM messages m
)
WHERE rn = 1;
</code></pre>
<pre><code class="language-typescript">(1, 'assistant', 'Here is a small example.')
(2, 'assistant', 'Day 1: Alfama and the castle.')
(3, 'assistant', 'Lead with your strongest project.')
</code></pre>
<p><code>PARTITION BY</code> splits the rows into groups, <code>ORDER BY</code> sorts within each group, and <code>ROW_NUMBER()</code> numbers them, so <code>rn = 1</code> is the latest one in each group.</p>
<h3>NULL is not a value</h3>
<p><code>NULL</code> means "unknown or missing," and it behaves in surprising ways. One of Ravi's messages has no token count, so it's <code>NULL</code>:</p>
<pre><code class="language-sql">SELECT COUNT(*) FROM messages WHERE tokens = NULL;     -- 0  (!)
SELECT COUNT(*) FROM messages WHERE tokens IS NULL;    -- 1
</code></pre>
<p>Nothing is ever equal to <code>NULL</code>, not even <code>NULL</code>. You have to ask with <code>IS NULL</code>. Aggregates also skip <code>NULL</code>s:</p>
<pre><code class="language-sql">SELECT COUNT(*), COUNT(tokens), SUM(tokens), AVG(tokens) FROM messages;
-- 8, 7, 44, 6.2857...
</code></pre>
<p>There are 8 messages, but <code>COUNT(tokens)</code> counts only the 7 that have a value, and the average is over those 7. That can quietly skew your numbers. When a column can be <code>NULL</code>, always ask yourself what the missing rows should mean.</p>
<h2>Indexes: why queries are fast</h2>
<p>Remember Post 18. A search through a list is O(n), and a search through a balanced tree is O(log n). A database faces the same choice. Without help, finding all the messages in one conversation means reading <strong>every row</strong> in the table: a <strong>full table scan</strong>, O(n). An <strong>index</strong> is an extra structure, usually a balanced tree (Post 16), that lets the database jump straight to the matching rows.</p>
<p>You can ask the database how it plans to run a query. In SQLite that's <code>EXPLAIN QUERY PLAN</code>. I filled a <code>messages</code> table with 200,000 rows spread over 2,000 conversations, and ran this:</p>
<pre><code class="language-sql">SELECT id, content FROM messages WHERE conversation_id = 777 ORDER BY created_at;
</code></pre>
<p>Without an index, the plan is:</p>
<pre><code class="language-css">SCAN messages
USE TEMP B-TREE FOR ORDER BY
</code></pre>
<p><code>SCAN</code> means every row gets read. It took about <strong>15 milliseconds</strong> per query. Then I created an index:</p>
<pre><code class="language-sql">CREATE INDEX idx_messages_conversation ON messages (conversation_id);
</code></pre>
<pre><code class="language-sql">SEARCH messages USING INDEX idx_messages_conversation (conversation_id=?)
USE TEMP B-TREE FOR ORDER BY
</code></pre>
<p><code>SEARCH ... USING INDEX</code> means it jumped straight to the right rows. About <strong>0.11 milliseconds</strong>. But there's still a sort step, <code>USE TEMP B-TREE FOR ORDER BY</code>. A <strong>composite index</strong> over two columns fixes that, because it keeps the rows sorted by the second column inside each value of the first:</p>
<pre><code class="language-sql">CREATE INDEX idx_messages_conv_time ON messages (conversation_id, created_at);
</code></pre>
<pre><code class="language-sql">SEARCH messages USING INDEX idx_messages_conv_time (conversation_id=?)
</code></pre>
<p>Now the sort is gone, because the index already delivers rows in order. About <strong>0.09 milliseconds</strong>, which is roughly <strong>170 times faster</strong> than the scan, on a table that's still small by real standards. (Your numbers will differ.)</p>
<h3>Rules of thumb for indexes</h3>
<ul>
<li><p><strong>Index the columns you filter, join, and sort by.</strong> Foreign key columns like <code>conversation_id</code> are classic candidates.</p>
</li>
<li><p><strong>Primary keys and</strong> <code>UNIQUE</code> <strong>columns are indexed automatically.</strong></p>
</li>
<li><p><strong>Column order matters in a composite index.</strong> An index on <code>(conversation_id, created_at)</code> helps queries that filter on <code>conversation_id</code>, or on both. It doesn't help a query that filters only on <code>created_at</code>.</p>
</li>
<li><p><strong>Indexes aren't free.</strong> Every insert, update, and delete has to update every index too. I inserted 200,000 rows into a three-column table with no indexes, and then with three indexes. The indexed one took about <strong>4 times longer</strong> (0.67 seconds versus 0.16). They also take space. So don't index everything. Index what your real queries need.</p>
</li>
<li><p><strong>Wrapping a column in a function can defeat the index.</strong> Look at what happens with email lookups. This uses the index:</p>
</li>
</ul>
<pre><code class="language-sql">SELECT id FROM users WHERE email = 'User5@Example.com'
-&gt; SEARCH users USING COVERING INDEX idx_users_email (email=?)
</code></pre>
<p>But if you make the search case-insensitive with <code>lower()</code>, the database can no longer use the index on <code>email</code>:</p>
<pre><code class="language-sql">SELECT id FROM users WHERE lower(email) = 'user5@example.com'
-&gt; SCAN users
</code></pre>
<p>The fix is an <strong>expression index</strong>, which indexes the lowercased value itself. After <code>CREATE INDEX idx_users_email_lower ON users (lower(email))</code>, the same query shows <code>SEARCH users USING INDEX idx_users_email_lower</code>.</p>
<ul>
<li><strong>Searching <em>inside</em> text can't use a normal index.</strong> A pattern like <code>LIKE '%word%'</code> has to look at every row. For real text search, databases have dedicated full-text search features.</li>
</ul>
<p><code>EXPLAIN</code> is the database's version of the profiler from Post 15. When a query is slow, look at the plan first, instead of guessing.</p>
<h2>Transactions: all or nothing</h2>
<p>Imagine saving a chat turn: you store the user's message, call the model, then store the model's reply. What if the model call fails after the first insert? You'd have a question with no answer sitting in your database.</p>
<p>A <strong>transaction</strong> groups several statements so that either <strong>all</strong> of them take effect, or <strong>none</strong> do. In Python's <code>sqlite3</code>, the connection itself works as a context manager (Post 8): it commits if the block finishes normally and rolls back if an exception escapes.</p>
<pre><code class="language-python">def save_exchange(conn, conversation_id, question, answer_func):
    with conn:    # commit on success, roll back on any error
        conn.execute(
            "INSERT INTO messages (conversation_id, role, content) VALUES (?, 'user', ?)",
            (conversation_id, question),
        )
        answer = answer_func(question)
        conn.execute(
            "INSERT INTO messages (conversation_id, role, content) VALUES (?, 'assistant', ?)",
            (conversation_id, answer),
        )
</code></pre>
<p>When I ran it once successfully, there were 2 messages in the table. Then I ran it with a model function that raises an error. The error propagated, and the count was <strong>still 2</strong>. The user message, which had already been inserted inside the block, was rolled back along with everything else.</p>
<p>(Note that <code>with conn:</code> doesn't close the connection. It only ends the transaction.)</p>
<p>A warning, though. There's a design trap hiding in this example. <strong>Don't hold a transaction open while you wait on something slow</strong>, such as a model call that takes 30 seconds. An open transaction can hold locks that block other users. A better pattern is to call the model first, then open a transaction that just saves the results, which is quick. The <code>add_exchange</code> method in the mini project does exactly that.</p>
<h3>ACID</h3>
<p>You'll hear transactions described with the word <strong>ACID</strong>:</p>
<ul>
<li><p><strong>Atomic</strong>: all or nothing, as above.</p>
</li>
<li><p><strong>Consistent</strong>: the rules (constraints) always hold. A transaction that would break one is rejected.</p>
</li>
<li><p><strong>Isolated</strong>: transactions running at the same time don't see each other's half-finished work.</p>
</li>
<li><p><strong>Durable</strong>: once committed, the data survives a crash or power cut.</p>
</li>
</ul>
<p>That's a lot of safety you'd have to build yourself with files.</p>
<h3>Concurrency and lost updates</h3>
<p>Isolation matters when two things happen at once. Here's a bug that catches people. Imagine a counter, and two workers that each read it, add one, and write it back:</p>
<pre><code class="language-python">a = conn.execute("SELECT n FROM counter").fetchone()[0]    # worker A reads 0
b = conn.execute("SELECT n FROM counter").fetchone()[0]    # worker B reads 0
conn.execute("UPDATE counter SET n = ?", (a + 1,))         # A writes 1
conn.execute("UPDATE counter SET n = ?", (b + 1,))         # B writes 1 (A's update is lost!)
</code></pre>
<p>Two increments happened, but the counter says 1. This "read, compute in Python, write back" pattern is a <strong>race condition</strong>, just like the thread problem in Post 14. The fix is to let the database do the arithmetic in one atomic statement:</p>
<pre><code class="language-sql">UPDATE counter SET n = n + 1;
</code></pre>
<p>Run that twice, and you get 2. This is the right way to handle things like usage quotas. Say each user has some credits:</p>
<pre><code class="language-python">def use_credit(conn, user_id):
    with conn:
        cursor = conn.execute(
            "UPDATE quotas SET credits = credits - 1 WHERE user_id = ? AND credits &gt; 0",
            (user_id,),
        )
        return cursor.rowcount == 1
</code></pre>
<p>The <code>AND credits &gt; 0</code> check and the subtraction happen together, atomically. <code>rowcount</code> tells you whether a row was actually updated, so <code>True</code> means "credit used" and <code>False</code> means "out of credits." With a user who had 2 credits, three calls gave <code>[True, True, False]</code>. A <code>CHECK (credits &gt;= 0)</code> constraint on the column adds a safety net, so even careless code can't push the balance below zero.</p>
<p>Postgres, which handles many users at once, offers more tools here, such as different isolation levels and <code>SELECT ... FOR UPDATE</code> to lock a row while you work on it. The principle is the same: make the database do the read-and-change in one step.</p>
<h2>Schema design tips</h2>
<ul>
<li><p><strong>Put each fact in one place.</strong> This is called <strong>normalization</strong>. Don't copy a user's email into every message. Point at the user instead.</p>
</li>
<li><p><strong>Many-to-many relationships need a third table.</strong> If a document can have many tags and a tag can belong to many documents, create a <code>document_tags</code> table with a <code>document_id</code> and a <code>tag_id</code>.</p>
</li>
<li><p><strong>Store timestamps in UTC.</strong> Format them consistently (ISO 8601 text works in SQLite, and Postgres has a proper <code>timestamptz</code> type). Convert to a user's local time only when showing them.</p>
</li>
<li><p><strong>A JSON column is fine for messy, changing data.</strong> Model APIs return nested JSON, and you often want to keep the whole raw response for debugging. Store it in a text column (SQLite) or a <code>jsonb</code> column (Postgres), and you can still query inside it:</p>
</li>
</ul>
<pre><code class="language-python">conn.execute("INSERT INTO runs (model, response) VALUES (?, ?)", ("some-model", json.dumps(payload)))

conn.execute(
    "SELECT json_extract(response, '$.usage.output_tokens') FROM runs"
).fetchall()
# [(30,)]
</code></pre>
<p>Keep the things you filter and sort on as proper columns, and use JSON for the rest.</p>
<ul>
<li><p><strong>Change schemas with migrations.</strong> Don't hand-edit a production database. Tools called migration tools keep a versioned history of every schema change, so you can apply the same changes to every environment. (Alembic is a common one in Python.)</p>
</li>
<li><p><strong>ORMs exist.</strong> An object-relational mapper, like SQLAlchemy, lets you work with tables as Python classes. They can save time, but they're built on SQL, and you'll understand them far better because you know SQL first.</p>
</li>
</ul>
<h2>Postgres: the production database</h2>
<p><strong>PostgreSQL</strong>, usually called Postgres, is the database most real services use. It's open source, very reliable, and rich in features. The SQL you've learned works almost unchanged. What it adds:</p>
<table>
<thead>
<tr>
<th></th>
<th>SQLite</th>
<th>Postgres</th>
</tr>
</thead>
<tbody><tr>
<td>Setup</td>
<td>None. It's a file.</td>
<td>A separate server program</td>
</tr>
<tr>
<td>Many users writing at once</td>
<td>One writer at a time</td>
<td>Many, handled well</td>
</tr>
<tr>
<td>Types</td>
<td>Loose by default (opt in to STRICT)</td>
<td>Strict</td>
</tr>
<tr>
<td>Placeholders (Python)</td>
<td>?</td>
<td>%s (with psycopg)</td>
</tr>
<tr>
<td>Auto-numbered ids</td>
<td>INTEGER PRIMARY KEY</td>
<td>GENERATED ... AS IDENTITY</td>
</tr>
<tr>
<td>JSON</td>
<td>Text with JSON functions</td>
<td>jsonb, which can be indexed</td>
</tr>
<tr>
<td>Extras</td>
<td>Few</td>
<td>Many extensions</td>
</tr>
<tr>
<td>Best for</td>
<td>Learning, tests, local apps</td>
<td>Production services</td>
</tr>
</tbody></table>
<p>The extension that matters most to AI engineers is <strong>pgvector</strong>. It lets Postgres store embeddings and find the nearest ones to a query. That means your users, documents, and vectors can live in one database, and you can filter by metadata and search by meaning in the same query. For moderate amounts of data that's often all you need, and we'll compare it with dedicated vector databases in Post 52.</p>
<p>To try Postgres, the easiest route is a container (we'll learn Docker properly in Post 23):</p>
<pre><code class="language-bash">docker run --name pg -e POSTGRES_PASSWORD=secret -p 5432:5432 -d postgres:16
</code></pre>
<p>And from Python, with the <code>psycopg</code> library (<code>pip install "psycopg[binary]"</code>):</p>
<pre><code class="language-python">import os

import psycopg

with psycopg.connect(os.environ["DATABASE_URL"]) as conn:
    row = conn.execute(
        "SELECT id FROM users WHERE email = %s", ("asha@example.com",)
    ).fetchone()
</code></pre>
<p>The connection string comes from an environment variable (Post 12), and looks like <code>postgresql://user:password@localhost:5432/dbname</code>. Notice the <code>%s</code> placeholder.</p>
<p>I should be upfront: my test environment had SQLite but no Postgres, so I ran all the SQL in this post on SQLite, and I'm describing Postgres from knowledge. The Postgres code above is a sketch. Check the current docs for your version.</p>
<p>One more thing for production: opening a database connection is slow and servers cap how many they allow, so real apps use a <strong>connection pool</strong>, a set of ready-to-use connections that requests borrow and return.</p>
<h2>NoSQL and friends</h2>
<p>"NoSQL" is a loose label for databases that aren't relational tables. Each kind has a job:</p>
<table>
<thead>
<tr>
<th>Kind</th>
<th>Example</th>
<th>Good for</th>
</tr>
</thead>
<tbody><tr>
<td>Key-value / in-memory</td>
<td>Redis</td>
<td>Caching model answers (using the hash-key idea from Post 16), rate limits, sessions, job queues</td>
</tr>
<tr>
<td>Document</td>
<td>MongoDB</td>
<td>Nested JSON-like records whose shape varies a lot</td>
</tr>
<tr>
<td>Search engine</td>
<td>Elasticsearch, OpenSearch</td>
<td>Full-text search with ranking</td>
</tr>
<tr>
<td>Graph</td>
<td>Neo4j</td>
<td>Data that's mostly relationships (Post 16's graphs)</td>
</tr>
<tr>
<td>Vector database</td>
<td>Specialized stores, or Postgres with pgvector</td>
<td>Nearest-neighbor search over embeddings</td>
</tr>
</tbody></table>
<p>My honest advice: <strong>start with Postgres</strong> (or SQLite while you're small). It handles an enormous range of needs, including JSON, full-text search, and vectors. Add something else when you have a specific reason: Redis when you need a very fast cache, a dedicated vector database when you outgrow pgvector. Choosing a trendy database before you have a problem is a classic way to create extra work.</p>
<h2>Habits for AI apps</h2>
<p>What should an AI app actually store? A few suggestions:</p>
<ul>
<li><p><strong>The prompts and responses</strong>, so you can debug and later build evals (Post 60). Decide how long you keep them, and treat anything containing personal information carefully.</p>
</li>
<li><p><strong>Which model and version</strong> produced each answer, because behavior changes between versions.</p>
</li>
<li><p><strong>Token counts and cost</strong> for every call, so you can see where the money goes.</p>
</li>
<li><p><strong>Latency</strong>, including time to first token (Post 20).</p>
</li>
<li><p><strong>User feedback</strong>, like thumbs up or down. It's gold for evaluation.</p>
</li>
<li><p><strong>Embeddings, along with the name of the model that made them.</strong> Vectors from different models can't be compared with each other.</p>
</li>
<li><p><strong>Never store secrets in plain text</strong>, and never in a column anyone can read.</p>
</li>
<li><p><strong>Back up your data</strong>, and test restoring from the backup.</p>
</li>
</ul>
<h2>Testing with a database</h2>
<p>From Post 13: tests should be fast and independent. An in-memory SQLite database is ideal, since each test can get a fresh, empty one:</p>
<pre><code class="language-python">import pytest

from chatlog import ChatLog, Message

@pytest.fixture
def log():
    chat_log = ChatLog(":memory:")
    yield chat_log      # the test runs here
    chat_log.close()    # cleanup afterward

def test_failed_exchange_leaves_nothing_behind(log):
    user = log.get_or_create_user("a@example.com")
    chat = log.start_conversation(user)
    with pytest.raises(Exception):
        log.add_exchange(chat, Message("user", "hi", 1), Message("robot", "bad", 1))
    assert log.history(chat) == []
</code></pre>
<p>This uses a fixture with <code>yield</code>, where the code after the <code>yield</code> runs as cleanup. (I couldn't run pytest here, but I ran the same steps by hand against the code below and the behavior is exactly as shown: the failed exchange leaves zero messages.)</p>
<h2>Mini project: a chat log</h2>
<p>Let's put it all together in one class, using the classes and dataclasses from Post 9. Create <code>chatlog.py</code>:</p>
<pre><code class="language-python">import sqlite3
from dataclasses import dataclass
from datetime import datetime, timezone

SCHEMA = """
CREATE TABLE IF NOT EXISTS users (
    id INTEGER PRIMARY KEY,
    email TEXT NOT NULL UNIQUE,
    created_at TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS conversations (
    id INTEGER PRIMARY KEY,
    user_id INTEGER NOT NULL REFERENCES users(id),
    title TEXT,
    created_at TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS messages (
    id INTEGER PRIMARY KEY,
    conversation_id INTEGER NOT NULL REFERENCES conversations(id),
    role TEXT NOT NULL CHECK (role IN ('user', 'assistant', 'system')),
    content TEXT NOT NULL,
    tokens INTEGER,
    created_at TEXT NOT NULL
);

CREATE INDEX IF NOT EXISTS idx_messages_conversation ON messages (conversation_id, id);
"""

def utc_now():
    return datetime.now(timezone.utc).isoformat(timespec="seconds")

@dataclass(frozen=True)
class Message:
    role: str
    content: str
    tokens: int | None = None

class ChatLog:
    def __init__(self, path=":memory:"):
        self.conn = sqlite3.connect(path)
        self.conn.row_factory = sqlite3.Row
        self.conn.execute("PRAGMA foreign_keys = ON")
        self.conn.executescript(SCHEMA)

    def close(self):
        self.conn.close()

    def get_or_create_user(self, email):
        with self.conn:
            self.conn.execute(
                "INSERT INTO users (email, created_at) VALUES (?, ?) ON CONFLICT (email) DO NOTHING",
                (email, utc_now()),
            )
        row = self.conn.execute("SELECT id FROM users WHERE email = ?", (email,)).fetchone()
        return row["id"]

    def start_conversation(self, user_id, title=None):
        with self.conn:
            row = self.conn.execute(
                "INSERT INTO conversations (user_id, title, created_at) VALUES (?, ?, ?) RETURNING id",
                (user_id, title, utc_now()),
            ).fetchone()
        return row["id"]

    def add_exchange(self, conversation_id, question, answer):
        """Save a question and its answer together, or neither."""
        with self.conn:
            for message in (question, answer):
                self.conn.execute(
                    "INSERT INTO messages (conversation_id, role, content, tokens, created_at) "
                    "VALUES (?, ?, ?, ?, ?)",
                    (conversation_id, message.role, message.content, message.tokens, utc_now()),
                )

    def history(self, conversation_id, limit=10):
        """The most recent messages, oldest first."""
        rows = self.conn.execute(
            """
            SELECT role, content, tokens FROM (
                SELECT id, role, content, tokens FROM messages
                WHERE conversation_id = ?
                ORDER BY id DESC
                LIMIT ?
            )
            ORDER BY id
            """,
            (conversation_id, limit),
        ).fetchall()
        return [Message(row["role"], row["content"], row["tokens"]) for row in rows]

    def usage_by_user(self):
        rows = self.conn.execute(
            """
            SELECT u.email, COUNT(m.id) AS messages, COALESCE(SUM(m.tokens), 0) AS tokens
            FROM users u
            LEFT JOIN conversations c ON c.user_id = u.id
            LEFT JOIN messages m ON m.conversation_id = c.id
            GROUP BY u.id
            ORDER BY tokens DESC, u.email
            """
        ).fetchall()
        return [(row["email"], row["messages"], row["tokens"]) for row in rows]

    def search(self, word, limit=5):
        rows = self.conn.execute(
            "SELECT conversation_id, content FROM messages WHERE content LIKE ? ORDER BY id LIMIT ?",
            (f"%{word}%", limit),
        ).fetchall()
        return [(row["conversation_id"], row["content"]) for row in rows]

if __name__ == "__main__":
    log = ChatLog()
    asha = log.get_or_create_user("asha@example.com")
    ravi = log.get_or_create_user("ravi@example.com")
    log.get_or_create_user("meera@example.com")
    assert log.get_or_create_user("asha@example.com") == asha    # no duplicate user

    chat = log.start_conversation(asha, "Python help")
    log.add_exchange(
        chat,
        Message("user", "How do generators work?", 6),
        Message("assistant", "A generator yields values lazily.", 8),
    )
    log.add_exchange(
        chat,
        Message("user", "Show an example", 4),
        Message("assistant", "Here is a small example.", 7),
    )
    other = log.start_conversation(ravi, "Resume review")
    log.add_exchange(
        other,
        Message("user", "Review my resume", 4),
        Message("assistant", "Lead with your strongest project.", 9),
    )

    print("Last 3 messages in the first chat:")
    for message in log.history(chat, limit=3):
        print(f"  {message.role}: {message.content}")

    print("\nUsage by user:")
    for email, messages, tokens in log.usage_by_user():
        print(f"  {email:&lt;20}{messages} messages, {tokens} tokens")

    print("\nSearch for 'generator':", log.search("generator"))

    try:
        log.add_exchange(
            chat,
            Message("user", "This will fail", 5),
            Message("robot", "bad role", 5),
        )
    except sqlite3.IntegrityError as error:
        print("\nRejected:", error)
    print("Messages in the first chat:", len(log.history(chat, limit=100)))

    plan = log.conn.execute(
        "EXPLAIN QUERY PLAN SELECT id FROM messages WHERE conversation_id = 1 ORDER BY id DESC LIMIT 3"
    ).fetchall()
    print("Plan:", [row[3] for row in plan])
</code></pre>
<p>Run it, and you should see:</p>
<pre><code class="language-sql">Last 3 messages in the first chat:
  assistant: A generator yields values lazily.
  user: Show an example
  assistant: Here is a small example.

Usage by user:
  asha@example.com    4 messages, 25 tokens
  ravi@example.com    2 messages, 13 tokens
  meera@example.com   0 messages, 0 tokens

Search for 'generator': [(1, 'How do generators work?'), (1, 'A generator yields values lazily.')]

Rejected: CHECK constraint failed: role IN ('user', 'assistant', 'system')
Messages in the first chat: 4
Plan: ['SEARCH messages USING COVERING INDEX idx_messages_conversation (conversation_id=?)']
</code></pre>
<p>Here's what to notice:</p>
<ul>
<li><p><code>get_or_create_user</code> uses <code>ON CONFLICT ... DO NOTHING</code>, so asking for the same email twice returns the same user instead of crashing or creating a duplicate. (That syntax works in both SQLite and Postgres.)</p>
</li>
<li><p><code>add_exchange</code> <strong>is one transaction.</strong> The deliberately bad exchange at the end has a valid user message followed by an assistant message with a role of <code>"robot"</code>. The second insert fails the <code>CHECK</code> constraint, and the whole transaction rolls back, so the first message isn't saved either. The chat still has 4 messages.</p>
</li>
<li><p><code>history</code> <strong>returns only the most recent messages</strong>, with a limit, then reorders them oldest-first. That's the same "keep only the last few messages" idea as <code>deque(maxlen=...)</code> from Post 16, and it keeps the cost of resending history under control (Post 18).</p>
</li>
<li><p><strong>Placeholders everywhere.</strong> Even the search term goes through <code>?</code>. (One caveat: <code>%</code> and <code>_</code> typed by a user are wildcards inside <code>LIKE</code>. For a real search feature, use full-text search.)</p>
</li>
<li><p><strong>The query plan shows the index at work.</strong> It says <code>COVERING INDEX</code>, meaning the index alone had everything the query needed, so the database never even touched the table.</p>
</li>
<li><p><strong>The database persists.</strong> Pass a filename instead of <code>":memory:"</code>, such as <code>ChatLog("app.db")</code>, close the log, open it again, and the data is still there. I checked that too.</p>
</li>
</ul>
<h3>Make it yours</h3>
<ol>
<li><p>Add full-text search. SQLite has a feature called FTS5 for this: look up <code>CREATE VIRTUAL TABLE ... USING fts5</code>.</p>
</li>
<li><p>Add a <code>deleted_at</code> column for soft deletes (marking rows as deleted instead of removing them), and update the queries to ignore them.</p>
</li>
<li><p>Add a <code>quotas</code> table and a method that uses an atomic update like the one above.</p>
</li>
<li><p>Write a pytest file using the fixture from earlier, with at least five tests.</p>
</li>
<li><p>Add a <code>feedback</code> column and a method that returns the average rating per day.</p>
</li>
<li><p>Port it to Postgres: switch to <code>psycopg</code>, change <code>?</code> to <code>%s</code>, and use <code>GENERATED ... AS IDENTITY</code> for the ids.</p>
</li>
<li><p>Run <code>EXPLAIN QUERY PLAN</code> on each query, and find one that scans the whole table.</p>
</li>
</ol>
<h2>Practice</h2>
<p>About 20 minutes of real effort on each before asking for help.</p>
<ol>
<li><p>Write SQL to find: users with no conversations, the average token count per message role, and the three conversations with the most messages.</p>
</li>
<li><p>Turn off foreign keys, insert an orphan row, then turn them on and see what changes.</p>
</li>
<li><p>Demonstrate SQL injection against a table of your own, then fix it with placeholders.</p>
</li>
<li><p>Fill a table with 200,000 rows and time the same query with and without an index. Look at the plan each time.</p>
</li>
<li><p>Show with <code>EXPLAIN QUERY PLAN</code> that wrapping an indexed column in <code>lower()</code> stops the index being used, then fix it with an expression index.</p>
</li>
<li><p>Write a transaction that inserts two related rows, make the second one fail, and prove that neither row remains.</p>
</li>
<li><p>In your own words, explain why "read the value in Python, add one, write it back" can lose an update, and how a single <code>UPDATE</code> avoids it.</p>
</li>
<li><p>Design a schema for a RAG app (documents, chunks, and embedding metadata). Which columns would you index, and why?</p>
</li>
</ol>
<p>Then ask an AI tutor to review your answers, and don't let it rewrite them.</p>
<h2>What to take away</h2>
<ul>
<li><p>A database gives you structure, fast search, concurrency, durability, and rules that protect your data. Files give you none of those.</p>
</li>
<li><p>Tables hold rows. Primary keys identify rows, and foreign keys link them. Each fact lives in one place.</p>
</li>
<li><p>Constraints (<code>NOT NULL</code>, <code>UNIQUE</code>, <code>CHECK</code>, foreign keys) stop bad data at the door. In SQLite, turn foreign keys on for every connection, and consider <code>STRICT</code> tables.</p>
</li>
<li><p><strong>Always use placeholders.</strong> Building SQL with f-strings is how SQL injection happens.</p>
</li>
<li><p>Joins combine tables, <code>LEFT JOIN</code> keeps unmatched rows, <code>GROUP BY</code> summarizes, and window functions rank within groups. <code>NULL</code> needs <code>IS NULL</code>.</p>
</li>
<li><p>An index turns an O(n) scan into an O(log n) search. Check with <code>EXPLAIN</code>, index what you really query, and remember that indexes slow writes.</p>
</li>
<li><p>A transaction is all or nothing. Don't hold one open during slow calls. Let the database do read-and-change in one atomic statement.</p>
</li>
<li><p>Start with Postgres (or SQLite while small). Add Redis, a search engine, or a vector database only when you have a specific reason.</p>
</li>
<li><p>For AI apps, store prompts, responses, model versions, token counts, latency, and feedback.</p>
</li>
</ul>
<hr />
<p><strong>Previous:</strong> Post 20, <em>Networking &amp; the Web</em></p>
<p><strong>Next in the series:</strong> Post 22, <em>Git &amp; GitHub Workflows</em>, where we cover branching, pull requests, and code review, so your work is safe, shareable, and easy to collaborate on.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/databases-in-depth-sql-indexes-transactions-and-when-to-use-what-pklyn?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Why Your Cron Job Fires at the Wrong Time: IST, UTC and the Midnight Day Shift]]></title><description><![CDATA[I scheduled a job for 9 PM IST and it ran at 2:30 AM. Cron uses the server's timezone, and most servers run UTC. Here's how the offset, the missing 30…]]></description><link>https://blog.zyvop.com/why-your-cron-job-fires-at-the-wrong-time-ist-utc-and-the-midnight-day-shift-z9s3h</link><guid isPermaLink="true">https://blog.zyvop.com/why-your-cron-job-fires-at-the-wrong-time-ist-utc-and-the-midnight-day-shift-z9s3h</guid><category><![CDATA[cron]]></category><category><![CDATA[Devops]]></category><category><![CDATA[Linux]]></category><category><![CDATA[Python]]></category><category><![CDATA[Timezones]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Thu, 05 Nov 2026 05:57:55 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/1960b1ba-2a86-4485-872a-4883615b9890.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>You write <code>0 21 * * *</code> to run a report at 9 PM. It runs, just not when you expected: at 2:30 AM India time. Nothing is broken. Cron did exactly what you asked, in a timezone you didn't know it was using.</p>
<p>This post explains why that happens, the two mistakes that catch a lot of people in India, and four ways to fix it, including a small tested Python helper.</p>
<h2>The cause: cron uses the server's timezone</h2>
<p>Cron has no idea where you live. It reads the clock of the machine it runs on, and most cloud servers and VPS images default to UTC. Check yours:</p>
<pre><code class="language-bash">timedatectl
</code></pre>
<p>Look for the <code>Time zone</code> line. If it says <code>Etc/UTC</code> or <code>UTC</code>, every cron expression on that machine is in UTC.</p>
<p>IST is UTC+5:30 and has no daylight saving. So <code>0 21 * * *</code> means 21:00 UTC, which is 02:30 IST the next day.</p>
<h2>Mistake 1: forgetting the half hour</h2>
<p>Most timezones differ from UTC by whole hours, so it's easy to subtract 5 instead of 5:30. 9 PM IST is 15:30 UTC, so the correct line is <code>30 15 * * *</code>. Write <code>0 15 * * *</code> and the job fires at 8:30 PM IST.</p>
<p>The conversion is always UTC = IST − 5:30:</p>
<table>
<thead>
<tr>
<th>IST</th>
<th>UTC</th>
<th>UTC day</th>
</tr>
</thead>
<tbody><tr>
<td>00:00</td>
<td>18:30</td>
<td>previous day</td>
</tr>
<tr>
<td>01:00</td>
<td>19:30</td>
<td>previous day</td>
</tr>
<tr>
<td>05:29</td>
<td>23:59</td>
<td>previous day</td>
</tr>
<tr>
<td>05:30</td>
<td>00:00</td>
<td>same day</td>
</tr>
<tr>
<td>09:00</td>
<td>03:30</td>
<td>same day</td>
</tr>
<tr>
<td>21:00</td>
<td>15:30</td>
<td>same day</td>
</tr>
<tr>
<td>23:30</td>
<td>18:00</td>
<td>same day</td>
</tr>
</tbody></table>
<h2>Mistake 2: the midnight day shift</h2>
<p>Anything between 00:00 and 05:29 IST lands on the previous day in UTC, so the day fields change too.</p>
<p>Take "every Monday at 1 AM IST". The server sees that moment as Sunday 19:30 UTC, so the line is <code>30 19 * * 0</code>. If you keep the 1 in the weekday field (<code>30 19 * * 1</code>), the job fires on Monday 19:30 UTC, which is Tuesday 1 AM IST, a full day late.</p>
<p>Day of the month is worse. "The 1st at 2 AM IST" is 20:30 UTC on the last day of the previous month, and that day is 28, 29, 30 or 31. Plain cron can't express it, so use one of the timezone-aware options below.</p>
<h2>Choosing a fix</h2>
<p><img src="https://mermaid.ink/img/pako:eNplkU1v2zAMhv8Kx7N86NaTDxuapOl62AYsyYAiyoGxmZiLLBqSEtcN8t8H2xsMdDeReD8eQlcstGTM8eC0LSoKCdYL6wEAHrYWX_QMLfkEBL91D9Q_DvLKJTyv1pCkZos7yLLPMLtaXJC4DjRAy3xynQHxQPCmnqGVVOk5QUmdk2OVINJF_PGLxdvYNoMsA4svHC0OgfM-UKH7R5AUTswNNI7EQxHU_2f-rn-9j1uLm8hAA2IPkFFLgSEWFZdnx8HibrTO3_cuthZ_NEnUw10OhfoLh6F9s56_d02F4_pxGJZTwsccYhcT1-VAMtWOyqdJ-Smf4EB8lJL72wNQ01jcocGaQ01SYn7FVHHdf1rJBzq7hGbc_KIgtHcce81BfVpSLa7DHDNqGsfZyGJg5sSfvlGxGual-mTA4oqPyrB5tmjgp-41qYGv7C6cpCADD0HIGYjkYxY5yAHNULKSt57l7r55xdvN4P44V6cBc_zQVpIYb38AOQnCow?type=png" alt="Mermaid Diagram" /></p>
<h2>Option 1: convert to UTC (works everywhere)</h2>
<p>Keep the server on UTC and do the arithmetic once, with code instead of by hand. This helper uses only the standard library (<code>zoneinfo</code> needs Python 3.9 or newer):</p>
<pre><code class="language-python">from datetime import datetime, timedelta, timezone
from zoneinfo import ZoneInfo

UTC = timezone.utc
DAYS = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"]   # cron numbering: 0 = Sunday

def _zone(tz):
    zone = ZoneInfo(tz)
    jan = datetime(2026, 1, 4, 12, tzinfo=zone).utcoffset()
    jul = datetime(2026, 7, 4, 12, tzinfo=zone).utcoffset()
    if jan != jul:
        raise ValueError(f"{tz} uses daylight saving; a fixed UTC cron line would drift. "
                         "Use a timezone-aware scheduler instead.")
    return zone

def _day(dow):
    # Any real week works for a zone without DST. 4 Jan 2026 is a Sunday.
    return datetime(2026, 1, 4) + timedelta(days=0 if dow in (None, "*") else int(dow))

def to_utc_cron(hour, minute, dow=None, tz="Asia/Kolkata"):
    """Local wall-clock time -&gt; UTC cron expression (daily, or weekly if dow is given)."""
    local = _day(dow).replace(hour=hour, minute=minute, tzinfo=_zone(tz))
    utc = local.astimezone(UTC)
    utc_dow = "*" if dow is None else (utc.weekday() + 1) % 7   # Python Mon=0 -&gt; cron Mon=1
    return f"{utc.minute} {utc.hour} * * {utc_dow}"

def explain(cron, tz="Asia/Kolkata"):
    """UTC cron expression -&gt; what it means in the local zone."""
    fields = cron.split()
    if len(fields) != 5:
        raise ValueError("expected 5 cron fields")
    minute, hour, dom, month, dow = fields
    if not (minute.isdigit() and hour.isdigit() and dom == month == "*"
            and (dow == "*" or (dow.isdigit() and int(dow) &lt;= 7))):
        raise ValueError("only simple daily or weekly lines are supported "
                         "(one minute, one hour, optional single weekday)")
    utc = _day(dow).replace(hour=int(hour), minute=int(minute), tzinfo=UTC)
    local = utc.astimezone(_zone(tz))
    when = "daily" if dow == "*" else DAYS[(local.weekday() + 1) % 7]
    return f"{cron:&lt;14} = {when} {local:%H:%M} {local.tzname()}"
</code></pre>
<p>Try it:</p>
<pre><code class="language-python">&gt;&gt;&gt; to_utc_cron(21, 0)             # every day at 9 PM IST
'30 15 * * *'
&gt;&gt;&gt; to_utc_cron(1, 0, dow=1)       # every Monday at 1 AM IST
'30 19 * * 0'
&gt;&gt;&gt; explain("0 21 * * *")          # the line from the intro
'0 21 * * *     = daily 02:30 IST'
</code></pre>
<p><code>explain</code> is the useful half: paste a daily or weekly line and see what it means in IST. It handles one minute, one hour and an optional single weekday, and raises an error for anything fancier (ranges, steps, day of month) instead of guessing. Run it on your existing jobs before you trust them.</p>
<p>This only works for zones without daylight saving, like IST, and the helper refuses zones that have it. For a zone that changes its clocks, a fixed UTC expression is off by an hour for part of the year, so use Option 2 or 3 there.</p>
<h2>Option 2: a systemd timer with an explicit timezone</h2>
<p>systemd timers accept a timezone name inside the schedule, so you can write the time you mean:</p>
<pre><code class="language-toml"># /etc/systemd/system/report.timer
[Unit]
Description=Daily report at 9 PM IST

[Timer]
OnCalendar=*-*-* 21:00:00 Asia/Kolkata
Persistent=true

[Install]
WantedBy=timers.target
</code></pre>
<pre><code class="language-toml"># /etc/systemd/system/report.service
[Unit]
Description=Daily report

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /opt/report/run.py
</code></pre>
<p>Enable it and check the schedule:</p>
<pre><code class="language-bash">sudo systemctl daemon-reload
sudo systemctl enable --now report.timer
systemd-analyze calendar "*-*-* 21:00:00 Asia/Kolkata"
systemctl list-timers report.timer
</code></pre>
<p><code>systemd-analyze calendar</code> prints the next run in your timezone and in UTC, so you can confirm the conversion without waiting. <code>Persistent=true</code> also runs a missed job after the server was down. Run <code>systemd-analyze calendar</code> first: on a very old systemd it will reject the timezone name, which is better to learn now than at 9 PM.</p>
<p>The monthly case from earlier is easy here. "The 1st at 2 AM IST" is just <code>OnCalendar=*-*-01 02:00:00 Asia/Kolkata</code>, and systemd's <code>~</code> syntax covers the last day of the month (<code>*-*~1</code>), which cron can't express.</p>
<p>A note on <code>CRON_TZ</code>: you will see advice to add <code>CRON_TZ=Asia/Kolkata</code> to your crontab. That works in cronie, the cron used on RHEL, Fedora, AlmaLinux and Rocky. Debian and Ubuntu's cron runs jobs in a single timezone, so the line can be silently ignored. Check with <code>man 5 crontab</code> on your machine, and test it before you rely on it.</p>
<h2>Option 3: let the app schedule itself</h2>
<p>If the job lives inside a long-running app, give its scheduler the zone. Most scheduling libraries take a timezone argument; pass <code>Asia/Kolkata</code> there and skip the conversion. Check your library's documentation for the exact name and version-specific syntax.</p>
<h2>Option 4: change the server's timezone (usually don't)</h2>
<p><code>sudo timedatectl set-timezone Asia/Kolkata</code> makes cron expressions mean IST, but it also changes every log timestamp and every other job on the machine, and anyone who later reads your logs will assume UTC. If you do it, restart cron afterwards (<code>sudo systemctl restart cron</code>, or <code>crond</code> on the RHEL family). I'd keep servers on UTC.</p>
<h2>Verify, don't assume</h2>
<p>Run these on the server:</p>
<pre><code class="language-bash">timedatectl                               # which timezone is this server in?
date; date -u; TZ=Asia/Kolkata date      # the same instant, three ways
journalctl -u cron --since today          # when did cron actually start it? (Debian/Ubuntu)
</code></pre>
<p>On older Debian and Ubuntu setups the same lines are in <code>/var/log/syslog</code>, and on RHEL-family systems the cron log is <code>/var/log/cron</code> (the service is <code>crond</code> there). Better still, make every job log its own start time with the zone, so a wrong schedule is obvious. Put this line in the script itself:</p>
<pre><code class="language-bash">echo "started $(date -u +%FT%TZ) / $(TZ=Asia/Kolkata date +%FT%T%z)" &gt;&gt; /var/log/report.log
</code></pre>
<p>Don't paste it straight into a crontab line: <code>%</code> has a special meaning there and must be written as <code>\%</code>.</p>
<h2>Two more traps</h2>
<p><code>cron.daily</code> <strong>isn't at midnight.</strong> On a stock Debian or Ubuntu server, scripts in <code>/etc/cron.daily</code> typically start somewhere between about 06:25 and 09:00 server time, depending on whether anacron is installed. On a UTC server that's late morning or early afternoon in India, not midnight. If you need a specific time, write your own cron line or timer.</p>
<p><strong>Naive datetimes in Python.</strong> <code>datetime.now()</code> returns the server's local time with no timezone attached, so the same code gives different answers on your laptop and on the server:</p>
<pre><code class="language-python">from datetime import datetime
from zoneinfo import ZoneInfo

datetime.now()                          # naive: server's timezone, UTC on most VPS
datetime.now(ZoneInfo("Asia/Kolkata"))  # explicit: correct anywhere
</code></pre>
<p>On Windows, <code>zoneinfo</code> needs the timezone database from <code>pip install tzdata</code>. Store timestamps in UTC and convert to IST only when you show them to a person.</p>
<h2>Cheat sheet</h2>
<ul>
<li><p>Cron uses the server's timezone, and most servers are UTC.</p>
</li>
<li><p>IST = UTC + 5:30. Subtract 5:30 to get UTC, not 5.</p>
</li>
<li><p>Anything from 00:00 to 05:29 IST moves to the previous UTC day, including the weekday and the date.</p>
</li>
<li><p>For daily and weekly jobs, convert to UTC with code. For monthly jobs or zones with daylight saving, use a systemd timer or an app scheduler with an explicit timezone.</p>
</li>
<li><p>Don't rely on <code>CRON_TZ</code> on Debian or Ubuntu.</p>
</li>
<li><p>Log start times in UTC and IST, and check the first run instead of assuming.</p>
</li>
</ul>
<hr />
<p><em>Published via <a href="https://zyvop.com/why-your-cron-job-fires-at-the-wrong-time-ist-utc-and-the-midnight-day-shift-z9s3h?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Building a /deploy Slack Bot in Python (and the Gotchas the Docs Skip)]]></title><description><![CDATA[Build a /deploy Slack bot using Python and Slack Bolt. Learn to handle slash commands, approval buttons, Socket Mode, Slack’s three-second acknowledgement…]]></description><link>https://blog.zyvop.com/building-a-deploy-slack-bot-in-python-and-the-gotchas-the-docs-skip-hp7zm</link><guid isPermaLink="true">https://blog.zyvop.com/building-a-deploy-slack-bot-in-python-and-the-gotchas-the-docs-skip-hp7zm</guid><category><![CDATA[automation]]></category><category><![CDATA[Devops]]></category><category><![CDATA[Python]]></category><category><![CDATA[slack]]></category><category><![CDATA[Slack Bolt]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Wed, 04 Nov 2026 07:59:39 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/2aaa7cbf-1739-4a0f-9cd4-3f3acb6e60d2.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Every deploy at my last team started the same way: someone typed “deploying api to staging, ok?” in a channel, waited for a thumbs-up emoji, then went off to run it by hand. Nobody could say afterwards who had approved what, and twice two people deployed at the same time.</p>
<p>So I built a <code>/deploy</code> Slack command. You type <code>/deploy api staging</code>, the bot posts a request with <strong>Approve</strong> and <strong>Reject</strong> buttons, a teammate clicks one, and the message updates to show what happened and who signed off. The whole bot is roughly 140 lines of Python.</p>
<p>This post builds it step by step. It also covers the things the Slack docs don’t make obvious: the 3-second acknowledgement rule, an easy-to-miss setting that leaves your buttons doing nothing, and how to stop a double click from deploying twice.</p>
<h2>What we’re building</h2>
<p>The flow looks like this:</p>
<ol>
<li><p>Someone runs <code>/deploy &lt;service&gt; &lt;env&gt;</code>.</p>
</li>
<li><p>The bot posts a message in the channel with <strong>Approve</strong> and <strong>Reject</strong> buttons.</p>
</li>
<li><p>A teammate clicks a button. Nobody can approve their own request.</p>
</li>
<li><p>The message is replaced with the outcome, so the buttons disappear.</p>
</li>
</ol>
<p>We’ll use Bolt, Slack’s official framework for Python, together with Socket Mode, so you can run everything from your laptop without a public URL. The deploy itself is a stand-in function; swap in your own CI trigger or script.</p>
<h2>Prerequisites</h2>
<ul>
<li><p>Python 3.9 or newer</p>
</li>
<li><p>A Slack workspace where you’re allowed to install apps (a free personal workspace is fine for testing)</p>
</li>
<li><p>Basic Python; no prior Slack experience needed</p>
</li>
</ul>
<h2>Step 1: Create the Slack app</h2>
<p>Go to api.slack.com/apps and choose <strong>Create New App → From scratch</strong>. Then work through these settings:</p>
<ol>
<li><p><strong>Socket Mode:</strong> Turn it on and generate an app-level token with the <code>connections:write</code> scope. This is your <code>xapp-</code> token.</p>
</li>
<li><p><strong>Slash Commands:</strong> Create a command called <code>/deploy</code>. With Socket Mode on, you don’t need a request URL.</p>
</li>
<li><p><strong>Interactivity &amp; Shortcuts:</strong> Turn it on. This is the one people miss. If you skip it, the buttons render fine but clicking them does nothing, and there’s no helpful error.</p>
</li>
<li><p><strong>OAuth &amp; Permissions:</strong> Add the <code>commands</code> bot scope, install the app to your workspace, and copy the <strong>Bot User OAuth Token</strong>. This is your <code>xoxb-</code> token.</p>
</li>
</ol>
<p>Slack renames and moves settings from time to time, so if a label doesn’t match, check the current docs for the same setting.</p>
<p>You now have two tokens, and mixing them up is a classic first-run error:</p>
<table>
<thead>
<tr>
<th>Token</th>
<th>Starts with</th>
<th>What it’s for</th>
</tr>
</thead>
<tbody><tr>
<td>Bot token</td>
<td>xoxb-</td>
<td>Calling Slack’s Web API as your bot</td>
</tr>
<tr>
<td>App-level token</td>
<td>xapp-</td>
<td>Opening the Socket Mode connection</td>
</tr>
</tbody></table>
<h2>Step 2: Set up the project</h2>
<pre><code class="language-bash">mkdir deploy-bot &amp;&amp; cd deploy-bot
python -m venv .venv &amp;&amp; source .venv/bin/activate
pip install slack-bolt

export SLACK_BOT_TOKEN=xoxb-your-token
export SLACK_APP_TOKEN=xapp-your-token
</code></pre>
<p>Keep tokens out of your repo. For anything beyond a demo, load them from a secrets manager or your platform’s environment settings.</p>
<h2>Step 3: The slash command and the 3-second rule</h2>
<p>Create <code>bot.py</code>, starting with the imports and the app:</p>
<pre><code class="language-python">import json
import os
import re
import time

from slack_bolt import App
from slack_bolt.adapter.socket_mode import SocketModeHandler

app = App(token=os.environ["SLACK_BOT_TOKEN"])

# Input rules: adjust to match your own services and environments
SERVICE_RE = re.compile(r"[a-z0-9][a-z0-9-]{0,40}")  # e.g. api, billing-worker
ALLOWED_ENVS = {"staging", "production"}
</code></pre>
<p>Now the command handler:</p>
<pre><code class="language-python">@app.command("/deploy")
def handle_deploy(ack, command, respond):
    ack()  # Slack expects an acknowledgement within 3 seconds, so do this first

    parts = command["text"].split()
    if len(parts) != 2:
        respond("Usage: `/deploy &lt;service&gt; &lt;env&gt;`, e.g. `/deploy api staging`")
        return

    service, env = parts
    if not SERVICE_RE.fullmatch(service) or env not in ALLOWED_ENVS:
        respond(
            "Invalid service or environment. Services use lowercase letters, "
            "digits and dashes; environment must be one of: "
            + ", ".join(sorted(ALLOWED_ENVS))
        )
        return

    requester = command["user_id"]
    respond(
        text=f"Deploy request: {service} -&gt; {env}",  # fallback for notifications
        blocks=request_blocks(service, env, requester),
        response_type="in_channel",
    )
</code></pre>
<p>The most important line is <code>ack()</code>. Slack gives you about three seconds to acknowledge a slash command, a button click, or any other interaction. If you miss that window, the user sees an error like “operation_timeout”, even if your code runs perfectly a moment later. So the rule is simple: acknowledge first, then do the real work.</p>
<p><code>respond()</code> posts through the command’s response URL, and <code>response_type="in_channel"</code> makes the message visible to everyone in the channel instead of just the person who typed the command. Because we post through the response URL, this demo doesn’t need the <code>chat:write</code> scope.</p>
<p>The <code>text</code> argument is easy to skip, but don’t. It’s the plain-text fallback used in notifications and by screen readers when you send blocks.</p>
<p>Notice the check before anything is posted. Whatever someone types after <code>/deploy</code> lands in a message others read and, later, in your deploy script, so I validate it at the door: a strict pattern for the service name and an allowlist for environments. Anything else is rejected, which is easier to get right than cleaning up bad input. Adjust <code>ALLOWED_ENVS</code> and the pattern to fit your setup.</p>
<h2>Step 4: Build the buttons with Block Kit</h2>
<p>Slack messages with rich layouts are described as “blocks”, which are plain Python dicts. Add these two helpers above the command handler:</p>
<pre><code class="language-python">def request_blocks(service: str, env: str, requester: str) -&gt; list[dict]:
    """The message with Approve / Reject buttons."""
    value = json.dumps({"service": service, "env": env, "requester": requester})
    return [
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": f"*Deploy request*\n&lt;@{requester}&gt; wants to deploy `{service}` to `{env}`",
            },
        },
        {
            "type": "actions",
            "elements": [
                {
                    "type": "button",
                    "text": {"type": "plain_text", "text": "Approve"},
                    "style": "primary",
                    "action_id": "deploy_approve",
                    "value": value,
                },
                {
                    "type": "button",
                    "text": {"type": "plain_text", "text": "Reject"},
                    "style": "danger",
                    "action_id": "deploy_reject",
                    "value": value,
                },
            ],
        },
    ]

def status_blocks(text: str) -&gt; list[dict]:
    """A buttons-free message, so the same request can't be clicked twice."""
    return [{"type": "section", "text": {"type": "mrkdwn", "text": text}}]
</code></pre>
<p>A few things are worth noticing:</p>
<ul>
<li><p><code>action_id</code> is how Slack routes a click to the right handler. We’ll use <code>deploy_approve</code> and <code>deploy_reject</code> in a moment.</p>
</li>
<li><p><code>value</code> is a string attached to the button and sent back to you on click. Packing the request details into JSON means this demo needs no database. Button values have a size limit (currently 2,000 characters), so store IDs rather than big payloads in real apps.</p>
</li>
<li><p><code>&lt;@{requester}&gt;</code> is Slack’s syntax for mentioning a user by ID, which renders as their name.</p>
</li>
<li><p><code>status_blocks</code> produces a message with no buttons. We’ll use it to replace the original, which is the double-click protection.</p>
</li>
</ul>
<h2>Step 5: Handle the clicks</h2>
<p>First, a stand-in for the real deployment:</p>
<pre><code class="language-python">def run_deploy(service: str, env: str) -&gt; bool:
    """Stand-in for the real work (CI trigger, kubectl, etc.)."""
    time.sleep(3)
    return True
</code></pre>
<p>Then the approve handler:</p>
<pre><code class="language-python">@app.action("deploy_approve")
def handle_approve(ack, body, action, respond):
    ack()
    data = json.loads(action["value"])
    approver = body["user"]["id"]

    if approver == data["requester"]:
        respond(
            text="You can't approve your own deploy.",
            response_type="ephemeral",
            replace_original=False,
        )
        return

    # Replace the message right away: buttons disappear, so no double clicks
    respond(
        text="Deploying...",
        blocks=status_blocks(
            f":hourglass_flowing_sand: &lt;@{approver}&gt; approved. "
            f"Deploying `{data['service']}` to `{data['env']}`..."
        ),
        replace_original=True,
    )

    ok = run_deploy(data["service"], data["env"])

    result = ":white_check_mark: Deployed" if ok else ":x: Deploy failed"
    respond(
        text=result,
        blocks=status_blocks(
            f"{result}: `{data['service']}` to `{data['env']}` "
            f"(requested by &lt;@{data['requester']}&gt;, approved by &lt;@{approver}&gt;)"
        ),
        replace_original=True,
    )
</code></pre>
<p>Three details do the real work here:</p>
<ol>
<li><p><code>ack()</code> <strong>comes first, again.</strong> Button clicks follow the same 3-second rule as slash commands.</p>
</li>
<li><p><strong>The self-approval check.</strong> Slack tells us who clicked (<code>body["user"]["id"]</code>), and we compare that with the requester stored in the button value. The warning is <code>ephemeral</code>, meaning only the person who clicked sees it, and <code>replace_original=False</code> leaves the request message untouched.</p>
</li>
<li><p><strong>Replace the message before the slow work.</strong> The moment the approval is accepted, we swap the buttons for a “Deploying…” status. Anyone who clicks late finds no button to press. Once the deploy finishes, we replace the message again with the result.</p>
</li>
</ol>
<p>The reject handler is shorter:</p>
<pre><code class="language-python">@app.action("deploy_reject")
def handle_reject(ack, body, action, respond):
    ack()
    data = json.loads(action["value"])
    rejecter = body["user"]["id"]
    respond(
        text="Deploy rejected",
        blocks=status_blocks(
            f":no_entry: &lt;@{rejecter}&gt; rejected the deploy of "
            f"`{data['service']}` to `{data['env']}`."
        ),
        replace_original=True,
    )
</code></pre>
<p>Finally, start the app at the bottom of the file:</p>
<pre><code class="language-python">if __name__ == "__main__":
    SocketModeHandler(app, os.environ["SLACK_APP_TOKEN"]).start()
</code></pre>
<h2>Step 6: Run it</h2>
<pre><code class="language-bash">python bot.py
</code></pre>
<p>Then try the flow in Slack:</p>
<ol>
<li><p>Type <code>/deploy api staging</code> in a channel.</p>
</li>
<li><p>Try clicking <strong>Approve</strong> on your own request. You should get a private “you can’t approve your own deploy” message.</p>
</li>
<li><p>Ask a teammate, or use a second test account, to click <strong>Approve</strong>. The message should change to “Deploying…” and then to “Deployed”.</p>
</li>
<li><p>Run the command again and click <strong>Reject</strong> to check the other path.</p>
</li>
<li><p>Try some bad input, like <code>/deploy API prod</code> (uppercase service, environment not on the list). You should get a private error message and no request in the channel.</p>
</li>
</ol>
<p>If the slash command works but the buttons do nothing, go back and check that <strong>Interactivity &amp; Shortcuts</strong> is on.</p>
<h2>Gotchas worth knowing about</h2>
<ol>
<li><p><strong>The Interactivity toggle.</strong> Covered above, but it’s the number one reason a button bot “doesn’t work”. The buttons appear, the clicks vanish, and nothing logs an error.</p>
</li>
<li><p><strong>Two different tokens.</strong> The <code>xoxb-</code> bot token and the <code>xapp-</code> app-level token do different jobs. If startup fails with an authentication error, check you haven’t swapped them.</p>
</li>
<li><p><strong>Acknowledging is not the same as finishing.</strong> <code>ack()</code> only tells Slack “I got it”. Your real work can continue afterwards, and you report the result with <code>respond()</code> or by posting a new message.</p>
</li>
<li><p><strong>Retries over HTTP.</strong> If you later receive events over HTTP instead of Socket Mode, Slack retries deliveries it thinks you missed. Look for the <code>X-Slack-Retry-Num</code> header, and make handlers idempotent so a retry can’t trigger a second deploy.</p>
</li>
<li><p><strong>Signature verification.</strong> With HTTP endpoints, Bolt verifies Slack’s request signatures using your signing secret. With Socket Mode, the connection is authenticated by the app-level token, so you don’t verify requests yourself. If you ever write your own HTTP endpoint without Bolt, verifying signatures is not optional.</p>
</li>
<li><p><strong>Anyone who can see the message can click it.</strong> A button doesn’t know who is “allowed” to approve. Checking that is your job. We’ll add an approver list below.</p>
</li>
</ol>
<h2>Socket Mode or HTTP?</h2>
<p>Socket Mode keeps an outbound WebSocket connection open to Slack, so there’s no public URL, no tunnel, and no certificate to manage. That makes it ideal for development and internal tools like this one.</p>
<p>HTTP mode needs a public HTTPS endpoint, but it’s stateless, which makes it easier to scale across several instances behind a load balancer. Slack’s documentation also notes that apps listed in the Slack Marketplace must use HTTP rather than Socket Mode, so check their current guidance if you plan to distribute your app.</p>
<p>Because Bolt hides most of the difference, switching later usually means changing how the app starts, not rewriting your handlers.</p>
<h2>What I’d change before using this for real</h2>
<p>This version is good for a demo, and not yet safe for production. Here’s what I’d add, roughly in priority order.</p>
<p><strong>1. Restrict who can approve.</strong> Right now any teammate who can see the message can approve. A simple allowlist is a good first step:</p>
<pre><code class="language-python">APPROVERS = set(os.environ.get("DEPLOY_APPROVERS", "").split(","))

# inside handle_approve, after ack():
if approver not in APPROVERS:
    respond(
        text="You are not allowed to approve deploys.",
        response_type="ephemeral",
        replace_original=False,
    )
    return
</code></pre>
<p><strong>2. Make double-approval impossible, not just unlikely.</strong> Replacing the message removes the buttons, but two clicks that land at nearly the same instant could both get past the self-approval check before the first replacement happens. The robust fix is to store each request with an ID and a status in a database or Redis, and flip it from <code>pending</code> to <code>approved</code> with an atomic compare-and-set. Only the click that wins the update gets to deploy.</p>
<p><strong>3. Run the deployment outside the handler, and treat input as untrusted.</strong> A <code>time.sleep</code> is harmless in a demo, but real deploys can take minutes. Hand the job to a worker or queue, and have it report back to the channel when it finishes. When you replace <code>run_deploy</code>, keep the discipline from Step 3: pass arguments as a list (<code>subprocess.run([...])</code>) and never build a shell string from what someone typed.</p>
<p><strong>4. Keep an audit trail.</strong> Log who requested, who approved, what was deployed, and when. This was the original problem, so it’s worth doing properly.</p>
<p><strong>5. Add per-environment locks and expiry.</strong> Don’t let two deploys to the same environment run at once, and expire requests nobody acted on so an old approval can’t be clicked days later.</p>
<p><strong>6. Keep it running.</strong> The Slack SDK handles reconnecting if the connection drops, but you still need something to restart the process if it crashes: a systemd service, a container with a restart policy, or your platform’s equivalent.</p>
<p><strong>7. Test the handlers.</strong> Bolt handlers are plain functions. You can unit-test them by passing in fake <code>ack</code>, <code>respond</code>, <code>body</code>, and <code>action</code> objects, with no Slack connection needed.</p>
<h2>Wrapping up</h2>
<p>The whole bot is a slash command, two buttons, and a handful of habits that matter more than the code itself: acknowledge fast, remove the buttons once a decision is made, check who is clicking, and treat the deploy as separate work from the reply.</p>
<p>The complete source is on GitHub: [link to your repo]. If you build on it, I’d like to hear what you add, especially if you solve the concurrency problem more cleanly than the sketch above.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/building-a-deploy-slack-bot-in-python-and-the-gotchas-the-docs-skip-hp7zm?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Redis Caching Patterns: Cache-Aside, Write-Through and the Stampede Problem]]></title><description><![CDATA[Caching is easy until a hot key expires and 50 requests hit your database at once. Here's how cache-aside and write-through work, why stale data and…]]></description><link>https://blog.zyvop.com/redis-caching-patterns-cache-aside-write-through-and-the-stampede-problem-ul6qw</link><guid isPermaLink="true">https://blog.zyvop.com/redis-caching-patterns-cache-aside-write-through-and-the-stampede-problem-ul6qw</guid><category><![CDATA[backend]]></category><category><![CDATA[caching]]></category><category><![CDATA[Python]]></category><category><![CDATA[Redis]]></category><category><![CDATA[systemdesign]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Wed, 04 Nov 2026 05:06:10 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/a7e66351-e8b2-4c29-ba06-d2a74ab3fecc.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Caching looks like the easiest win in backend work. Put Redis in front of the slow query and the page loads in 5 ms instead of 300.</p>
<p>Then the problems show up, one at a time. Users edit something and still see the old version. A hot key expires at 9:00:00 and fifty requests hit your database in the same second. Redis restarts and the whole app crawls.</p>
<p>None of this is exotic. These are the standard failure modes of the standard patterns.</p>
<p>This post covers the patterns, where each one breaks, and tested fixes. The code is Python, but the ideas apply to any language.</p>
<p>Everything here runs against an in-memory stand-in for Redis, so you can try it without installing anything. A real <code>RedisStore</code> is included for when you're ready.</p>
<h2>The patterns at a glance</h2>
<p>There are four names you'll hear:</p>
<table>
<thead>
<tr>
<th>Pattern</th>
<th>Who loads the data</th>
<th>How data gets into the cache</th>
<th>Main risk</th>
</tr>
</thead>
<tbody><tr>
<td>Cache-aside (also called lazy loading)</td>
<td>Your app</td>
<td>On a miss, the app loads from the database and stores the result</td>
<td>Stale data, slow misses, stampedes</td>
</tr>
<tr>
<td>Read-through</td>
<td>The cache layer</td>
<td>Same idea, but the cache calls your loader for you</td>
<td>The same, and you need a cache that supports loaders</td>
</tr>
<tr>
<td>Write-through</td>
<td>Your app, or the cache layer</td>
<td>Every write updates the database and the cache</td>
<td>Slower writes, and a cache full of data nobody reads</td>
</tr>
<tr>
<td>Write-behind</td>
<td>The cache layer, later</td>
<td>Writes land in the cache and are flushed to the database asynchronously</td>
<td>Data loss if the cache dies before it flushes</td>
</tr>
</tbody></table>
<p>Most teams use <strong>cache-aside</strong> for reads, plus either <strong>delete-on-write</strong> or <strong>write-through</strong> for updates, plus a <strong>TTL</strong> as a safety net. That combination is what we'll build.</p>
<p>Write-behind needs a warning. Redis keeps data in memory, so a crash between "written to the cache" and "flushed to the database" can lose that write. Persistence settings narrow the window but don't close it. Use it only when you can afford to lose recent writes.</p>
<h2>The building blocks</h2>
<p>Everything goes through five operations. Here's the real Redis adapter:</p>
<pre><code class="language-python">class StoreUnavailable(Exception):
    """The cache server can't be reached. The app should carry on without it."""

# From the Redis docs: only delete the lock if it still holds OUR token.
RELEASE_LOCK = """
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end
"""

class RedisStore:
    """The five operations the cache needs, on top of redis-py."""

    def __init__(self, url="redis://localhost:6379/0"):
        import redis

        self._redis = redis
        self._client = redis.Redis.from_url(
            url, decode_responses=True, socket_timeout=0.1, socket_connect_timeout=0.1
        )
        self._release = self._client.register_script(RELEASE_LOCK)

    def _run(self, command, *args, **kwargs):
        try:
            return command(*args, **kwargs)
        except self._redis.RedisError as error:
            raise StoreUnavailable(str(error)) from error

    def get(self, key):
        return self._run(self._client.get, key)

    def set(self, key, value, px=None, nx=False):
        return bool(self._run(self._client.set, key, value, px=px, nx=nx))  # None (NX refused) -&gt; False

    def delete(self, key):
        self._run(self._client.delete, key)

    def incr(self, key):
        return self._run(self._client.incr, key)

    def delete_if_equal(self, key, value):
        return bool(self._run(self._release, keys=[key], args=[value]))
</code></pre>
<p>Some notes on it:</p>
<ul>
<li><p><code>SET</code> <strong>with</strong> <code>NX</code> <strong>and</strong> <code>PX</code> <strong>is one atomic command.</strong> <code>NX</code> means "only if the key doesn't exist", and <code>PX</code> sets a lifetime in milliseconds. It's how we'll take locks. Redis marks the old <code>SETNX</code> command as deprecated in favor of <code>SET</code> with <code>NX</code>.</p>
</li>
<li><p><strong>The Lua script comes from the Redis docs.</strong> It deletes a lock only if the value still matches your token, so you never delete someone else's lock. Redis 8.4 added <code>DELEX key IFEQ value</code>, which does the same in one command. The script works everywhere, so I've used that.</p>
</li>
<li><p><code>StoreUnavailable</code> <strong>is the important part.</strong> Any Redis error turns into one exception type that the cache catches. A cache outage must never become an app outage.</p>
</li>
<li><p><strong>The timeouts are short on purpose.</strong> 100 ms is plenty for a cache. If Redis hangs, you want to give up and move on.</p>
</li>
</ul>
<p>Now the cache itself. It has six ways to read, which we'll meet one at a time:</p>
<pre><code class="language-python">import json
import math
import random
import threading
import time
import uuid

from store import StoreUnavailable

def should_refresh_early(now, fresh_until, delta, beta, rand):
    """XFetch: the closer we get to expiry, the likelier a reader volunteers to refresh.
    `delta` is how long the last load took, `rand` is a random number in (0, 1]."""
    return now - delta * beta * math.log(rand) &gt;= fresh_until

def in_a_thread(fn):
    threading.Thread(target=fn, daemon=True).start()

class Cache:
    def __init__(self, store, clock=time.time, sleep=time.sleep, rng=None):
        self.store, self.clock, self.sleep = store, clock, sleep
        self.rng = rng or (lambda: 1.0 - random.random())  # in (0, 1], so log() is always safe

    # ---- plumbing: a cache outage must never become an app outage -------------------------

    def _read(self, key):
        try:
            raw = self.store.get(key)
        except StoreUnavailable:
            return None
        return json.loads(raw) if raw is not None else None

    def _put(self, key, value, ttl, grace=0, delta=0.0, spread=0.1):
        ttl = ttl * (1 + spread * (2 * self.rng() - 1))  # jitter: +/- 10% so keys don't expire together
        entry = {"value": value, "fresh_until": self.clock() + ttl, "delta": delta}
        try:
            self.store.set(key, json.dumps(entry), px=int((ttl + grace) * 1000))
        except StoreUnavailable:
            pass

    def _load(self, key, loader, ttl, grace=0):
        started = self.clock()
        value = loader()
        self._put(key, value, ttl, grace, delta=self.clock() - started)
        return value

    def _acquire(self, key, ms):
        token = uuid.uuid4().hex
        return token if self.store.set(f"lock:{key}", token, px=ms, nx=True) else None

    def _release(self, key, token):
        try:
            self.store.delete_if_equal(f"lock:{key}", token)  # never deletes someone else's lock
        except StoreUnavailable:
            pass

    # ---- 1. cache-aside ----------------------------------------------------------------------

    def get_or_load(self, key, loader, ttl=300):
        entry = self._read(key)
        return entry["value"] if entry is not None else self._load(key, loader, ttl)

    def invalidate(self, key):
        try:
            self.store.delete(key)
        except StoreUnavailable:
            pass  # the TTL is the backstop

    # ---- 2. write-through: call it right after the database write succeeds --------------------

    def write_through(self, key, value, ttl=300):
        self._put(key, value, ttl)

    # ---- 3. generation keys: cache-aside without stale sets -----------------------------------

    def get_versioned(self, key, loader, ttl=300):
        try:
            generation = self.store.get(f"gen:{key}") or "0"  # read it BEFORE loading
        except StoreUnavailable:
            return loader()
        return self.get_or_load(f"{key}:g{generation}", loader, ttl)

    def invalidate_versioned(self, key):
        try:
            self.store.incr(f"gen:{key}")  # older entries become unreachable
        except StoreUnavailable:
            pass

    # ---- 4. one loader per key ---------------------------------------------------------------

    def get_locked(self, key, loader, ttl=300, grace=0, lock_ms=10_000, poll=0.05, max_wait=5.0):
        entry = self._read(key)
        if entry is not None:
            return entry["value"]

        give_up_at = self.clock() + max_wait
        while True:
            try:
                token = self._acquire(key, lock_ms)
            except StoreUnavailable:
                return loader()  # no cache, no coordination: just load
            if token:
                break
            entry = self._read(key)  # someone else is loading, so wait for their result
            if entry is not None:
                return entry["value"]
            if self.clock() &gt;= give_up_at:
                return loader()  # waited long enough: do it ourselves
            self.sleep(poll)

        try:
            entry = self._read(key)  # it may have been filled while we were acquiring the lock
            return entry["value"] if entry is not None else self._load(key, loader, ttl, grace)
        finally:
            self._release(key, token)

    # ---- 5. serve stale data while one caller refreshes it -------------------------------------

    def get_swr(self, key, loader, ttl=60, grace=300, run=in_a_thread):
        entry = self._read(key)
        if entry is None:
            return self.get_locked(key, loader, ttl, grace)  # nothing cached at all
        if self.clock() &lt; entry["fresh_until"]:
            return entry["value"]

        try:
            token = self._acquire(key, 30_000)
        except StoreUnavailable:
            token = None
        if token:  # stale: exactly one caller volunteers to refresh

            def refresh():
                try:
                    self._load(key, loader, ttl, grace)
                finally:
                    self._release(key, token)

            run(refresh)
        return entry["value"]  # everyone gets the stale value straight away

    # ---- 6. refresh a little early, at random (XFetch) ----------------------------------------

    def get_xfetch(self, key, loader, ttl=300, beta=1.0):
        entry = self._read(key)
        if entry is not None and not should_refresh_early(
            self.clock(), entry["fresh_until"], entry["delta"], beta, self.rng()
        ):
            return entry["value"]
        return self._load(key, loader, ttl)
</code></pre>
<p>Each entry is stored as a small JSON envelope: the value, when it stops being fresh, and how long the last load took. That extra bookkeeping is what lets the later patterns work.</p>
<h2>Cache-aside</h2>
<p>This is the one you'll use most. The app asks the cache first. On a miss it loads from the database and fills the cache.</p>
<p><img src="https://mermaid.ink/img/pako:eNpVj81uwkAMhF_F9XlzaMUph1b8qkj0QmgvLAcncciqu1m665RSyLtXCRUSR4-_8YzPWPiSMcXK-mNRUxDYzHQDADDealzzV8tRoPIB2sghHT1p3EGSPMPkrHHZwJpLE180dlfTpN9dNNZGNF5gOtyQNjTQC7t7yJkYe2q21bjyVEIVvAOpGUoSyinyzTIbMudbjZn4wGD-k-FopAaCzWZ1Y-cDO0WFjoMjU2J6RqnZ9X-WXFFrBdVV-aBgKLcce6byjSzIGXvCFBM6HCwn8RSFnYKJNc3nGxXZMC98Iwo0Zrz3DO9LjQrWPvfiFbyy_WYxBSkYB0NWQaQmJpGDqVANIZn57bs8jg4_2HUK8_3UWx8wxYdjbYSx-wPXSYI3?type=png" alt="Mermaid Diagram" /></p>
<p>Using it takes one line:</p>
<pre><code class="language-python">cache = Cache(RedisStore())
user = cache.get_or_load("user:42", lambda: db.load_user(42), ttl=300)
</code></pre>
<p>The advantages are real. Only data that's actually requested gets cached, and a cache that's empty or down still returns correct answers, just more slowly. The AWS docs list two costs: each miss is slower than not caching at all, and data can go stale.</p>
<p>A few habits make it work better:</p>
<ul>
<li><p><strong>Always set a TTL.</strong> It's your safety net for every bug in your invalidation logic.</p>
</li>
<li><p><strong>Jitter the TTL.</strong> The code randomizes each lifetime by plus or minus 10%, so keys loaded together don't all expire together.</p>
</li>
<li><p><strong>Cache "not found" too.</strong> A <code>None</code> result is stored like any other. Without that, a request for something that doesn't exist hits the database every single time.</p>
</li>
<li><p><strong>Put the version in the key.</strong> When the shape of the data changes, change the key prefix. Old entries just age out.</p>
</li>
</ul>
<h3>Updating: delete, don't overwrite</h3>
<p>When data changes, the usual advice is to <strong>delete</strong> the cache key and let the next read reload it. Overwriting the cached value in place has its own race: two writers can finish in the wrong order and leave the older value behind.</p>
<pre><code class="language-python">db.update_user(42, new_name)
cache.invalidate("user:42")   # database first, then delete
</code></pre>
<p>That's simple and almost always right. But there's one race it doesn't close.</p>
<h3>The stale-set race</h3>
<p>Here a reader and a writer overlap:</p>
<p><img src="https://mermaid.ink/img/pako:eNplkMFOwkAQhl9lnJMmy0HDqYkk2BY1QWMoyIXL0A5l43a37k5BJLy7KZWg4TjzfZn_z-wxdwVjhIE_G7Y5J5pKT9XCAgDU5EXnuiYrMAEKMGEq2F_CuIOFDpcsaVlCQksKfInnLZ57Lae7k95gEEfwmE6hCeyj_h1cVzqEmzNOIsjScRp3BrRGyRJA1gzOFLAh0_CvP-_82VsynKZn3_L2QosjSNLxKfVfm-xPm_tzRue8OmFwG_bQmkKGobGizbHPdDoG39gArhFUWLGvSBcY7VHWXLWvL3hFjWnhcfNOXtPScGidlbMyokqbHUbYo7o23Au7IFwpeDDafrxQnh3nkbOiYIEZl45h9rxABRO3dOIUPLHZsOicFAy9JqMgkA29wF6vUB1DMv3ddrnt1194OChclrEzzmOEV9u1FsbDD8wXshE?type=png" alt="Mermaid Diagram" /></p>
<p>The reader loaded the old value, got delayed, and wrote it back after the writer's delete. The cache now holds stale data, and nothing will fix it until the TTL runs out.</p>
<p>The test <code>test_cache_aside_with_delete_on_write_can_cache_a_stale_value</code> reproduces this exactly, with two threads and a pair of events that force the order.</p>
<p>It's rare on any one request. At scale it happens every day. Facebook described this problem, and the thundering herd, in their <em>Scaling Memcache at Facebook</em> paper, and fixed both with <strong>leases</strong>. On a miss, the server hands out a token. A delete invalidates outstanding tokens, so a late writer's <code>set</code> is refused.</p>
<p>You can get a similar effect without server support, using <strong>generation keys</strong>:</p>
<pre><code class="language-python">user = cache.get_versioned("user:42", lambda: db.load_user(42))   # reads
cache.invalidate_versioned("user:42")                             # after a write
</code></pre>
<p>Instead of deleting the entry, a write bumps a counter. Readers look up the counter first, then use it in the key: <code>user:42:g7</code>. The late reader above stores its stale value under <code>g0</code>, but everyone now reads from <code>g1</code>, so it's never seen.</p>
<p>The order matters. The reader must read the counter <strong>before</strong> it loads from the database. The test <code>test_generation_keys_survive_the_same_race</code> runs the same interleaving and gets the new value.</p>
<p>The costs are one extra read per lookup, and one rule: the counter key must live at least as long as the longest entry TTL. If Redis evicts it, the counter restarts from zero and old entries could come back. Keep entry TTLs short as a backstop.</p>
<h2>Write-through</h2>
<p>Write-through updates the cache on every write, right after the database write succeeds:</p>
<pre><code class="language-python">db.update_user(42, new_name)                              # 1. the database is the source of truth
cache.write_through("user:42", db.load_user(42))          # 2. then the cache
</code></pre>
<p>The AWS docs sum up the trade: data in the cache is never stale, but every write now makes two trips. It also fills the cache with data nobody may ever read. Pairing it with cache-aside and a TTL fixes that, because anything missing just gets loaded on demand.</p>
<p>Do the database first. If it fails, nothing has changed. If you update the cache first and the database write then fails, you've cached something that never happened.</p>
<h3>When the second write fails</h3>
<p>There's no transaction spanning Redis and your database. If the cache update fails after the database commit, the cache is wrong.</p>
<p>The test <code>test_if_the_cache_write_fails_the_ttl_is_the_backstop</code> shows what happens: the stale value is served, then replaced once the TTL runs out. Your options are to retry, to delete the key instead, or to accept the staleness window. Whichever you choose, the TTL puts a ceiling on how wrong the cache can be.</p>
<p>In textbook write-through, the cache layer itself writes to the database. With Redis you usually do both writes in your own code. The behavior is the same, but the failure handling is yours.</p>
<h2>The stampede problem</h2>
<p>A cache stampede, also called a thundering herd or dogpile, goes like this. A popular key expires. Many requests arrive before anyone has reloaded it. Every one of them misses, and every one of them queries the database.</p>
<p>Here's a demo you can run. It fires 50 requests at the same instant, with an empty cache and a database that takes 200 ms:</p>
<pre><code class="language-python">from test_cache import stampede

for method in ("get_or_load", "get_locked", "get_swr"):
    reads, _ = stampede(method)
    print(f"{method:12s} -&gt; {reads} database read(s) for 50 simultaneous requests")
</code></pre>
<p>The result:</p>
<pre><code class="language-text">get_or_load  -&gt; 50 database read(s) for 50 simultaneous requests
get_locked   -&gt; 1 database read(s) for 50 simultaneous requests
get_swr      -&gt; 1 database read(s) for 50 simultaneous requests
</code></pre>
<p>Fifty identical queries for one answer. And it gets worse, because the database slows down under the load:</p>
<p><img src="https://mermaid.ink/img/pako:eNpNkD1vwkAMhv-K6_kyVOrEUImviEplgbZLjsFJnHDicg53R0OK-O9VQgXd_PH4tV9fsJCScYKVla7Yk4_wvtEOAGCaaVxJhAP3wOfWeA4ad5AkrzDLNK7J9eD5eOIQAzQmBKAI4grWuLspzEZ4nmmcWgtSQdxzA8cT-34IoaRIOYXHwHwcWGQaP_61IVjpApTSuTu5GMllpnFJxR6sUAmRDhzAiqvZ38HlCKZ_kuOZnXGldFD7QZVcCY14flhpjWUwj1XpzQUqbNg3ZEqcXHBwMryt5IpONqK6Vb7IG8oth4GpxMWUGmN7nGBCbWs5CX2I3CiYWeMOayq2Y56Kiwo0brkWhs83jQo2kksUBSu23xxNQQqm3pBVEMiFJLA3Fapxydb8DLc8v7RnvF4V5vVcrHic4FO3N5Hx-guyPZ4r?type=png" alt="Mermaid Diagram" /></p>
<p>A slow load makes the window wider, which brings in more requests, which makes the load slower. That loop is how a single expired key becomes an outage.</p>
<h2>Five defenses</h2>
<table>
<thead>
<tr>
<th>Defense</th>
<th>How it works</th>
<th>What it costs</th>
<th>Best for</th>
</tr>
</thead>
<tbody><tr>
<td>TTL jitter</td>
<td>Randomize TTLs so keys don't expire together</td>
<td>Nothing</td>
<td>Many keys loaded at the same time</td>
</tr>
<tr>
<td>Lock</td>
<td>One request loads, the rest wait for its result</td>
<td>Waiting, plus a lock to manage</td>
<td>Expensive loads where stale data isn't OK</td>
</tr>
<tr>
<td>Stale-while-revalidate</td>
<td>Serve the old value while one request refreshes it</td>
<td>Slightly stale reads</td>
<td>Hot keys where freshness can slip</td>
</tr>
<tr>
<td>XFetch</td>
<td>Readers randomly refresh a little before expiry</td>
<td>A few extra loads</td>
<td>The hottest keys, with no coordination</td>
</tr>
<tr>
<td>Scheduled refresh</td>
<td>A background job refreshes known hot keys</td>
<td>A job to run</td>
<td>Keys you know are always hot</td>
</tr>
</tbody></table>
<p>You'll often combine two or three. Let's go through them.</p>
<h3>1. TTL jitter</h3>
<p>Already built in. If you load 10,000 keys at startup with a flat 300-second TTL, they all expire in the same second five minutes later. Plus or minus 10% spreads them over a minute.</p>
<p>It does nothing for a single hot key, though. For that you need one of the next four.</p>
<h3>2. A lock, so only one request loads</h3>
<pre><code class="language-python">user = cache.get_locked("user:42", lambda: db.load_user(42), ttl=300)
</code></pre>
<p>The first request takes a lock with <code>SET lock:user:42 &lt;token&gt; NX PX 10000</code>. The rest poll the cache and pick up the result when it appears. In the demo above, that's 1 database read instead of 50.</p>
<p>The details that matter:</p>
<ul>
<li><p><strong>A random token per request.</strong> The lock holds a unique value, and release uses the compare-and-delete script. Otherwise a slow request could delete a lock that expired and was taken by someone else. <code>test_releasing_a_lock_never_deletes_someone_elses</code> checks it.</p>
</li>
<li><p><strong>A double-check after taking the lock.</strong> The value may have been filled while you waited. Without that check, the waiters you just held back all reload one after another. The test catches it.</p>
</li>
<li><p><strong>Always release, even on failure.</strong> The <code>finally</code> block frees the lock if the loader raises.</p>
</li>
<li><p><strong>A lock TTL.</strong> If a request crashes holding the lock, the TTL frees it. Set it longer than a normal load.</p>
</li>
<li><p><strong>A way out.</strong> If the lock holder never finishes, waiters give up after <code>max_wait</code> and load it themselves.</p>
</li>
</ul>
<p>Is a Redis lock safe enough? For this, yes. Martin Kleppmann's well-known article splits locks into two kinds.</p>
<p>Locks for <strong>efficiency</strong> avoid doing the same work twice, and an occasional failure just wastes a little effort. Locks for <strong>correctness</strong> protect data from corruption, and those need much stronger guarantees.</p>
<p>A cache refresh lock is the first kind. If it fails and two requests load, you've lost nothing but a query.</p>
<h3>3. Stale-while-revalidate</h3>
<p>Some data can be slightly old without anyone noticing. So don't throw it away at expiry. Keep it for a grace period, and serve it while one request fetches a fresh copy:</p>
<pre><code class="language-python">product = cache.get_swr("product:7", lambda: db.load_product(7), ttl=60, grace=300)
</code></pre>
<p>Each entry has two clocks. After <code>ttl</code> seconds it's <strong>stale</strong>. After <code>ttl + grace</code> seconds Redis actually removes it. A stale read returns the old value immediately, and one request, chosen with the same lock trick, refreshes it in the background.</p>
<p>Nobody waits. <code>test_stale_while_revalidate_serves_stale_and_refreshes_exactly_once</code> fires 50 requests at a stale key while the loader is blocked, and all 50 get the old value straight away. Then it releases the loader and checks that exactly one refresh happened.</p>
<p>The cold case, where nothing is cached at all, falls back to the lock. That's why the demo above shows 1 read for <code>get_swr</code> too.</p>
<h3>4. XFetch: refresh a little early, at random</h3>
<p>The first three defenses react after trouble starts, or at expiry. XFetch avoids expiry altogether. It comes from a 2015 VLDB paper by Vattani, Chierichetti and Lowenstein, <em>Optimal Probabilistic Cache Stampede Prevention</em>.</p>
<p>Every reader makes an independent random decision. The closer the key is to expiring, the likelier that reader volunteers to refresh it. The rule is one line:</p>
<pre><code class="language-text">refresh now if:  now - delta * beta * ln(random) &gt;= expiry
</code></pre>
<p><code>delta</code> is how long the last load took, so slow loads get refreshed earlier. <code>beta</code> is a dial, with 1.0 as the sensible default. <code>random</code> is a number between 0 and 1, so its logarithm is negative, which means the left side jumps ahead of "now" by a random amount.</p>
<p>The test <code>test_xfetch_probability_matches_the_formula</code> checks the maths. With one second left and a half-second load, the chance of a refresh should be <code>exp(-1 / 0.5)</code>, about 13.5%. Over 100,000 draws it lands within half a percentage point of that.</p>
<p>Does it work? A simulation reads one hot key 50 times a second for 200 seconds. The TTL is 10 seconds and each load takes half a second:</p>
<table>
<thead>
<tr>
<th></th>
<th>Reads that found the key missing</th>
<th>Loads</th>
</tr>
</thead>
<tbody><tr>
<td>Plain cache-aside</td>
<td>20</td>
<td>20</td>
</tr>
<tr>
<td>XFetch</td>
<td>1</td>
<td>24</td>
</tr>
</tbody></table>
<p>The one miss for XFetch is the very first read, on an empty cache. After that, a volunteer always refreshed the key before it expired. The price is about four extra loads over 200 seconds.</p>
<p>Each of the 20 misses in the plain run is a stampede waiting to happen. At 50 reads a second and half-second loads, up to about 25 requests would miss together every time.</p>
<p>XFetch is probabilistic, so it isn't a guarantee. Under heavy concurrency two readers can still volunteer at once. It's best on its own for the very hottest keys, or combined with a lock.</p>
<h3>5. Scheduled refresh</h3>
<p>If you know which keys are always hot, like a home page or a pricing table, don't let them expire. A background job refreshes them on a timer, a bit shorter than the TTL.</p>
<p>There's no code to show, because it's just a cron job calling your loader and <code>write_through</code>. The catch is that you have to know the hot keys in advance.</p>
<h2>Redis settings that matter</h2>
<p>Caching code is half the job. The server has to be set up like a cache:</p>
<ul>
<li><p><strong>Set</strong> <code>maxmemory</code> <strong>and an eviction policy.</strong> The Redis docs suggest <code>allkeys-lru</code> as a good default if you have no reason to prefer another. It evicts the least recently used keys, whether or not they have a TTL.</p>
</li>
<li><p><strong>Beware the</strong> <code>volatile-*</code> <strong>policies.</strong> They only evict keys that have a TTL, and they behave like <code>noeviction</code> if none do. Fill the server with keys that never expire and writes start failing.</p>
</li>
<li><p><strong>Measure your hit rate.</strong> <code>INFO stats</code> reports <code>keyspace_hits</code> and <code>keyspace_misses</code>. The hit rate is <code>hits / (hits + misses)</code>.</p>
</li>
<li><p><strong>Watch</strong> <code>evicted_keys</code><strong>.</strong> A rising count with a falling hit rate means the cache is too small, or the wrong policy is evicting your hot keys.</p>
</li>
<li><p><strong>Fail open, and fail fast.</strong> The <code>RedisStore</code> has a 100 ms timeout, and <code>Cache</code> treats an outage as a miss. <code>test_a_cache_outage_falls_back_to_the_database</code> checks that nothing raises when the store is down.</p>
</li>
<li><p><strong>Remember what an outage means.</strong> With Redis down, every request goes to the database. That's a stampede by definition, so think about load shedding or a circuit breaker for your database.</p>
</li>
<li><p><strong>Don't use</strong> <code>KEYS</code> <strong>in production.</strong> It scans every key. Use <code>SCAN</code>.</p>
</li>
</ul>
<h2>Which one do you need?</h2>
<p><img src="https://mermaid.ink/img/pako:eNpVkUuPGjEMgP-K63PmUKknDl3xWNqVtj0s9CXCwUwMSQkxSgzTKct_r8K0W-3Rif3Zn33BVhzjCLdRutZTVljObAIAGK8sfvOkQJmhlxO01PqQdncW19A072FysTilBJnJcS5QmEGiA0dKsJUMBMVLVuh8iHxn8TqAJ7X62eIPLhafYbqqmNZzQyU4hi6oB4Ll8tEAJQeOIyuDJOhyULa4fo15osyxr6TZyuLYuSGvUZ_ltPMGJMOOE2fSIAn23JcXxvQmcn-x-FCAUl9_4cy5By9aC0uUDlQgCrn_BrOhbAjuX-vM_w5RlCI3N_Um85licFQ16lqitPtB7vuctfW3bann2lW56Mt8_9ifpaI_rCwul4_wM6hyhlCAU1W0uEaDB84HCg5HF1TPh3pUx1s6RUUzvHylHGgTudScrSSd0yHEHkfY0PEYuSl9UT4YmMSQ9p-oXdziuSQ1YHHBO2H48mDRwJNsRMXAR45n1tCSgXEOFA0USqUpnMMWza3JIvyus7x9d_yF16vBzW4qUTKO8E3ngzJe_wB1XtI_?type=png" alt="Mermaid Diagram" /></p>
<h2>Try it against real Redis</h2>
<p>The tests use the in-memory stand-in. Once you have a Redis server, a quick check that the real adapter works:</p>
<pre><code class="language-python">from cache import Cache
from store import RedisStore

cache = Cache(RedisStore())  # defaults to redis://localhost:6379/0
print(cache.get_locked("smoke:1", lambda: "hello", ttl=30))
</code></pre>
<p>It should print <code>hello</code>. Then <code>redis-cli GET smoke:1</code> shows the JSON envelope the cache stored.</p>
<h2>The tests</h2>
<p>The stand-in is a small fake that behaves like Redis for the five operations we use. Atomic commands, <code>NX</code> refusing existing keys, expiry, <code>INCR</code> keeping the TTL, and compare-and-delete:</p>
<pre><code class="language-python">import threading
import time
from collections import Counter

from store import StoreUnavailable

class FakeClock:
    def __init__(self, now=1_000_000.0):
        self.now = now

    def __call__(self):
        return self.now

    def advance(self, seconds):
        self.now += seconds

class FakeStore:
    """An in-memory stand-in with Redis's semantics for the five commands the cache uses:
    SET with NX and PX, GET, DEL, INCR, and compare-and-delete. Each command is atomic."""

    def __init__(self, clock=time.time):
        self.clock, self.down = clock, False
        self.commands = Counter()
        self._data, self._lock = {}, threading.Lock()

    def _enter(self, name):
        if self.down:
            raise StoreUnavailable("connection refused")
        self.commands[name] += 1
        self._lock.acquire()

    def _live(self, key):  # an expired key behaves as if it was never there
        item = self._data.get(key)
        if item and item[1] is not None and item[1] &lt;= self.clock():
            del self._data[key]
            return None
        return item

    def get(self, key):
        self._enter("get")
        try:
            item = self._live(key)
            return item[0] if item else None
        finally:
            self._lock.release()

    def set(self, key, value, px=None, nx=False):
        self._enter("set")
        try:
            if nx and self._live(key):
                return False  # Redis replies nil
            self._data[key] = (value, self.clock() + px / 1000 if px else None)
            return True
        finally:
            self._lock.release()

    def delete(self, key):
        self._enter("delete")
        try:
            self._data.pop(key, None)
        finally:
            self._lock.release()

    def incr(self, key):
        self._enter("incr")
        try:
            item = self._live(key)
            number = int(item[0]) + 1 if item else 1
            self._data[key] = (str(number), item[1] if item else None)  # INCR keeps the TTL
            return number
        finally:
            self._lock.release()

    def delete_if_equal(self, key, value):
        self._enter("delete_if_equal")
        try:
            item = self._live(key)
            if item and item[0] == value:
                del self._data[key]
                return True
            return False
        finally:
            self._lock.release()
</code></pre>
<p>And the tests. They use real threads for the races and stampedes, and a fake clock wherever time needs to pass:</p>
<pre><code class="language-python">import json
import random
import threading
import time

from cache import Cache, should_refresh_early
from fake_store import FakeClock, FakeStore

def make(**kwargs):
    clock = FakeClock()
    store = FakeStore(clock)
    return Cache(store, clock=clock, sleep=clock.advance, **kwargs), store, clock

class Counter:
    """A loader that remembers how often it was called."""

    def __init__(self, value="v1"):
        self.value, self.calls = value, 0

    def __call__(self):
        self.calls += 1
        return self.value

# ---- cache-aside ------------------------------------------------------------------------------

def test_cache_aside_loads_once_then_hits_until_the_ttl_runs_out():
    cache, _, clock = make()
    load = Counter()
    assert [cache.get_or_load("k", load, ttl=60) for _ in range(3)] == ["v1"] * 3
    assert load.calls == 1
    clock.advance(70)  # even with +10% jitter, 60 seconds is long gone
    cache.get_or_load("k", load, ttl=60)
    assert load.calls == 2

def test_a_missing_value_is_cached_too():
    cache, _, _ = make()
    load = Counter(value=None)
    assert cache.get_or_load("nobody", load) is None and cache.get_or_load("nobody", load) is None
    assert load.calls == 1  # asking for something that doesn't exist doesn't hammer the database

def test_ttls_are_jittered_by_ten_percent():
    for rand, expected in ((1.0, 330), (1e-9, 270)):
        cache, store, clock = make(rng=lambda: rand)
        cache.get_or_load("k", Counter(), ttl=300)
        left = json.loads(store.get("k"))["fresh_until"] - clock.now
        assert abs(left - expected) &lt; 0.01

def test_a_cache_outage_falls_back_to_the_database():
    cache, store, _ = make()
    store.down = True
    load = Counter()
    assert cache.get_or_load("k", load) == "v1" and cache.get_locked("k", load) == "v1"
    cache.write_through("k", "v2")
    cache.invalidate("k")  # none of this raises

# ---- write-through ------------------------------------------------------------------------------

def test_write_through_serves_the_new_value_without_asking_the_database():
    cache, _, _ = make()
    db = {"u": "old"}
    load = lambda: db["u"]
    assert cache.get_or_load("u", load) == "old"
    db["u"] = "new"
    cache.write_through("u", db["u"])  # database first, then the cache
    reads = Counter(value="should not be called")
    assert cache.get_or_load("u", reads) == "new" and reads.calls == 0

def test_if_the_cache_write_fails_the_ttl_is_the_backstop():
    cache, store, clock = make()
    db = {"u": "old"}
    load = lambda: db["u"]
    cache.get_or_load("u", load, ttl=60)
    db["u"] = "new"
    store.down = True
    cache.write_through("u", "new")  # the database write worked, the cache update didn't
    store.down = False
    assert cache.get_or_load("u", load) == "old"  # stale...
    clock.advance(70)
    assert cache.get_or_load("u", load) == "new"  # ...but only until the TTL runs out

# ---- the stale-set race ---------------------------------------------------------------------------

def race(read_through_cache, invalidate):
    """A reader loads the old value, a writer updates and invalidates, then the reader writes its stale value back."""
    db = {"u": "old"}
    loaded, proceed = threading.Event(), threading.Event()

    def slow_loader():
        value = db["u"]  # the reader sees "old"
        loaded.set()
        proceed.wait(5)  # ...and gets stuck for a moment
        return value

    reader = threading.Thread(target=lambda: read_through_cache("u", slow_loader))
    reader.start()
    loaded.wait(5)
    db["u"] = "new"  # the writer updates the database
    invalidate("u")  # and invalidates the cache, before the reader has stored anything
    proceed.set()
    reader.join()
    return read_through_cache("u", lambda: db["u"])  # what does the next reader see?

def test_cache_aside_with_delete_on_write_can_cache_a_stale_value():
    cache, _, _ = make()
    assert race(cache.get_or_load, cache.invalidate) == "old"  # stale until the TTL runs out

def test_generation_keys_survive_the_same_race():
    cache, _, _ = make()
    assert race(cache.get_versioned, cache.invalidate_versioned) == "new"

# ---- the stampede -------------------------------------------------------------------------------------

def stampede(method, threads=50, delay=0.2):
    """`threads` requests arrive at the same instant, with nothing in the cache."""
    cache, _, _ = make()
    cache.clock, cache.sleep = time.time, time.sleep  # real threads need real time
    loads, lock, barrier, results = [], threading.Lock(), threading.Barrier(threads), []

    def slow_loader():
        time.sleep(delay)
        with lock:
            loads.append(1)
        return "fresh"

    def request():
        barrier.wait()
        results.append(getattr(cache, method)("hot", slow_loader))

    workers = [threading.Thread(target=request) for _ in range(threads)]
    for w in workers:
        w.start()
    for w in workers:
        w.join()
    return len(loads), results

def test_without_protection_every_request_hits_the_database():
    loads, results = stampede("get_or_load")
    assert loads &gt;= 45 and set(results) == {"fresh"}

def test_with_a_lock_only_one_request_hits_the_database():
    loads, results = stampede("get_locked")
    assert loads == 1 and results == ["fresh"] * 50

def test_the_lock_is_released_when_the_load_finishes_and_when_it_fails():
    cache, store, _ = make()
    cache.get_locked("k", Counter())
    assert store.get("lock:k") is None

    def broken():
        raise RuntimeError("database down")

    try:
        cache.get_locked("other", broken)
        raise AssertionError("the error should have been raised")
    except RuntimeError:
        assert store.get("lock:other") is None  # a failed load doesn't leave the key locked

def test_a_waiter_stops_waiting_when_the_lock_holder_never_finishes():
    cache, store, _ = make()
    cache.clock, cache.sleep = time.time, time.sleep
    store.set("lock:k", "someone-who-crashed", px=60_000)
    load = Counter()
    assert cache.get_locked("k", load, max_wait=0.2, poll=0.02) == "v1" and load.calls == 1

def test_releasing_a_lock_never_deletes_someone_elses():
    cache, store, clock = make()

    def slow_loader():
        clock.advance(11)  # we took longer than the 10-second lock
        store.set("lock:k", "second-holder", px=10_000, nx=True)  # so someone else took it
        return "v1"

    cache.get_locked("k", slow_loader, lock_ms=10_000)
    assert store.get("lock:k") == "second-holder"

def test_stale_while_revalidate_serves_stale_and_refreshes_exactly_once():
    cache, _, clock = make()
    cache.get_swr("hot", lambda: "stale", ttl=60)  # warm the key
    clock.advance(100)  # now past its freshness, but still in the cache
    gate, loads, refreshers = threading.Event(), [], []

    def loader():
        loads.append(1)
        gate.wait(5)
        return "fresh"

    def run(fn):
        refreshers.append(threading.Thread(target=fn))
        refreshers[-1].start()

    results = []
    workers = [threading.Thread(target=lambda: results.append(cache.get_swr("hot", loader, ttl=60, run=run))) for _ in range(50)]
    for w in workers:
        w.start()
    for w in workers:
        w.join()

    assert results == ["stale"] * 50  # nobody waited: the refresh is still blocked on the gate
    gate.set()
    for r in refreshers:
        r.join()
    assert len(loads) == 1 and len(refreshers) == 1
    assert cache.get_swr("hot", loader, ttl=60) == "fresh"

def test_stale_while_revalidate_cold_start_loads_once():
    loads, results = stampede("get_swr")
    assert loads == 1 and results == ["fresh"] * 50

# ---- XFetch ---------------------------------------------------------------------------------------------

def test_xfetch_probability_matches_the_formula():
    rng, draws = random.Random(1), 100_000
    # 1 second left, a load that takes 0.5s, beta 1: probability is exp(-1 / 0.5) = 13.5%
    hits = sum(should_refresh_early(0, 1.0, 0.5, 1.0, 1.0 - rng.random()) for _ in range(draws))
    assert abs(hits / draws - 0.1353) &lt; 0.005
    assert should_refresh_early(2.0, 1.0, 0.5, 1.0, 1.0)  # past expiry: always
    assert not any(should_refresh_early(0, 100.0, 0.5, 1.0, 1.0 - rng.random()) for _ in range(draws))  # far away: never

def simulate(method, seconds=200, rate=50, ttl=10, load_time=0.5):
    """One hot key read `rate` times a second. Returns (reads that found nothing, total loads)."""
    rng = random.Random(7)
    cache, store, clock = make(rng=lambda: 1.0 - rng.random())
    loads = [0]

    def loader():
        loads[0] += 1
        clock.advance(load_time)  # loading takes a while
        return loads[0]

    empty = 0
    for _ in range(seconds * rate):
        empty += store.get("hot") is None
        getattr(cache, method)("hot", loader, ttl=ttl)
        clock.advance(1 / rate)
    return empty, loads[0]

def test_xfetch_refreshes_before_the_key_expires():
    plain_empty, plain_loads = simulate("get_or_load")
    early_empty, early_loads = simulate("get_xfetch")
    assert plain_empty &gt;= 15  # one empty read per expiry: in real life, a stampede each time
    assert early_empty &lt;= 3
    assert early_loads &lt;= plain_loads * 1.3  # the price: a few extra loads

# ---- the stand-in itself ---------------------------------------------------------------------------------

def test_the_fake_store_behaves_like_redis_for_the_commands_we_use():
    clock = FakeClock()
    store = FakeStore(clock)
    assert store.set("k", "a", px=1000, nx=True) is True
    assert store.set("k", "b", px=1000, nx=True) is False and store.get("k") == "a"  # NX refuses
    clock.advance(1.5)
    assert store.get("k") is None and store.set("k", "c", nx=True) is True  # expired keys are gone
    assert store.incr("n") == 1 and store.incr("n") == 2
    assert store.delete_if_equal("k", "wrong") is False and store.get("k") == "c"
    assert store.delete_if_equal("k", "c") is True and store.get("k") is None
</code></pre>
<p>Run them with <code>pytest -q</code>. They check the caching logic, not Redis itself. The fake follows the documented Redis behavior, but only a real server proves your configuration, so run the smoke check above and watch your hit rate in production.</p>
<h2>In short</h2>
<ul>
<li><p><strong>Cache-aside</strong> for reads, <strong>delete on write</strong>, and <strong>always a TTL</strong>.</p>
</li>
<li><p>Overlapping reads and writes can leave stale data behind. Generation keys or leases close that gap.</p>
</li>
<li><p><strong>Write-through</strong> keeps the cache fresh at the price of slower writes. The second write can still fail.</p>
</li>
<li><p>A <strong>stampede</strong> turns one expired key into many database queries. A lock, stale-while-revalidate and XFetch each stop it in a different way.</p>
</li>
<li><p>Treat Redis as optional. Short timeouts and fall back to the database.</p>
</li>
</ul>
<hr />
<p><em>Details checked against the Redis documentation, the AWS ElastiCache docs and the papers below on October 29, 2026. Commands and defaults change between Redis versions, so confirm them on the version you run.</em></p>
<p><strong>Sources</strong></p>
<ul>
<li><p><a href="https://redis.io/docs/latest/commands/set/">Redis docs: SET</a></p>
</li>
<li><p><a href="https://redis.io/docs/latest/develop/reference/eviction/">Redis docs: Key eviction</a></p>
</li>
<li><p><a href="https://redis.io/docs/latest/commands/redis-8-4-commands/">Redis 8.4 commands, including DELEX</a></p>
</li>
<li><p><a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/Strategies.html">Amazon ElastiCache: Caching strategies</a></p>
</li>
<li><p><a href="http://cseweb.ucsd.edu/~avattani/papers/cache_stampede.pdf">Vattani, Chierichetti and Lowenstein: Optimal Probabilistic Cache Stampede Prevention (VLDB 2015)</a></p>
</li>
<li><p><a href="https://en.wikipedia.org/wiki/Cache_stampede">Wikipedia: Cache stampede</a></p>
</li>
<li><p><a href="https://www.usenix.org/conference/nsdi13/technical-sessions/presentation/nishtala">Nishtala et al.: Scaling Memcache at Facebook (NSDI 2013)</a></p>
</li>
<li><p><a href="https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html">Martin Kleppmann: How to do distributed locking</a></p>
</li>
</ul>
<hr />
<p><em>Published via <a href="https://zyvop.com/redis-caching-patterns-cache-aside-write-through-and-the-stampede-problem-ul6qw?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Zero-Downtime Database Migrations Without Breaking Production]]></title><description><![CDATA[A migration that takes milliseconds on your laptop can freeze production. Here's how locks cause outages, how expand and contract keeps old and new code…]]></description><link>https://blog.zyvop.com/zero-downtime-database-migrations-without-breaking-production-0c300</link><guid isPermaLink="true">https://blog.zyvop.com/zero-downtime-database-migrations-without-breaking-production-0c300</guid><category><![CDATA[backend]]></category><category><![CDATA[database-migrations]]></category><category><![CDATA[Devops]]></category><category><![CDATA[PostgreSQL]]></category><category><![CDATA[ZeroDowntime]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Tue, 03 Nov 2026 09:54:12 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/93339009-adb3-4868-986e-9f5632272ca2.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>You rename a column. On your laptop it takes a few milliseconds. In production the error rate jumps, the pager goes off, and the rename has already happened, so you can't just roll back the deploy.</p>
<p>Migrations fail in production for two reasons, and neither shows up in testing.</p>
<p>The first is <strong>locks</strong>. A statement that waits for a lock can freeze every query behind it, even when the statement itself is fast.</p>
<p>The second is <strong>compatibility</strong>. While you deploy, old and new code run side by side. A schema change that only the new code understands breaks the old code.</p>
<p>Fix both and most scary migrations turn boring. This post covers each fix, the Postgres rules behind it, and code you can run.</p>
<blockquote>
<p><strong>Versions:</strong> PostgreSQL 18 is the current stable release as of late October 2026. Version 19 is at the release-candidate stage, and 14 reaches end of life on November 12. Most of the advice here holds for Postgres 12 and later. I'll say where it doesn't.</p>
</blockquote>
<h2>Problem 1: the lock queue</h2>
<p>Most <code>ALTER TABLE</code> forms take an <code>ACCESS EXCLUSIVE</code> lock. The Postgres docs put it plainly: that's the lock you get unless the docs say otherwise for that form. It conflicts with every other lock mode, including the <code>ACCESS SHARE</code> lock that a plain <code>SELECT</code> holds.</p>
<p>That sounds fine if your <code>ALTER</code> takes a millisecond. The trap is the waiting.</p>
<p>If any query or transaction already holds a lock on the table, your <code>ALTER</code> has to wait for it. And while it waits, everything that arrives after it queues up behind it, even simple reads.</p>
<p><img src="https://mermaid.ink/img/pako:eNptkcFu2zAQRH9lylMKUEAL5KSDAFlVEANuDpYTtIAva2llEaa4CknZdQL_eyHLyaU9knjDebt8V7U0rFIV-HVkV_MPQ3tP_dYBwEA-mtoM5CJyUECOYOUEz4P4iLtObBOQF0VZVage83X59d_gYgr-ltEjX23KNTb5YlXi7kQufmbLX8XquVq-_C9fTPnyyP4sjsE28MzkSZblKfzonHF7tOJx_w2Ba3FNmJFFkmWLFCcyMVyB2PGHfBS0xpnQzWgxo45PqMpVWWwCyDVYPlXlehPwOvLI2HFnXHN95TrLHH2SyJAjexQpNh0j0s4yrMghoPXyxu7DeC65Gcz1_OmaZFmR3pYU6cABhN5Ya-aZ9E2i8WRcUFr17HsyjUrfVey4n36x4ZZGG5Web17Im8klTEwrLj5Qb-xZpSqhYbCchHOI3GssrHGHn1RX1_ODuKixVRXvhfG83CqNtewkisYj2yNHU5NG7g1ZjUAuJIG9aZW-llTmbXL5fj_8UZeLVrt9IVa8StWXU2ciq8tftEHNnA?type=png" alt="Mermaid Diagram" /></p>
<p>So a fast migration can still cause an outage. All it needs is one long-running query at the wrong moment.</p>
<h3>The fix: give up quickly, then retry</h3>
<p>Set a short <code>lock_timeout</code> before the statement. If the lock isn't granted in time, the <code>ALTER</code> fails, the queue drains, and you try again a bit later.</p>
<p>Here's a helper that does exactly that:</p>
<pre><code class="language-python">import time

LOCK_NOT_AVAILABLE = "55P03"  # the SQLSTATE Postgres uses when lock_timeout expires

def is_lock_timeout(error) -&gt; bool:
    code = getattr(error, "sqlstate", None) or getattr(error, "pgcode", None)  # psycopg 3, psycopg2
    return code == LOCK_NOT_AVAILABLE

def run_ddl(conn, statement, lock_timeout="2s", attempts=6, wait=2.0, sleep=time.sleep):
    """Run one DDL statement without letting it wait in the lock queue.
    It waits up to `lock_timeout` for its lock, gives up, and tries again a little later.
    Returns the attempt number that worked."""
    for attempt in range(1, attempts + 1):
        try:
            with conn.transaction():
                conn.execute(f"SET LOCAL lock_timeout = '{lock_timeout}'")
                conn.execute(statement)
            return attempt
        except Exception as error:
            if not is_lock_timeout(error) or attempt == attempts:
                raise
            sleep(wait * attempt)
</code></pre>
<p>A few things to know about it:</p>
<ul>
<li><p><strong>One transaction per attempt.</strong> <code>SET LOCAL</code> lasts only until the transaction ends, so the timeout can't leak into other work.</p>
</li>
<li><p><strong>Only lock timeouts are retried.</strong> Postgres reports them with SQLSTATE <code>55P03</code>. Any other error is real and should stop the migration.</p>
</li>
<li><p><strong>The pauses grow.</strong> Each retry waits a little longer, which gives busy periods time to pass.</p>
</li>
<li><p><strong>Use an autocommit connection.</strong> In psycopg 3 that's <code>autocommit=True</code>. Otherwise the transaction block may turn into a savepoint inside a bigger transaction, and the lock stays held.</p>
</li>
</ul>
<p>Two limits on <code>lock_timeout</code> are worth remembering:</p>
<ul>
<li><p><strong>It limits waiting, not running.</strong> It controls how long a statement waits for a lock. Once the lock is granted, <code>statement_timeout</code> is what bounds the run time.</p>
</li>
<li><p><strong>It's awkward for</strong> <code>CREATE INDEX CONCURRENTLY</code><strong>.</strong> That command waits on other transactions as part of its normal work, so a very short <code>lock_timeout</code> can make it fail. Use <code>0</code> or a longer value for those, and check for invalid indexes afterwards.</p>
</li>
</ul>
<blockquote>
<p><strong>Look before you leap.</strong> Before a risky <code>ALTER</code>, check for long-running transactions on the table:</p>
<pre><code class="language-sql">SELECT pid, now() - xact_start AS age, state, left(query, 80) AS query
FROM pg_stat_activity
WHERE xact_start IS NOT NULL AND now() - xact_start &gt; interval '30 seconds'
ORDER BY xact_start;
</code></pre>
</blockquote>
<p>And keep migrations small. When one <code>ALTER TABLE</code> has several subcommands, Postgres takes the strictest lock any of them needs. Splitting them up lets the cheap ones stay cheap.</p>
<h2>Problem 2: two versions of your code run at once</h2>
<p>Even a short deploy has an overlap. For a few minutes, some servers run the old code and some run the new. Blue-green and canary releases make that window longer.</p>
<p>So the rule for every schema change is this:</p>
<blockquote>
<p><strong>Every schema state must work with the code before it and the code after it.</strong></p>
</blockquote>
<p>A plain rename breaks that rule. Here it is in action, using SQLite so you can try it with no setup:</p>
<pre><code class="language-python"># from the tests below
db.execute("ALTER TABLE users RENAME COLUMN name TO full_name")
works(db, V1(), V1())  # False: the deployed code now errors on every request
</code></pre>
<p>In Postgres, <code>RENAME</code> is instant and doesn't touch stored data. That's the problem. The change lands immediately, and the old code has no idea the column moved.</p>
<h2>The expand and contract pattern</h2>
<p>The fix is to never change things in place. Add the new thing next to the old one, move everyone over, then remove the old one.</p>
<p>This is called <em>parallel change</em>, or <em>expand and contract</em>. Danilo Sato wrote it up on Martin Fowler's site in 2014, and Joshua Kerievsky had documented it as a refactoring strategy back in 2006. It has three phases: <strong>expand, migrate, contract</strong>. Because each phase is safe on its own, your code can be released in any of them.</p>
<p>Here's our rename done that way:</p>
<p><img src="https://mermaid.ink/img/pako:eNpV0MtOwzAQBdBfGWbtBPUBiwpV6lMgwYYINnGFpvGktXA8keMSCuq_o6SoUpe2zr2e8S8WYhgnWDppiz2FCM-v2gMAzHKNgxRW3zV587ANt1MyBsqDcx-eKta4gSSZwjzXOExheSCXtMFG7qnh2skRvoYaN-e6ea8XucZRCnMqPkvrXG8LqY8gzkCQtrn4Re-XucZxCllrY7GHwGSaq_7RxS97v8o13qWQRamhm8b6XVd9lRlfMqs-s8413qewEB8DFfFsg9TwvyYqrDhUZA1OfjHuueo-zHBJBxdRnW_eKVjaOm46U4qPa6qsO-IEE6prx0lzbCJXCubO-s8XKrL-vBYfFWjMeCcMb08aFbzKVqIoeGT3xdEWpGAWLDkFDfkmaTjYElX_SGZ_ulkG4_obTyeF291CnASc4E27t5Hx9Ad7GJmr?type=png" alt="Mermaid Diagram" /></p>
<p>And the detail for each step:</p>
<table>
<thead>
<tr>
<th>Step</th>
<th>What happens</th>
<th>Versions running</th>
<th>Can you undo it?</th>
</tr>
</thead>
<tbody><tr>
<td>1. Expand</td>
<td>Add a nullable full_name column</td>
<td>v1</td>
<td>Yes, drop the column</td>
</tr>
<tr>
<td>2. Dual-write</td>
<td>Deploy v2: writes both columns, reads name</td>
<td>v1 and v2</td>
<td>Yes</td>
</tr>
<tr>
<td>3. Backfill</td>
<td>Copy name into full_name for old rows</td>
<td>v2</td>
<td>Yes</td>
</tr>
<tr>
<td>4. Switch reads</td>
<td>Deploy v3: reads full_name, falls back to name, still writes both</td>
<td>v2 and v3</td>
<td>Yes</td>
</tr>
<tr>
<td>5. Stop the old</td>
<td>Deploy v4: only knows full_name</td>
<td>v3 and v4</td>
<td>Yes, back to v3</td>
</tr>
<tr>
<td>6. Contract</td>
<td>Drop name</td>
<td>v4</td>
<td>No</td>
</tr>
</tbody></table>
<p>Only the last step is one-way.</p>
<h3>The four app versions</h3>
<p>Here they are as code. The tests below use these to check that every step is safe:</p>
<pre><code class="language-python">import sqlite3
from contextlib import closing

ORIGINAL = "CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT, email TEXT)"
EXPAND = "ALTER TABLE users ADD COLUMN full_name TEXT"
CONTRACT = "ALTER TABLE users DROP COLUMN name"

def connect(path):
    return sqlite3.connect(path, timeout=10, isolation_level=None)  # autocommit: one statement, one transaction

def create_database(path):
    with closing(connect(path)) as db:
        db.execute("PRAGMA journal_mode = WAL")  # lets a backfill and a writer overlap
        db.execute(ORIGINAL)

class V1:
    """The app today. It only knows about `name`."""

    def create(self, db, name, email):
        return db.execute("INSERT INTO users (name, email) VALUES (?, ?)", (name, email)).lastrowid

    def read(self, db, user_id):
        return db.execute("SELECT name FROM users WHERE id = ?", (user_id,)).fetchone()[0]

class V2(V1):
    """Dual-write: writes both columns, still reads the old one."""

    def create(self, db, name, email):
        return db.execute(
            "INSERT INTO users (name, full_name, email) VALUES (?, ?, ?)", (name, name, email)
        ).lastrowid

class V3(V2):
    """Reads the new column, falls back to the old one. Still writes both."""

    def read(self, db, user_id):
        return db.execute("SELECT COALESCE(full_name, name) FROM users WHERE id = ?", (user_id,)).fetchone()[0]

class V4:
    """Only knows about `full_name`."""

    def create(self, db, name, email):
        return db.execute("INSERT INTO users (full_name, email) VALUES (?, ?)", (name, email)).lastrowid

    def read(self, db, user_id):
        return db.execute("SELECT full_name FROM users WHERE id = ?", (user_id,)).fetchone()[0]

def works(db, writer, reader, name="Ada Lovelace"):
    """Can `reader` see the data that `writer` saved, without errors?"""
    try:
        return reader.read(db, writer.create(db, name, "ada@example.com")) == name
    except sqlite3.OperationalError:
        return False
</code></pre>
<h3>What the tests found</h3>
<p>Once the new column exists, here's what happens when each version writes a row and each version reads it back:</p>
<table>
<thead>
<tr>
<th>Row written by</th>
<th>read by v1</th>
<th>read by v2</th>
<th>read by v3</th>
<th>read by v4</th>
</tr>
</thead>
<tbody><tr>
<td>v1</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
<td>broken</td>
</tr>
<tr>
<td>v2</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
</tr>
<tr>
<td>v3</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
<td>ok</td>
</tr>
<tr>
<td>v4</td>
<td>broken</td>
<td>broken</td>
<td>ok</td>
<td>ok</td>
</tr>
</tbody></table>
<p>"Broken" means the reader gets <code>None</code> instead of the name. No error is raised, which is worse. The data is just wrong.</p>
<p>Three lessons fall out of that table:</p>
<ul>
<li><p><strong>Neighbours are safe.</strong> v1 with v2, v2 with v3 and v3 with v4 can all run together in both directions. A normal rolling deploy only ever overlaps neighbours.</p>
</li>
<li><p><strong>Skipping a step isn't safe.</strong> v1 with v4 and v2 with v4 both break. Don't jump from v1 straight to v4.</p>
</li>
<li><p><strong>v3 is the bridge.</strong> Its fallback to the old column is what lets v4 start writing only the new one.</p>
</li>
</ul>
<h3>Traps in the steps</h3>
<ul>
<li><p><strong>Backfill only after the last v1 is gone.</strong> An old instance that's still writing will keep creating rows with no <code>full_name</code>. Run the backfill once everything writes both columns, then check for stragglers at the end.</p>
</li>
<li><p><strong>Relax the old column first.</strong> If <code>name</code> is <code>NOT NULL</code>, v4 can't insert without it. Drop that constraint before step 5. In Postgres, <code>ALTER COLUMN name DROP NOT NULL</code> is quick.</p>
</li>
<li><p><strong>Hunt for hidden readers.</strong> Indexes, views, triggers, reports, BI tools, other services and replication consumers may all depend on the old column. The app code isn't the only reader.</p>
</li>
<li><p><strong>Watch your ORM.</strong> Some cache column lists or use <code>SELECT *</code>. Deploy and restart before you rely on the new shape.</p>
</li>
<li><p><strong>Wait before you contract.</strong> The drop is the one step you can't undo. Leave the old column for a while, and take a backup first.</p>
</li>
</ul>
<blockquote>
<p><strong>Prefer a trigger to dual-writing in the app?</strong> A database trigger can keep the two columns in sync, so old app versions don't need to change. The trade-off is logic that lives in the database, where it's easy to forget. Tools like pgroll automate a version of the whole pattern.</p>
</blockquote>
<h2>Backfilling without hurting anyone</h2>
<p>The backfill is where good migrations go wrong. The tempting version is one statement:</p>
<pre><code class="language-sql">UPDATE users SET full_name = name;
</code></pre>
<p>On a big table that's a bad idea. It runs as one huge transaction. In Postgres an <code>UPDATE</code> writes a new version of every row it touches, so the table bloats, the write-ahead log floods, and replicas fall behind.</p>
<p>Do it in small batches instead. The code below uses SQLite so it runs anywhere. To use it on Postgres, swap the driver and change the <code>?</code> placeholders to <code>%s</code>. The SQL itself is valid Postgres.</p>
<pre><code class="language-python">import time
from contextlib import closing

from rollout import connect

def backfill_full_name(path, batch_size=1000, pause=0.0, max_batches=None):
    """Copy users.name into users.full_name in small batches, one transaction each.
    Safe to stop and re-run: it only touches rows where full_name is still NULL."""
    last_id, copied, batches = 0, 0, 0
    while max_batches is None or batches &lt; max_batches:
        with closing(connect(path)) as db:
            ids = [row[0] for row in db.execute(
                "UPDATE users SET full_name = name"
                " WHERE full_name IS NULL AND id IN ("
                "   SELECT id FROM users WHERE id &gt; ? AND full_name IS NULL ORDER BY id LIMIT ?)"
                " RETURNING id",
                (last_id, batch_size),
            )]
        if not ids:
            break
        last_id, copied, batches = max(ids), copied + len(ids), batches + 1
        time.sleep(pause)  # breathe: let replicas and other queries catch up
    return copied
</code></pre>
<p>What makes it safe:</p>
<ul>
<li><p><strong>Small transactions.</strong> Each batch is its own transaction, so locks are short and nothing runs for long.</p>
</li>
<li><p><strong>Walking forward by</strong> <code>id</code><strong>.</strong> The loop never revisits a row. That's what guarantees it finishes, even for rows that stay <code>NULL</code> because <code>name</code> was <code>NULL</code>.</p>
</li>
<li><p><strong>A</strong> <code>NULL</code> <strong>filter.</strong> It only touches rows that still need a value, so you can stop and re-run it at any time, and it never overwrites a value the new app wrote in the meantime.</p>
</li>
<li><p><strong>A pause between batches.</strong> On a real system, also watch replication lag and slow down when replicas fall behind.</p>
</li>
</ul>
<p>When it finishes, verify before you move on:</p>
<pre><code class="language-sql">SELECT count(*) FROM users WHERE full_name IS NULL AND name IS NOT NULL;
-- must be 0
</code></pre>
<h2>Risky statements and their safer versions</h2>
<p>Not every change needs the full pattern. Many have a safe form that runs in place. These facts come from the PostgreSQL 18 docs:</p>
<table>
<thead>
<tr>
<th>You want to</th>
<th>The risk</th>
<th>The safer way</th>
</tr>
</thead>
<tbody><tr>
<td>Add a nullable column</td>
<td>None: it's a metadata change</td>
<td>Just run it, with a lock_timeout</td>
</tr>
<tr>
<td>Add a column with a non-volatile default, such as a constant or now()</td>
<td>Fast on modern Postgres</td>
<td>Just run it. The default is stored in metadata, with no rewrite</td>
</tr>
<tr>
<td>Add a column with a volatile default, such as gen_random_uuid()</td>
<td>Rewrites the whole table</td>
<td>Add it nullable, then backfill in batches</td>
</tr>
<tr>
<td>Add NOT NULL to an existing column</td>
<td>Scans the table under a heavy lock</td>
<td>Use a NOT VALID check first (below)</td>
</tr>
<tr>
<td>Add a CHECK constraint</td>
<td>Scans the table under a heavy lock</td>
<td>ADD ... NOT VALID, then VALIDATE CONSTRAINT</td>
</tr>
<tr>
<td>Add a foreign key</td>
<td>Scans, and locks both tables</td>
<td>ADD ... NOT VALID, then VALIDATE CONSTRAINT</td>
</tr>
<tr>
<td>Create an index</td>
<td>Blocks writes while it builds</td>
<td>CREATE INDEX CONCURRENTLY</td>
</tr>
<tr>
<td>Add a unique constraint</td>
<td>Builds an index under lock</td>
<td>Build the unique index concurrently, then attach it</td>
</tr>
<tr>
<td>Change a column's type</td>
<td>Usually rewrites the table and its indexes</td>
<td>Expand and contract with a new column</td>
</tr>
<tr>
<td>Rename a column or table</td>
<td>Instant, but breaks running code</td>
<td>Expand and contract</td>
</tr>
<tr>
<td>Drop a column</td>
<td>Fast, but breaks code still using it</td>
<td>Stop using it first. Then drop</td>
</tr>
</tbody></table>
<p>A couple of details behind that table:</p>
<ul>
<li><p><strong>Some type changes skip the rewrite.</strong> If the new type is binary coercible and the <code>USING</code> clause doesn't change the data, there's no rewrite. Changing <code>text</code> to <code>varchar</code> is one example. Anything else, assume a rewrite, which can need up to double the disk space.</p>
</li>
<li><p><code>DROP COLUMN</code> <strong>doesn't free space.</strong> It only hides the column. The space comes back gradually as rows are updated.</p>
</li>
<li><p><code>VALIDATE CONSTRAINT</code> <strong>is gentle.</strong> It takes only a <code>SHARE UPDATE EXCLUSIVE</code> lock, so reads and writes carry on while it scans.</p>
</li>
</ul>
<h3>Adding NOT NULL</h3>
<p>On PostgreSQL 12 and later, a valid <code>CHECK</code> constraint lets <code>SET NOT NULL</code> skip its scan:</p>
<pre><code class="language-sql">ALTER TABLE users ADD CONSTRAINT users_email_not_null CHECK (email IS NOT NULL) NOT VALID;
ALTER TABLE users VALIDATE CONSTRAINT users_email_not_null;  -- scans, but doesn't block writes
ALTER TABLE users ALTER COLUMN email SET NOT NULL;           -- skips the scan: the CHECK proves it
ALTER TABLE users DROP CONSTRAINT users_email_not_null;      -- tidy up, in a separate command
</code></pre>
<p>The <code>DROP</code> has to be its own command. The docs say the scan is skipped only if the <code>CHECK</code> isn't dropped in the same command.</p>
<p>PostgreSQL 18 can do this directly, because it allows not-null constraints to be <code>NOT VALID</code>:</p>
<pre><code class="language-sql">ALTER TABLE users ADD CONSTRAINT users_email_not_null NOT NULL email NOT VALID;
ALTER TABLE users VALIDATE CONSTRAINT users_email_not_null;
</code></pre>
<h3>Adding a foreign key</h3>
<pre><code class="language-sql">ALTER TABLE orders ADD CONSTRAINT orders_user_fk
  FOREIGN KEY (user_id) REFERENCES users (id) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_user_fk;
</code></pre>
<p>The first statement skips the scan and commits at once. New rows are checked from then on. The second checks the old rows without blocking writes.</p>
<h3>Creating an index</h3>
<pre><code class="language-sql">CREATE INDEX CONCURRENTLY idx_orders_user_id ON orders (user_id);
</code></pre>
<p>Plain <code>CREATE INDEX</code> lets reads through but blocks inserts, updates and deletes until it finishes. The <code>CONCURRENTLY</code> form avoids that, at a price:</p>
<ul>
<li><p><strong>It does more work.</strong> It scans the table twice and waits for existing transactions to finish.</p>
</li>
<li><p><strong>It can't run inside a transaction block.</strong> That's why <code>run_ddl</code> above isn't for it.</p>
</li>
<li><p><strong>It can fail and leave a mess.</strong> A failed build leaves an <em>invalid</em> index behind. Drop it and try again.</p>
</li>
</ul>
<pre><code class="language-sql">SELECT indexrelid::regclass FROM pg_index WHERE NOT indisvalid;
DROP INDEX CONCURRENTLY idx_orders_user_id;
</code></pre>
<p>For rebuilding an existing index, <code>REINDEX CONCURRENTLY</code> is available from Postgres 12.</p>
<h3>Adding a unique constraint</h3>
<p>Build the index first, then promote it. The promotion is quick:</p>
<pre><code class="language-sql">CREATE UNIQUE INDEX CONCURRENTLY users_email_key ON users (email);
ALTER TABLE users ADD CONSTRAINT users_email_key UNIQUE USING INDEX users_email_key;
</code></pre>
<h2>Tools that help</h2>
<p>You don't have to do all of this by hand:</p>
<table>
<thead>
<tr>
<th>Tool</th>
<th>What it does</th>
</tr>
</thead>
<tbody><tr>
<td>squawk</td>
<td>A linter for Postgres migrations. It reads your SQL without connecting to a database and flags statements that risk downtime. Good in CI</td>
</tr>
<tr>
<td>pgroll</td>
<td>Runs a migration as two schema versions at once, using views, so old and new clients both work. You start it, complete it, or roll it back. Needs Postgres 14 or later</td>
</tr>
<tr>
<td>strong_migrations</td>
<td>Catches unsafe migrations in Rails apps</td>
</tr>
<tr>
<td>django-pg-zero-downtime-migrations</td>
<td>The same idea for Django</td>
</tr>
</tbody></table>
<p>For MySQL, the rules differ and shops often reach for tools like gh-ost or pt-online-schema-change. The expand and contract pattern itself works on any database.</p>
<h2>Which technique do you need?</h2>
<p>Here's the question that decides it:</p>
<p><img src="https://mermaid.ink/img/pako:eNplkU1rGzEQhv_KdM5aSqGnpaTYa5sGUgf8kRKsUGZ3Zy1hrWSkcTau4_9etO6l7UEHiXee5x10wSa0jCV2LgyNoSiwmWkPADDZaZxAagz3BI0hv2eNL1AUdzC9aJwFTiCGIQNADAnYBPHkvfV78GH4UsePd0msczCEeADqhCNY-arxejNMM-xd4zMnje9QXTT-yHErYIJrgcAwvZ7BheYw0kKEyEO0wqNZqHb8P24ZMm220zh_O5JvIZ8meInUiMaXW7r6Kz3faVydfFYPVsyo_Cm253CScT6yRJt7_jP-p_tip3GbbrUSdQxdiH0Jy8cNPE0e7mcKqsdltV2t5svNw7OCmqQxIw4V9hx7si2WFxTDff6Oljs6OUF1e3miaPOyKWe64GVBvXVnLLGg49Fxkc5JuFcwddYfvlOzHu-L4EWBxjXvA8P2XqOCVaiDBAXf2L2y2IYUTKIlpyCRT0XiaDtUo2Rtf-Uunz4f3_B6VVjvq-BCxBI_DMYK4_U3tZ-8dg?type=png" alt="Mermaid Diagram" /></p>
<h2>Before you run it</h2>
<ul>
<li><p><strong>Rehearse on production-sized data.</strong> A migration that takes a second on 10,000 rows can take an hour on 100 million. Test on a copy.</p>
</li>
<li><p><strong>Lint in CI.</strong> Let a tool like squawk reject the dangerous statements before a human has to remember them.</p>
</li>
<li><p><strong>Run migrations as their own step.</strong> Not at app startup, where ten instances can race each other.</p>
</li>
<li><p><strong>Set a default</strong> <code>lock_timeout</code> <strong>for the migration role.</strong> Then a forgotten timeout can't hurt you.</p>
</li>
<li><p><strong>Keep each migration small.</strong> One risky statement at a time is easy to reason about and easy to retry.</p>
</li>
<li><p><strong>Pick a quiet moment.</strong> Retries still work at peak, but fewer long reports are running at night.</p>
</li>
<li><p><strong>Decide the undo first.</strong> Write down how you'd reverse each step before you start.</p>
</li>
</ul>
<h2>The short version</h2>
<ul>
<li><p>Migrations break production through <strong>locks</strong> and through <strong>version overlap</strong>.</p>
</li>
<li><p>Always run <code>ALTER</code> with a short <code>lock_timeout</code> and retry. Don't let it wait in the queue.</p>
</li>
<li><p>Never change a column in place. <strong>Expand, migrate, contract</strong>, and only contract once nothing uses the old shape.</p>
</li>
<li><p>Backfill in small, resumable batches, and verify at the end.</p>
</li>
<li><p>Use the safe forms: <code>NOT VALID</code> with <code>VALIDATE</code>, <code>CONCURRENTLY</code>, and the unique-index trick.</p>
</li>
</ul>
<h2>The tests</h2>
<p>The results above come from these tests. Create <code>test_migrations.py</code> next to the other files:</p>
<pre><code class="language-python">import os
import tempfile
import threading
from contextlib import closing, contextmanager
from itertools import product

from backfill import backfill_full_name
from ddl import run_ddl
from rollout import CONTRACT, EXPAND, V1, V2, V3, V4, connect, create_database, works

ALL = {"v1": V1(), "v2": V2(), "v3": V3(), "v4": V4()}

def new_db():
    path = os.path.join(tempfile.mkdtemp(), "app.db")
    create_database(path)
    return path

def null_count(path):
    with closing(connect(path)) as db:
        return db.execute("SELECT COUNT(*) FROM users WHERE full_name IS NULL AND name IS NOT NULL").fetchone()[0]

def coexist(db, running):
    """Every running version must be able to read what every other running version writes."""
    return all(works(db, ALL[w], ALL[r]) for w, r in product(running, repeat=2))

# ---- the problem: a plain rename -------------------------------------------------------------

def test_a_plain_rename_breaks_the_running_app():
    path = new_db()
    with closing(connect(path)) as db:
        assert works(db, V1(), V1())
        db.execute("ALTER TABLE users RENAME COLUMN name TO full_name")
        assert not works(db, V1(), V1())  # the code that is still deployed now errors on every request

# ---- the pattern: which versions can run together once the new column exists -----------------

def test_compatibility_matrix_after_expand():
    path = new_db()
    with closing(connect(path)) as db:
        db.execute(EXPAND)
        ok = {(w, r): works(db, ALL[w], ALL[r]) for w, r in product(ALL, repeat=2)}
    # rows written by v1 have no full_name; rows written by v4 have no name
    assert ok[("v1", "v4")] is False and ok[("v4", "v1")] is False and ok[("v4", "v2")] is False
    assert ok[("v1", "v1")] and ok[("v1", "v2")] and ok[("v1", "v3")]
    assert ok[("v2", "v1")] and ok[("v2", "v4")] and ok[("v3", "v4")] and ok[("v4", "v3")]

def test_neighbouring_versions_are_safe_but_skipping_a_step_is_not():
    path = new_db()
    with closing(connect(path)) as db:
        db.execute(EXPAND)
        for running in (["v1", "v2"], ["v2", "v3"], ["v3", "v4"]):
            assert coexist(db, running), running
        assert not coexist(db, ["v1", "v4"])            # skipped two steps
        assert not coexist(db, ["v2", "v4"])            # skipped one

def test_before_the_expand_step_the_new_code_cannot_run():
    path = new_db()
    with closing(connect(path)) as db:
        assert not works(db, V2(), V2())                # deploying v2 first would break
        assert coexist(db, ["v1"])

# ---- the whole rollout, step by step -----------------------------------------------------------

def test_full_rollout_never_breaks_a_running_version():
    path = new_db()
    with closing(connect(path)) as db:
        for i in range(50):
            V1().create(db, f"user {i}", f"u{i}@example.com")       # data that exists before we start

        assert coexist(db, ["v1"])
        db.execute(EXPAND)
        assert coexist(db, ["v1"])
        assert coexist(db, ["v1", "v2"])          # rolling deploy of v2
        assert coexist(db, ["v2"])
        assert backfill_full_name(path, batch_size=20) &gt; 0           # only after every v1 is gone
        assert null_count(path) == 0
        assert coexist(db, ["v2", "v3"])
        assert coexist(db, ["v3", "v4"])
        assert coexist(db, ["v4"])

        # Contract. Safe only because v4 is the only version left.
        db.execute(CONTRACT)
        assert coexist(db, ["v4"])
        assert not works(db, V3(), V3())                              # too early for v3 would have been an outage

# ---- the backfill ---------------------------------------------------------------------------------

def seed(path, rows):
    with closing(connect(path)) as db:
        db.execute("BEGIN")
        db.executemany("INSERT INTO users (name, email) VALUES (?, ?)", [(f"user {i}", f"u{i}@x.com") for i in range(rows)])
        db.execute("COMMIT")
        db.execute(EXPAND)

def test_backfill_copies_everything_in_small_batches():
    path = new_db()
    seed(path, 2_500)
    assert backfill_full_name(path, batch_size=500) == 2_500
    assert null_count(path) == 0
    with closing(connect(path)) as db:
        assert db.execute("SELECT COUNT(*) FROM users WHERE name != full_name").fetchone()[0] == 0

def test_backfill_finishes_even_when_some_names_are_null():
    path = new_db()
    seed(path, 10)
    with closing(connect(path)) as db:
        db.executemany("INSERT INTO users (name, email) VALUES (NULL, ?)", [(f"n{i}@x.com",) for i in range(10)])
    # Rows whose name is NULL stay NULL after the copy. Walking forward by id means we never revisit them.
    assert backfill_full_name(path, batch_size=5, max_batches=50) == 20

def test_backfill_can_be_stopped_and_resumed():
    path = new_db()
    seed(path, 2_000)
    assert backfill_full_name(path, batch_size=500, max_batches=2) == 1_000   # interrupted halfway
    assert null_count(path) == 1_000
    assert backfill_full_name(path, batch_size=500) == 1_000                  # picks up where it left off
    assert backfill_full_name(path, batch_size=500) == 0                      # re-running is harmless

def test_backfill_does_not_overwrite_newer_values():
    path = new_db()
    seed(path, 10)
    with closing(connect(path)) as db:
        db.execute("UPDATE users SET full_name = 'changed by the new app' WHERE id = 3")
    backfill_full_name(path, batch_size=4)
    with closing(connect(path)) as db:
        assert db.execute("SELECT full_name FROM users WHERE id = 3").fetchone()[0] == "changed by the new app"

def test_backfill_while_the_new_app_keeps_writing():
    path = new_db()
    seed(path, 3_000)
    stop = threading.Event()
    written = []

    def writer():
        with closing(connect(path)) as db:
            while not stop.is_set():
                written.append(V2().create(db, f"live {len(written)}", "live@example.com"))

    thread = threading.Thread(target=writer)
    thread.start()
    backfill_full_name(path, batch_size=250)
    stop.set()
    thread.join()

    assert written, "the writer should have inserted rows during the backfill"
    assert null_count(path) == 0                          # old rows were backfilled, new rows were dual-written

def test_a_straggling_v1_after_the_backfill_leaves_a_gap():
    path = new_db()
    seed(path, 10)
    backfill_full_name(path)
    with closing(connect(path)) as db:
        V1().create(db, "late", "late@example.com")        # an old instance that was still running
    assert null_count(path) == 1                          # why you verify at the end, and re-run if needed
    backfill_full_name(path)
    assert null_count(path) == 0

# ---- lock_timeout + retry -------------------------------------------------------------------------

class LockTimeout(Exception):
    sqlstate = "55P03"

class OtherError(Exception):
    sqlstate = "42601"

class FakeConn:
    def __init__(self, fail_times=0, error=LockTimeout):
        self.fail_times, self.error, self.log, self.statements_attempted = fail_times, error, [], 0

    @contextmanager
    def transaction(self):
        self.log.append("BEGIN")
        try:
            yield
        except Exception:
            self.log.append("ROLLBACK")
            raise
        self.log.append("COMMIT")

    def execute(self, sql):
        self.log.append(sql)
        if not sql.startswith("SET LOCAL"):
            self.statements_attempted += 1
            if self.statements_attempted &lt;= self.fail_times:
                raise self.error("could not obtain lock on relation \"users\"")

def test_ddl_runs_with_a_short_lock_timeout_inside_one_transaction():
    conn = FakeConn()
    assert run_ddl(conn, "ALTER TABLE users ADD COLUMN full_name text") == 1
    assert conn.log == ["BEGIN", "SET LOCAL lock_timeout = '2s'", "ALTER TABLE users ADD COLUMN full_name text", "COMMIT"]

def test_ddl_retries_with_growing_pauses_when_the_lock_times_out():
    conn, pauses = FakeConn(fail_times=2), []
    assert run_ddl(conn, "ALTER TABLE users DROP COLUMN name", sleep=pauses.append) == 3
    assert pauses == [2.0, 4.0]
    assert conn.log.count("ROLLBACK") == 2 and conn.log.count("COMMIT") == 1

def test_ddl_gives_up_after_the_last_attempt():
    conn = FakeConn(fail_times=99)
    try:
        run_ddl(conn, "ALTER TABLE users DROP COLUMN name", attempts=3, sleep=lambda s: None)
        raise AssertionError("should have raised")
    except LockTimeout:
        assert conn.statements_attempted == 3

def test_other_errors_are_not_retried():
    conn = FakeConn(fail_times=1, error=OtherError)
    try:
        run_ddl(conn, "ALTER TABLE", sleep=lambda s: None)
        raise AssertionError("should have raised")
    except OtherError:
        assert conn.statements_attempted == 1
</code></pre>
<p>Run it with <code>pytest -q</code>. The tests use SQLite, so they check the compatibility story: which versions can run together, and that the backfill behaves. They need SQLite 3.35.5 or newer. <code>DROP COLUMN</code> and <code>RETURNING</code> arrived in 3.35.0, and 3.35.5 fixed <code>DROP COLUMN</code> bugs that could corrupt a database file. Check yours with <code>python -c "import sqlite3; print(sqlite3.sqlite_version)"</code>.</p>
<p>They can't check Postgres locking. For that, the docs and your own staging database are the source of truth.</p>
<hr />
<p><em>Checked against the PostgreSQL 18 documentation and the other sources below on October 29, 2026. Postgres behavior is stable but not frozen, so confirm details for your version before you rely on them.</em></p>
<p><strong>Sources</strong></p>
<ul>
<li><p><a href="https://www.postgresql.org/docs/current/sql-altertable.html">PostgreSQL docs: ALTER TABLE</a></p>
</li>
<li><p><a href="https://www.postgresql.org/docs/current/sql-createindex.html">PostgreSQL docs: CREATE INDEX</a></p>
</li>
<li><p><a href="https://www.postgresql.org/docs/current/sql-reindex.html">PostgreSQL docs: REINDEX</a></p>
</li>
<li><p><a href="https://www.postgresql.org/">PostgreSQL.org: version and release status</a></p>
</li>
<li><p><a href="https://www.martinfowler.com/bliki/ParallelChange.html">Martin Fowler: Parallel Change</a></p>
</li>
<li><p><a href="https://xata.io/blog/migrations-and-exclusive-locks">Xata: Schema changes and the Postgres lock queue</a></p>
</li>
<li><p><a href="https://v2.postgres.ai/blog/20210923-zero-downtime-postgres-schema-migrations-lock-timeout-and-retries">Postgres.ai: Zero-downtime schema migrations need lock_timeout and retries</a></p>
</li>
<li><p><a href="https://thebuild.com/blog/all-your-gucs-in-a-row-lock_timeout/">Christophe Pettus: lock_timeout</a></p>
</li>
<li><p><a href="https://enterprisedb.com/blog/changes-not-null-postgres-18">EDB: Changes to NOT NULL in Postgres 18</a></p>
</li>
<li><p><a href="https://www.psycopg.org/psycopg3/docs/basic/transactions.html">psycopg 3 docs: Transactions management</a></p>
</li>
<li><p><a href="https://www2.sqlite.org/releaselog/3_35_0.html">SQLite 3.35.0 release notes</a></p>
</li>
<li><p><a href="https://xata.io/open-source/pgroll">pgroll by Xata</a></p>
</li>
<li><p><a href="https://github.com/sbdchd/squawk">squawk: a linter for Postgres migrations</a></p>
</li>
<li><p><a href="https://github.com/github/gh-ost">gh-ost: GitHub's online schema migration for MySQL</a></p>
</li>
</ul>
<hr />
<p><em>Published via <a href="https://zyvop.com/zero-downtime-database-migrations-without-breaking-production-0c300?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[How Computers Work: CPU, GPU, Memory, and the Operating System]]></title><description><![CDATA[Why does a 7B model need 14 GB, and why does your program sometimes just say Killed? This post explains bits, floating point, the memory hierarchy, CPUs…]]></description><link>https://blog.zyvop.com/how-computers-work-cpu-gpu-memory-and-the-operating-system-bp2ot</link><guid isPermaLink="true">https://blog.zyvop.com/how-computers-work-cpu-gpu-memory-and-the-operating-system-bp2ot</guid><category><![CDATA[#aiengineer]]></category><category><![CDATA[computerscience]]></category><category><![CDATA[GPU]]></category><category><![CDATA[memory]]></category><category><![CDATA[operatingsystems ]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Tue, 03 Nov 2026 06:11:22 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/6e7489b9-1112-4097-9dbc-4bbdd22a2ecf.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Up to now we've treated the computer as a box that runs our code. This post opens the box a little. It's core, and I'll explain why it matters in AI work: nearly every confusing thing about GPUs, "out of memory" errors, slow data loading, quantized models, and Docker containers makes sense once you have a simple picture of what's inside a computer and who's in charge of it.</p>
<p>We'll keep it practical. By the end you should be able to answer questions like "will this model fit on my GPU?" and "why did my program just say <code>Killed</code>?"</p>
<h2>Bits, bytes, and numbers</h2>
<p>Everything in a computer is stored as <strong>bits</strong>, which are tiny switches that are either 0 or 1. Eight bits make a <strong>byte</strong>. A kilobyte is about a thousand bytes, a megabyte about a million, and a gigabyte about a billion.</p>
<p>You can see numbers in binary right in Python:</p>
<pre><code class="language-python">print(bin(5))                # 0b101
print(format(5, "08b"))      # 00000101
print(int("1010", 2))        # 10
</code></pre>
<h3>Fixed-size integers</h3>
<p>Python integers can grow as large as you like (<code>2 ** 100</code> works fine). But the computer's hardware works with numbers of a <em>fixed</em> size, and NumPy and PyTorch use those. An 8-bit integer has only 256 possible values:</p>
<pre><code class="language-python">import numpy as np

for t in (np.int8, np.uint8, np.int16, np.int32):
    info = np.iinfo(t)
    print(t.__name__, info.min, info.max)

# int8 -128 127
# uint8 0 255
# int16 -32768 32767
# int32 -2147483648 2147483647
</code></pre>
<p>And when a number goes past the edge, it can quietly wrap around:</p>
<pre><code class="language-python">a = np.array([127], dtype=np.int8)
print(a + 1)    # [-128]
</code></pre>
<p>No error. The answer is just wrong. It's worth remembering whenever you work with small number types, such as 8-bit image pixels.</p>
<h3>Floating-point numbers</h3>
<p>Decimals are stored in a format called <strong>floating point</strong>: a sign, a size (the exponent), and the digits (the fraction). You met its side effect in Post 4, when <code>0.1 + 0.2</code> gave <code>0.30000000000000004</code>. Now we can see why. Ask Python for the exact value that's stored when you type <code>0.1</code>:</p>
<pre><code class="language-python">from decimal import Decimal

print(Decimal(0.1))
# 0.1000000000000000055511151231257827021181583404541015625
</code></pre>
<p>The computer can't store 0.1 exactly in binary, just like we can't write 1/3 exactly in decimal. It stores the nearest value it can.</p>
<p>The number of bits you spend on each number is a trade-off between precision and size. These are the ones that matter in AI:</p>
<table>
<thead>
<tr>
<th>Type</th>
<th>Bytes</th>
<th>Roughly how many digits of precision</th>
<th>Notes</th>
</tr>
</thead>
<tbody><tr>
<td>float64</td>
<td>8</td>
<td>about 16</td>
<td>Python's default float, and the NumPy default</td>
</tr>
<tr>
<td>float32</td>
<td>4</td>
<td>about 7</td>
<td>Long the standard for deep learning</td>
</tr>
<tr>
<td>float16</td>
<td>2</td>
<td>about 3</td>
<td>Half the size, but a small range</td>
</tr>
<tr>
<td>bfloat16</td>
<td>2</td>
<td>about 2 to 3</td>
<td>Same range as float32, less precision. Very popular for training</td>
</tr>
</tbody></table>
<p>You can see the limits yourself:</p>
<pre><code class="language-python">print(float(np.float32(0.1)))     # 0.10000000149011612
print(float(np.float16(0.1)))     # 0.0999755859375
print(np.finfo(np.float16).max)   # 65504.0
print(np.float16(70000))          # inf  (NumPy also warns about an overflow)
</code></pre>
<p><code>float16</code> can't even hold the number 70,000. It turns into <code>inf</code>, which is infinity. That's the kind of thing that makes a training run blow up, and it's why <code>bfloat16</code> exists: it keeps the wide range of <code>float32</code> and gives up digits instead. (NumPy doesn't have <code>bfloat16</code>, but PyTorch does, and we'll meet it in Post 40.)</p>
<p>One curious detail: in <code>float32</code>, <code>0.1 + 0.2 == 0.3</code> comes out <code>True</code>, while in <code>float64</code> it's <code>False</code>. That doesn't mean <code>float32</code> is more accurate. It's <em>less</em> precise, so the tiny difference gets rounded away. Don't rely on it either way. Compare floats with a tolerance (<code>math.isclose</code>), not with <code>==</code>.</p>
<h3>How big is a model?</h3>
<p>Here's where this pays off. A language model is mostly a huge pile of numbers called <strong>parameters</strong>, or weights. The memory it needs is simply:</p>
<p><strong>number of parameters × bytes per parameter</strong></p>
<p>A model with 7 billion parameters ("7B"), stored at different precisions:</p>
<table>
<thead>
<tr>
<th>Precision</th>
<th>Bytes per parameter</th>
<th>Memory for the weights</th>
</tr>
</thead>
<tbody><tr>
<td>float32</td>
<td>4</td>
<td>28 GB</td>
</tr>
<tr>
<td>float16 or bfloat16</td>
<td>2</td>
<td>14 GB</td>
</tr>
<tr>
<td>8-bit integers (int8)</td>
<td>1</td>
<td>7 GB</td>
</tr>
<tr>
<td>4-bit integers (int4)</td>
<td>0.5</td>
<td>3.5 GB</td>
</tr>
</tbody></table>
<p>(A handy shortcut: a billion parameters at one byte each is exactly one gigabyte.)</p>
<p>This is why you hear about <strong>quantization</strong>: storing weights with fewer bits so the model fits on cheaper hardware. It's a trade-off, since fewer bits means a little less accuracy, and we'll cover it in Post 69. And it explains why a "7B model" and a "70B model" are such different things to run: the 70B one needs ten times the memory.</p>
<p>These numbers are for the weights only. Running a model needs extra memory on top, for the working calculations and for a cache of the conversation so far (Post 48), and training needs a lot more. We'll build a calculator in the mini project.</p>
<h3>Text is bytes too</h3>
<p>When you send text to a model, or save it to a file, it's turned into bytes, usually with an encoding called UTF-8. English letters take one byte each. Other characters take more:</p>
<pre><code class="language-python">for text in ["hello", "héllo", "नमस्ते", "😀"]:
    print(repr(text), "characters:", len(text), "bytes:", len(text.encode("utf-8")))

# 'hello' characters: 5 bytes: 5
# 'héllo' characters: 5 bytes: 6
# 'नमस्ते' characters: 6 bytes: 18
# '😀' characters: 1 bytes: 4
</code></pre>
<p>So characters, bytes, and the <em>tokens</em> a model uses (Post 43) are three different ways to measure the same text, and they don't line up. A Hindi sentence has far more bytes than characters, and many tokenizers also split non-English text into more tokens than English text of the same meaning. That's worth keeping in mind when you estimate cost.</p>
<h2>The CPU</h2>
<p>The <strong>CPU</strong> (central processing unit) is the computer's general-purpose brain. It follows a list of tiny instructions, billions per second: add these two numbers, compare these, jump to that instruction, fetch this from memory.</p>
<p>Modern CPUs have several <strong>cores</strong>, and each core can follow its own list of instructions at the same time. That's the "several chefs" from Post 14. It's why processes can run in parallel on multiple cores, and why one Python thread can only use one core's worth of Python work at a time (the GIL).</p>
<p>You can ask Python how many cores you have:</p>
<pre><code class="language-python">import os

print(os.cpu_count())
</code></pre>
<p>A CPU is designed to be good at <em>anything</em>, and especially at finishing one task quickly: lots of branching, unpredictable decisions, a bit of everything. Keep that in mind, because the GPU is built on a very different idea.</p>
<h2>The memory hierarchy</h2>
<p>Here's a surprising fact: for many programs, the CPU isn't limited by how fast it can <em>calculate</em>. It's limited by how fast it can get the <em>data</em> to calculate with. Memory comes in layers, and each layer is bigger, but slower, than the one above it:</p>
<table>
<thead>
<tr>
<th>Layer</th>
<th>Typical size</th>
<th>Roughly how long to get data</th>
</tr>
</thead>
<tbody><tr>
<td>Registers and L1 cache</td>
<td>tens of KB</td>
<td>about 1 nanosecond</td>
</tr>
<tr>
<td>L2 and L3 cache</td>
<td>MBs to tens of MBs</td>
<td>a few to tens of nanoseconds</td>
</tr>
<tr>
<td>RAM (main memory)</td>
<td>GBs to hundreds of GBs</td>
<td>about 100 nanoseconds</td>
</tr>
<tr>
<td>SSD</td>
<td>hundreds of GBs to TBs</td>
<td>about 100,000 nanoseconds (0.1 ms)</td>
</tr>
<tr>
<td>Hard disk</td>
<td>TBs</td>
<td>about 10,000,000 nanoseconds (10 ms)</td>
</tr>
<tr>
<td>Network (same data center)</td>
<td>unlimited</td>
<td>about 500,000 nanoseconds (0.5 ms)</td>
</tr>
<tr>
<td>Network (across an ocean)</td>
<td>unlimited</td>
<td>about 150,000,000 nanoseconds (150 ms)</td>
</tr>
</tbody></table>
<p>These are rough, ballpark numbers, and they vary by hardware. What matters is the <em>ratio</em> between them. Nanoseconds are hard to feel, so let's scale it to human time. If getting something from the fastest cache took <strong>1 second</strong>, then:</p>
<table>
<thead>
<tr>
<th>Layer</th>
<th>Scaled to human time</th>
</tr>
</thead>
<tbody><tr>
<td>L2 cache</td>
<td>4 seconds</td>
</tr>
<tr>
<td>RAM</td>
<td>about 1.7 minutes</td>
</tr>
<tr>
<td>SSD</td>
<td>about 1.2 days</td>
</tr>
<tr>
<td>Hard disk</td>
<td>about 4 months</td>
</tr>
<tr>
<td>Network, across an ocean</td>
<td>about 5 years</td>
</tr>
</tbody></table>
<p>That's why reading a file for every item in a loop is so painful, why you load data once and keep it in memory, why caching helps, and why Post 14's concurrency matters when you wait on the network.</p>
<h3>Locality: why access order matters</h3>
<p>The CPU doesn't fetch one number at a time from RAM. It grabs a chunk, typically 64 bytes, and keeps it in cache, betting that you'll want the neighbors next. So reading memory <strong>in order</strong> is cheap, and jumping around is costly. You can feel the difference in NumPy. We'll pick up the same numbers from an array two ways, in order and shuffled:</p>
<pre><code class="language-python">import time

import numpy as np

def best_time(func, repeats=5):
    times = []
    for _ in range(repeats):
        start = time.perf_counter()
        func()
        times.append(time.perf_counter() - start)
    return min(times)

rng = np.random.default_rng(0)

for n in (50_000, 1_000_000, 10_000_000):
    data = rng.random(n)
    in_order = np.arange(n)
    shuffled = rng.permutation(n)
    t_ordered = best_time(lambda: data[in_order])
    t_shuffled = best_time(lambda: data[shuffled])
    print(f"{n:&gt;10,} floats: in order {t_ordered * 1000:7.2f} ms, "
          f"shuffled {t_shuffled * 1000:7.2f} ms, {t_shuffled / t_ordered:.1f}x slower")
</code></pre>
<p>Here's what I saw (yours will differ, and ten million floats needs about 80 MB of memory, so lower the size if yours is tight):</p>
<pre><code class="language-sql">50,000 floats: in order    0.16 ms, shuffled    0.23 ms, 1.5x slower
 1,000,000 floats: in order    2.72 ms, shuffled   13.73 ms, 5.1x slower
10,000,000 floats: in order   40.21 ms, shuffled  162.11 ms, 4.0x slower
</code></pre>
<p>Both versions read the same numbers and do the same amount of work. Only the <em>order</em> differs. With 50,000 floats, everything fits in the fast cache, so the order barely matters. Once the data is too big for the cache, jumping around gets several times slower. This is part of why NumPy arrays beat Python lists (Post 15): their numbers sit side by side in one block of memory.</p>
<h3>RAM versus GPU memory</h3>
<p>One more layer to know about. A GPU has its own memory, called <strong>VRAM</strong>. It's much faster than normal RAM, but much smaller, and moving data between RAM and the GPU is slow compared with working on data that's already there. So a lot of GPU programming comes down to two rules: keep the data on the GPU, and make sure the model's weights fit in VRAM. We'll come back to this in Post 70.</p>
<h2>CPU versus GPU</h2>
<p>A <strong>GPU</strong> (graphics processing unit) was invented to draw video game graphics, which means doing the same small calculation for millions of pixels at once. It turns out that's exactly the shape of work that neural networks need.</p>
<p>Here's the difference, in kitchen terms again:</p>
<ul>
<li><p>A <strong>CPU</strong> is a handful of expert chefs. Each can cook anything, make tricky decisions, and finish a single dish fast.</p>
</li>
<li><p>A <strong>GPU</strong> is an enormous team of line cooks, thousands of them, each with simple skills. Nobody is individually impressive, but if the job is "chop ten thousand onions," the team finishes far sooner.</p>
</li>
</ul>
<p>CPUs are built for <em>low latency</em> (finish one thing quickly). GPUs are built for <em>high throughput</em> (finish a huge amount of identical work overall).</p>
<p>Neural networks are mostly <strong>matrix multiplication</strong>: giant grids of numbers multiplied and added together, with each result independent of the others. You've already seen the idea in Post 15, where one matrix operation replaced thousands of loop iterations. A GPU takes that further, doing thousands of these multiply-adds simultaneously.</p>
<p>Let's measure how my CPU handles matrix multiplication. Multiplying two n × n matrices takes about 2n³ operations (n³ multiplications and n³ additions):</p>
<pre><code class="language-python">n = 2048
x = rng.random((n, n), dtype=np.float32)
y = rng.random((n, n), dtype=np.float32)

t = best_time(lambda: x @ y, repeats=3)
flops = 2 * n ** 3
print(f"{t:.3f} s, {flops / t / 1e9:.0f} GFLOPS")
</code></pre>
<p>"FLOPS" means floating-point operations per second, and a GFLOPS is a billion of them. On my small test machine, with a single core, this took about 0.15 seconds, which is roughly <strong>116 GFLOPS</strong>. With <code>float64</code> instead, it took twice as long. Half the bytes means half the memory traffic and room for twice as many numbers in each step, so smaller number types often run faster as well as fitting in less memory.</p>
<p>Notice the 2n³ too. Matrix multiplication is O(n³), and by the logic of Post 18, doubling n makes it eight times more work. That's a lot of work, and it's why this operation is worth building special hardware for.</p>
<p>A current data-center GPU does matrix math at rates measured in <em>trillions</em> of operations per second, hundreds of times what my little machine managed. That's the core reason deep learning took off when GPUs became available. Other accelerators exist too, such as Google's TPUs, built on the same principle.</p>
<p>GPUs aren't magic, though. They don't help much when:</p>
<ul>
<li><p>The job is small, so the time spent sending data to the GPU outweighs the savings.</p>
</li>
<li><p>The code is full of branching and decisions, which is CPU territory.</p>
</li>
<li><p>The data doesn't fit in VRAM.</p>
</li>
</ul>
<h2>Memory-bound or compute-bound?</h2>
<p>A task is <strong>compute-bound</strong> when the chip's math speed is the limit, and <strong>memory-bound</strong> when the limit is how fast data can be moved in from memory. This matters, because the fix is different for each.</p>
<p>Here's a surprising fact about chatbots. When a language model writes a reply for one user, one token at a time, it has to read essentially <em>all of its weights</em> out of memory to produce each single token. The math per token is small compared with the amount of data moved. So generation speed is often limited by <strong>memory bandwidth</strong> (how many GB per second the memory can deliver), not by raw calculation speed.</p>
<p>That gives a back-of-the-envelope ceiling:</p>
<p><strong>tokens per second ≈ memory bandwidth ÷ size of the model in memory</strong></p>
<p>If a GPU can read 1,000 GB per second, and a model takes 14 GB (the 7B model at 16 bits), the ceiling is about 71 tokens per second. Quantize it to 4 bits (3.5 GB), and the ceiling rises to about 286. That's an upper bound, not a promise, since real systems have other overheads, and serving many users at once changes the picture. But it explains something that otherwise seems odd: shrinking a model makes it both <em>fit</em> on smaller hardware and <em>run faster</em>.</p>
<h2>The operating system</h2>
<p>Between your program and the hardware sits the <strong>operating system</strong>: Windows, macOS, Linux. Its job is to share the machine among everything that's running. It decides which program gets which CPU core and when, hands out memory, manages files, and controls access to the network and devices. Your programs don't touch the hardware directly. They ask the OS to do it for them, through requests called <strong>system calls</strong>.</p>
<h3>Processes</h3>
<p>A running program is a <strong>process</strong>. Each process gets its own private slice of memory and a number called its process ID. Threads (Post 14) live <em>inside</em> a process and share its memory, and that's why they can trip over each other, while separate processes can't.</p>
<p>You can start a new process from Python and see this for yourself:</p>
<pre><code class="language-python">import os
import subprocess
import sys

result = subprocess.run(
    [sys.executable, "-c", "import os; print(os.getpid(), os.getppid())"],
    capture_output=True,
    text=True,
)
child_pid, child_parent = map(int, result.stdout.split())

print("the child's parent is this process:", child_parent == os.getpid())    # True
print("the child has its own process ID:", child_pid != os.getpid())          # True
</code></pre>
<p>The new Python is a separate process, with its own ID and its own memory. And it knows who started it.</p>
<h3>Virtual memory, swapping, and "Killed"</h3>
<p>Each process is told it has a big, private block of memory all to itself. That's an illusion called <strong>virtual memory</strong>, and the OS keeps track of where everything really lives. When the real RAM fills up, the OS can push less-used data out to the disk (called <strong>swapping</strong>). That keeps programs alive, but it's brutally slow. Remember the human-time scale above: RAM is minutes, and disk is days. A machine that's swapping heavily can become almost unusable.</p>
<p>And if memory runs out completely, on Linux the OS may simply <strong>kill</strong> a process to save the system. You just see a terse <code>Killed</code> message, with no Python traceback, because Python never gets the chance to say anything. If you've ever loaded a model that was too big for your machine and the program vanished, that's likely what happened. The fix is the arithmetic from earlier: work out the memory <em>before</em> you load.</p>
<h3>Standard output, standard error, and pipes</h3>
<p>Every process starts with three connections to the outside world: <strong>standard input</strong> (where it reads from), <strong>standard output</strong> (where <code>print</code> writes), and <strong>standard error</strong> (a second output channel meant for errors and diagnostics). Here's something that surprises people. Python's <code>logging</code> writes to standard <em>error</em> by default, not standard output:</p>
<pre><code class="language-python">import subprocess
import sys

script = '''
import logging, sys
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
print("this is normal output")
logging.info("this is a log line")
'''

r = subprocess.run([sys.executable, "-c", script], capture_output=True, text=True)
print("STDOUT:", repr(r.stdout))    # 'this is normal output\n'
print("STDERR:", repr(r.stderr))    # 'INFO: this is a log line\n'
</code></pre>
<p>Having two channels is useful. A program's real output can flow into another program, while its chatter goes somewhere you can read it. In a terminal, you redirect them separately:</p>
<pre><code class="language-bash">python my_script.py &gt; results.txt 2&gt; errors.log
</code></pre>
<p>And a <strong>pipe</strong> (<code>|</code>) connects one program's output straight to another's input. You used one back in Post 12 with <code>echo "the dog and the cat" | textkit</code>. Here's a longer chain of small tools:</p>
<pre><code class="language-bash">echo 'the cat the dog' | tr ' ' '\n' | sort | uniq -c | sort -rn | head -2
#       2 the
#       1 dog
</code></pre>
<p>That pattern of small programs connected by pipes is how a lot of data work gets done.</p>
<h3>Environment variables and exit codes</h3>
<p>A process inherits a copy of its parent's <strong>environment variables</strong>. That's how the <code>MODEL_API_KEY</code> trick from Post 12 works, and it's also how a child process can be given a different environment than the parent's:</p>
<pre><code class="language-python">env = dict(os.environ, MODEL_API_KEY="demo-key")
r = subprocess.run(
    [sys.executable, "-c", "import os; print(os.environ.get('MODEL_API_KEY'))"],
    env=env, capture_output=True, text=True,
)
print(r.stdout.strip())    # demo-key
</code></pre>
<p>When a process ends, it hands back an <strong>exit code</strong>: a number where 0 means success and anything else means failure. <code>sys.exit(3)</code> exits with code 3, and an uncaught exception exits with code 1:</p>
<pre><code class="language-python">r = subprocess.run([sys.executable, "-c", "raise ValueError('boom')"], capture_output=True)
print(r.returncode)    # 1
</code></pre>
<p>Tools like CI systems (Post 28) and shell scripts rely on exit codes to decide whether to carry on or stop. That's what turns a failing test into a red build.</p>
<h3>Signals</h3>
<p>When you press Ctrl+C, the OS sends your process a <strong>signal</strong> called SIGINT. Python turns that into the <code>KeyboardInterrupt</code> exception, which is why a bare <code>except:</code> that swallows everything (Post 11) can make a program refuse to stop.</p>
<h3>Containers</h3>
<p>You'll meet Docker in Post 23. Here's a preview that depends on what you just learned. On Linux, a <strong>container</strong> isn't a miniature computer. It's an ordinary process (or a group of them) that the OS has <em>isolated</em>, so it sees its own files, its own process list, and limited resources. It shares the machine's operating system kernel instead of running its own. That's why containers start in a moment and use far less than a full virtual machine.</p>
<h2>Mini project: will it fit?</h2>
<p>Let's turn the memory arithmetic into a small tool you'll genuinely use. Create <code>will_it_fit.py</code>:</p>
<pre><code class="language-python">BYTES_PER_PARAM = {
    "float32": 4,
    "float16": 2,
    "bfloat16": 2,
    "int8": 1,
    "int4": 0.5,
}

OVERHEAD = 1.2    # a rough allowance for activations, caches, and framework overhead

def weights_gb(params_billions, precision):
    """1 billion parameters at 1 byte each is 1 GB, so this is a simple product."""
    return params_billions * BYTES_PER_PARAM[precision]

def check_fit(name, params_billions, precision, memory_gb):
    needed = weights_gb(params_billions, precision) * OVERHEAD
    verdict = "fits" if needed &lt;= memory_gb else "does NOT fit"
    print(f"{name:&lt;12}{precision:&lt;10}{needed:7.1f} GB needed vs {memory_gb:&gt;3} GB: {verdict}")

def max_tokens_per_second(params_billions, precision, bandwidth_gb_per_s):
    """Rough ceiling for one-at-a-time generation: every token reads all the weights once."""
    return bandwidth_gb_per_s / weights_gb(params_billions, precision)

scenarios = [
    ("7B model", 7, "float16", 16),
    ("7B model", 7, "int8", 16),
    ("7B model", 7, "int4", 16),
    ("13B model", 13, "float16", 24),
    ("13B model", 13, "int4", 24),
    ("70B model", 70, "float16", 80),
    ("70B model", 70, "int4", 80),
]

print("Does it fit in GPU memory?")
for name, params, precision, memory in scenarios:
    check_fit(name, params, precision, memory)

print("\nRough ceiling on tokens per second, with 1,000 GB/s of memory bandwidth:")
for precision in ("float16", "int8", "int4"):
    rate = max_tokens_per_second(7, precision, 1000)
    print(f"7B model, {precision:&lt;8}about {rate:5.0f} tokens per second")
</code></pre>
<p>You should see:</p>
<pre><code class="language-sql">Does it fit in GPU memory?
7B model    float16      16.8 GB needed vs  16 GB: does NOT fit
7B model    int8          8.4 GB needed vs  16 GB: fits
7B model    int4          4.2 GB needed vs  16 GB: fits
13B model   float16      31.2 GB needed vs  24 GB: does NOT fit
13B model   int4          7.8 GB needed vs  24 GB: fits
70B model   float16     168.0 GB needed vs  80 GB: does NOT fit
70B model   int4         42.0 GB needed vs  80 GB: fits

Rough ceiling on tokens per second, with 1,000 GB/s of memory bandwidth:
7B model, float16 about    71 tokens per second
7B model, int8    about   143 tokens per second
7B model, int4    about   286 tokens per second
</code></pre>
<p>A few things worth noticing:</p>
<ul>
<li><p><strong>The first line is the instructive one.</strong> A 7B model at 16 bits has 14 GB of weights, and it <em>looks</em> like it would fit in 16 GB. But once you leave a margin for everything else, it doesn't. Memory problems usually show up as "just barely not enough."</p>
</li>
<li><p><strong>The 1.2 overhead is a rough guess, not a law.</strong> Real overhead depends on how long your prompts are and how many requests you run at once. Treat this tool as a quick sanity check before you download something large, not as a guarantee.</p>
</li>
<li><p><strong>The bandwidth figure is made up.</strong> I picked 1,000 GB/s to make the arithmetic easy. Look up the real number for the hardware you're considering.</p>
</li>
<li><p><strong>Quantization buys two things at once</strong>: it fits the model on smaller hardware, and it raises the speed ceiling.</p>
</li>
</ul>
<h3>Make it yours</h3>
<ol>
<li><p>Add a function for <em>training</em> memory. A commonly quoted rule of thumb for mixed-precision training with the Adam optimizer is about 16 bytes per parameter in total, since it keeps weights, gradients, and optimizer state. Find out how much a 7B model would need.</p>
</li>
<li><p>Read the actual memory of your own machine or GPU, and add it as a scenario.</p>
</li>
<li><p>Support multiple GPUs by splitting the weights across them.</p>
</li>
<li><p>Look up the memory and bandwidth of two real GPUs, and compare the ceilings for a model you care about.</p>
</li>
<li><p>Add the KV cache once you've read Post 48.</p>
</li>
</ol>
<h2>Practice</h2>
<p>About 20 minutes of real effort on each before asking for help.</p>
<ol>
<li><p>Write the numbers 5, 100, and 255 in binary by hand, then check them with <code>format(n, "08b")</code>. What's the range of a 16-bit unsigned integer?</p>
</li>
<li><p>Compute <code>0.1 + 0.2</code> in <code>float64</code> and in <code>float32</code>, and print how each is stored using <code>Decimal</code>. Explain in your own words why <code>==</code> is a bad way to compare floats.</p>
</li>
<li><p>Run the locality experiment with array sizes between 10,000 and 20,000,000. Where does the gap start to appear on your machine?</p>
</li>
<li><p>Work out the memory for a 3B model at <code>float16</code> and at <code>int4</code>, and decide whether each fits in 8 GB.</p>
</li>
<li><p>Use <code>subprocess</code> to start a child Python process and print both process IDs. Then make the child exit with a code of your choice and read it back.</p>
</li>
<li><p>Write a script that both prints and logs. Run it with <code>&gt;</code> and <code>2&gt;</code> redirects and check which output went to which file.</p>
</li>
<li><p>Look up the <code>psutil</code> library, and use it to print how much memory your Python process is using before and after creating a big list.</p>
</li>
<li><p>For a hypothetical GPU with a bandwidth of your choosing, estimate the tokens-per-second ceiling for a 13B model at 8 bits.</p>
</li>
</ol>
<p>Then ask an AI tutor to review your answers, and don't let it rewrite them.</p>
<h2>What to take away</h2>
<ul>
<li><p>Everything is bits. Fixed-size number types can overflow quietly, and floats can't store most decimals exactly.</p>
</li>
<li><p>The number type you choose sets the memory: <code>float32</code> is 4 bytes, <code>float16</code> and <code>bfloat16</code> are 2, and quantized weights are 1 or even half a byte.</p>
</li>
<li><p><strong>Model memory = parameters × bytes per parameter</strong>, plus overhead. Do this arithmetic before you load anything big.</p>
</li>
<li><p>Memory comes in layers, and each is bigger but much slower. Reading data in order is cheaper than jumping around.</p>
</li>
<li><p>CPUs are a few flexible cores. GPUs are thousands of simple ones, and neural networks are mostly the matrix math they're good at.</p>
</li>
<li><p>Generating text for one user is often limited by memory bandwidth, so smaller models are faster as well as smaller.</p>
</li>
<li><p>The OS shares the machine. Processes have private memory, <code>Killed</code> usually means you ran out of it, <code>print</code> and logging use different output channels, and exit codes tell other tools whether you succeeded.</p>
</li>
</ul>
<hr />
<p><strong>Previous:</strong> Post 18, <em>Big-O Without the Fear</em></p>
<p><strong>Next in the series:</strong> Post 20, <em>Networking &amp; the Web</em>, covering HTTP, REST, WebSockets, and Server-Sent Events, which is how streaming from a model really works.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/how-computers-work-cpu-gpu-memory-and-the-operating-system-bp2ot?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item><item><title><![CDATA[Database Transaction Isolation: 5 Concurrency Bugs and How to Fix Them]]></title><description><![CDATA[Your code works alone, then two users act at once and a balance goes wrong. This guide shows five classic concurrency bugs as timelines, what each…]]></description><link>https://blog.zyvop.com/database-transaction-isolation-5-concurrency-bugs-and-how-to-fix-them-76877</link><guid isPermaLink="true">https://blog.zyvop.com/database-transaction-isolation-5-concurrency-bugs-and-how-to-fix-them-76877</guid><category><![CDATA[concurrency]]></category><category><![CDATA[database transactions]]></category><category><![CDATA[isolationlevels]]></category><category><![CDATA[MySQL]]></category><category><![CDATA[PostgreSQL]]></category><dc:creator><![CDATA[ZyVOP]]></dc:creator><pubDate>Mon, 02 Nov 2026 11:34:04 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6a1714c0badcd8afcb06dd34/4f55c204-19d7-4c05-9cbc-4efd885c23dc.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Your code works perfectly when you test it alone. Then two users act at the same moment, and an account ends up with the wrong balance, or a hospital ends up with no doctor on call.</p>
<p>These bugs are hard to find because they depend on timing. Every statement is correct. The problem is how two transactions interleave.</p>
<p><em>Isolation levels</em> are the database setting that decides which interleavings are allowed. This post shows five classic bugs as timelines, explains what the four standard levels promise, and covers what PostgreSQL and MySQL actually do. It ends with fixes that work at any level and a lost update you can reproduce yourself in a few lines of Python.</p>
<h2>What isolation means</h2>
<p>A transaction is a group of statements that succeed or fail together. <em>Isolation</em> is the promise that transactions running at the same time don’t trip over each other.</p>
<p>The ideal is <em>serializable</em> behavior: the result is the same as if the transactions had run one at a time. Databases can deliver that, but it limits concurrency. So most offer weaker levels that allow some odd results in exchange for speed.</p>
<p>Your job is to know which odd results your level allows, and whether your code can survive them.</p>
<p>In the timelines below, A and B are two transactions talking to one database. Time runs downward, and the numbers show the order of the steps.</p>
<h2>Five bugs, one at a time</h2>
<h3>1. Dirty read</h3>
<p>A dirty read means you read a change that another transaction has not committed yet. If that transaction rolls back, you acted on a value that never existed.</p>
<p><img src="https://mermaid.ink/img/pako:eNplkEtv20AMhP8Kw1MCrAAXaC4CakBrOUgQ94HIzskXakXbi-5D2aXcuIH_e6EoOaQ5DucbDsEXNLFjLDHz08DBcG1pn8hvAwAADRLD4FtOk-4piTW2pyBQAWVYJwqZjNgYoPrM1HqEahJqKfNnX_-_Q09MVczntS5h86uu1ktoyVEwDN_gegaXIQqY6L0V4Q5OLFdTSL-FmuVquVi_hyav1kUxn-sSrmcfGx5-rla6WtxP0x9RGOKRE-hyPM6MDTEAwZHcwCAHEgg8Avxss3CHCj0nT7bD8gXlwH58Zsc7GpygmiaPlCy1jvPI7GKQG_LWnbDEgvrecZFPWdgr0M6G39_JNK_6JgZRsMWG95Fhc7dFBQ-xjRIV3LI7slhDCqpkySnIFHKROdkdqteSxv4db_nytX_G81lhu19EFxOWePHnYIXx_A9-BqTk?type=png" alt="Mermaid Diagram" /></p>
<p>You only see dirty reads at the lowest level, Read uncommitted, which few applications use.</p>
<h3>2. Non-repeatable read</h3>
<p>You read the same row twice in one transaction and get different values, because another transaction committed a change in between.</p>
<p><img src="https://mermaid.ink/img/pako:eNp1kEFrwkAQhf_KdE4tbEClp0CFxCgValua2JOXSTLRpZvduLvRWvG_lxihoPQ4b7733jBHLEzJGKLjbcu64ETS2lK90gAA1Hqj2zpn288NWS8L2ZD2EAE5yCxpR4WXRkN0yyRxByXkKSfHt_v4OiPumSgYj5M4hHT6Mp1k0FhZXNxJHATjcRTCcDDolfjCLt-TKJv2LDzBcDQQMHlbLObZ_5lw76hm8H8XPFzXjC41r8YzmB1biEJIO9e2ZXsQUMqqYsvaA2m3Z4sCa7Y1yRLDI_oN1917S66oVR5Fr3ySlZQrdh1TGe1nVEt1wBADahrFgTs4z7WAWEn9taAiPc8zo72AFaa8NgzL-QoFfJjceCPgmdWOvSxIQGQlKQGOtAscW1mhOJek8qe7ZfjYfOPpJDBfT4wyFkO822-kZzz9AhZvqHU?type=png" alt="Mermaid Diagram" /></p>
<p>This breaks reports and calculations that read a value more than once and expect it to stay put.</p>
<h3>3. Phantom read</h3>
<p>This is the same problem for a set of rows. You run the same query twice, and the set of matching rows changes because another transaction inserted or deleted one.</p>
<p><img src="https://mermaid.ink/img/pako:eNqtkU9Lw0AQxb_KOJeqbA6ip4CBpEmxYBWaqJdeJsk0XUx24-7E-od-d4kRBHv1-GZ-894M84mVrRlD9PwysKk41dQ46jYGAIAGsWboSnaT7smJrnRPRiAG8lA4Mp4q0dZAfMykyQilJFSS5-N-8tcjmZg4iKI0CSHPbrN5AZUdjJyen8HTTbbOwAvJ4OEaZj2bWptmNk2lSRBEURzC5aSTH5flXZ6tCyD4wcHZvYL5_Wq1LP4l72rSd1YY7Cs7iEOIwfB-TALqeybHNXS6DuT3WFTYsetI1xh-ouy4G_9Q85aGVlBNlUdymsqW_chsrZEFdbp9xxAD6vuWA__uhTsFSavN84qq_FsvrBEFG8y5sQwPyw0qWNvSilVww-0ri65IQew0tQo8GR94dnqL6jsk1x_jLhdX_RseDgrLZm5b6zDEk_1OC-PhC7dhtgs?type=png" alt="Mermaid Diagram" /></p>
<p>Counts, totals and “does any row exist?” checks are the usual victims.</p>
<h3>4. Lost update</h3>
<p>Two transactions read the same value, each computes a new one, and each writes it back. The second write silently overwrites the first.</p>
<p><img src="https://mermaid.ink/img/pako:eNqFkcFOAkEMhl-l9qLG2WQJctlEkh0WIomoEfDEpbsUmDg7s84MKBLe3SwIUTh47N_va5t0g4WdMibo-X3JpuBM0dxROTEAALQM1izLnN2-rsgFVaiKTIAUyMPIkfFUBGUNpOdMJmsoo0A5eT7vy9MZcs-kUbudyQSG3YduZwQ5aTLFj5_JKGq30wQacbxP5L-0_EUfZo-fs3TUPdBwB41WDFeNOIYbaMXXAjpPg0F_9HfFudQ8Ss1T6dEGBrtiB6mQCfSUIX1Ula9dAd5CeulhypX1KtSxtj6gwJJdSWqKyQbDgsv6R1Oe0VLXzV3ySk5RrtnXzMya0KNS6TUmGFFVaY782gcuBUitzNuAiuGu7lkTBExwyHPLMO5PUMCLzW2wAu5ZrzioggSkTpEW4Mn4yLNTMxS7JUP1Vd_SuK0-cbsVmM87VluHCV58LFRg3H4DoMq2tw?type=png" alt="Mermaid Diagram" /></p>
<p>Both deposits were valid, but the balance should be 180, not 130. This is probably the most common concurrency bug in application code.</p>
<h3>5. Write skew</h3>
<p>Two transactions read the same data, then each updates a <em>different</em> row based on what it read. Each decision was valid on its own, but together they break a rule.</p>
<p><img src="https://mermaid.ink/img/pako:eNqVkUGP00AMhf-K8QnQ5ADilEOlTLMVK9FdaVs4VVo5U6cZdTIOM86Wsup_R2kACVYcOD77e35P8jM62TOWmPnryNFx7emQqN9FAAAaVeLYN5xmPVBS7_xAUaECyrBNFDM59RKhesnUdoJqUmoo88u9_fuGnZmqWCxqW8Lm5tPNcgtOxqiv376B1cP9GiQ-OgphJmtbFItFVcL7Wdv_dNrfzl-ZWzoyVME7BmlbmAwGlvfr9e32z4graKX5B3YnyiBPnKAytoQ7aWR_Bp9B4k_6zApMrgPXsTvCiTJoGhlOHUfwCokiGuw59eT3WD6jdtxPv9pzS2NQNPPkCyVPTeA8Ma1EXVHvwxlLLGgYAhf5nJV7Azb4eFyT21z1SqIa2OGGD8Lw-XaHBh6kERUDHzk8sXpHBqrkKRjIFHOROfkWzTVk479PXd59GL7h5WKwOSwlSMISX506r4yXH1YZwAc?type=png" alt="Mermaid Diagram" /></p>
<p>The rule here is that at least one doctor must stay on call.</p>
<p>Because the two transactions wrote different rows, there is no row-level conflict to detect. That is why many databases miss it.</p>
<h2>The four standard levels</h2>
<p>The SQL standard defines four levels by which of the first three bugs they forbid. This table is from the <a href="https://www.postgresql.org/docs/current/transaction-iso.html">PostgreSQL documentation</a>:</p>
<table>
<thead>
<tr>
<th>Level</th>
<th>Dirty read</th>
<th>Non-repeatable read</th>
<th>Phantom read</th>
</tr>
</thead>
<tbody><tr>
<td>Read uncommitted</td>
<td>Possible</td>
<td>Possible</td>
<td>Possible</td>
</tr>
<tr>
<td>Read committed</td>
<td>Not possible</td>
<td>Possible</td>
<td>Possible</td>
</tr>
<tr>
<td>Repeatable read</td>
<td>Not possible</td>
<td>Not possible</td>
<td>Possible</td>
</tr>
<tr>
<td>Serializable</td>
<td>Not possible</td>
<td>Not possible</td>
<td>Not possible</td>
</tr>
</tbody></table>
<p>Notice what is missing: lost updates and write skew don’t appear in the table. A well-known 1995 paper, <a href="https://doi.org/10.1145/568271.223785">A Critique of ANSI SQL Isolation Levels</a>, pointed out this gap.</p>
<p>The standard also sets only <em>minimum</em> protections. A database may do better than the table says, or behave differently, at the same level name. So a level’s name alone can’t tell you whether your code is safe.</p>
<h2>What PostgreSQL and MySQL actually do</h2>
<p>Here is how the two most popular open-source databases implement each level:</p>
<table>
<thead>
<tr>
<th>Level</th>
<th>PostgreSQL</th>
<th>MySQL (InnoDB)</th>
</tr>
</thead>
<tbody><tr>
<td>Default</td>
<td>Read Committed</td>
<td>Repeatable Read</td>
</tr>
<tr>
<td>Read Uncommitted</td>
<td>Behaves exactly like Read Committed, so no dirty reads</td>
<td>Real dirty reads are possible</td>
</tr>
<tr>
<td>Read Committed</td>
<td>Each statement sees a fresh snapshot</td>
<td>Each read sees a fresh snapshot</td>
</tr>
<tr>
<td>Repeatable Read</td>
<td>One snapshot for the whole transaction. A concurrent change to a row you also change makes you fail with an error and retry. No phantoms</td>
<td>One snapshot, taken at the first read. But UPDATE, DELETE and locking reads use the latest committed rows, so read-then-write code can lose updates</td>
</tr>
<tr>
<td>Serializable</td>
<td>True serializable behavior. Conflicting transactions are aborted, and you retry them</td>
<td>Plain SELECTs become locking reads when autocommit is off, so readers and writers can block each other</td>
</tr>
</tbody></table>
<p>These details come from the <a href="https://www.postgresql.org/docs/current/transaction-iso.html">PostgreSQL</a> and <a href="https://dev.mysql.com/doc/refman/8.4/en/innodb-transaction-isolation-levels.html">MySQL</a> documentation.</p>
<p>To see the biggest difference, replay the lost update from earlier at Repeatable Read:</p>
<p><img src="https://mermaid.ink/img/pako:eNptUl1PGzEQ_CtTvxRUR7qo5eWkRrrLhTYq4SMJfeJl49skFj77au9BAfHfqyOhFOjjjmdnRzN-UCbUrHKV-FfH3nBlaROpufIAQJ0E3zUrjru5pSjW2Ja8oAAlLCP5REZs8Cjec6qyJ1UktKLE79_LtxrljlMMRqOqzFFOvk1PcTDnlnsNx5gz1Ycai8nJZLzEihx5sxeuysFgNCpyDLNsh5R7mcvzqlhOntn4iuHnTGN8NptNl68PvmceZTgYZhk-4Sg73KfiBOchySby4uJkh70yMJnPz-YwoXM1fBAkjpacvWeQMZwS6o4hASZ408XIXtC1NQm_aJ0GYYQbjihyFIjBuYQVmWuQrxFZouWEdQwNZMtIQlF22-wSY3a3uDjBwdT7UJWH_7F49kM_LcZwC5v25-sX4nMg_4b0xpcucxxbT-5vXDb1eWmkgPJjQs1tSFZ62IX0bM_XSquGY0O2VvmDki03_QeseU2dE6V3yE-Ktm889Zx18HJMjXV3KlcDalvHg3SXhBuN0ll_PSOzeJqPgxeNK7XgTWBcTq-UxjysggSN7-xuWKwhjaIvRCORT4O-nbXST0cW9r73MvzS_laPj1qtNuPgQlS5-nC7tcLq8Q_34f2L?type=png" alt="Mermaid Diagram" /></p>
<p>The timeline is the same, but the outcome isn’t. PostgreSQL stops A, so your code must be ready to catch the error and retry. MySQL lets A overwrite B’s change with a value computed from an old snapshot.</p>
<p>Write skew survives Repeatable Read in both databases. In PostgreSQL only Serializable stops it, and in MySQL, Serializable stops it by locking.</p>
<p>Other databases such as Oracle and SQL Server differ again. <a href="https://github.com/ept/hermitage">Hermitage</a>, a test suite by Martin Kleppmann, records how many databases behave at each level, so it’s the place to check if you use something else.</p>
<h2>The fixes</h2>
<p>Raising the isolation level is one fix, but it isn’t always the best one. These tools work at any level, and they often cost less.</p>
<p><strong>1. Let the database do the math.</strong> Instead of reading a balance and writing back a new one, send one statement:</p>
<pre><code class="language-sql">UPDATE accounts SET balance = balance + 50 WHERE id = 1;
</code></pre>
<p>The read and the write happen together, so a concurrent change can’t slip in between. This fixes lost updates even at Read Committed.</p>
<p><strong>2. Lock what you read.</strong> <code>SELECT ... FOR UPDATE</code> makes other transactions wait until you finish. It fixes lost updates, and it fixes write skew if you lock every row your decision depends on. It can’t lock a row that doesn’t exist yet, so it doesn’t cover phantoms.</p>
<p>With a lock, the two deposits from the lost update example come out right (shown at Read Committed):</p>
<p><img src="https://mermaid.ink/img/pako:eNqNksFOGzEQQH9l8AmEV0oEkaqViLTOJgKpgYqEnnKZ3cwmI7yere2QUpR_R85K0JJLj2O_eX4Hv6la1qRyFejXjlxNJePGY7tyAAC4i-J2bUW-nzv0kWvu0EUoAAMsPbqAdWRxUJwypUlQiRErDHR6b746TM8U2XhcmhwW0-_TyRIqtOhqgtnDIzz9KIvltMdKk2XjcZHDcDCAcy974ABO9mClfqb1RU-Z_5PdSySQF_JgdMIN7JFjgEY8FBAFGnYctv8G9oIP5w0MRwM4TzmXMBpcaJg8zOd3y79zTd5DqRE8WcJwknqq_Za0o6S9-qr9DC-0yWHGDu3HKoe0q8FJ3LLbpNlKiEqrlnyLvFb5m4pbatMfWFODO5sujyc_0TNWlkJiGnFxhi3bV5WrDLvOUhZeQ6RWg7HsnudYL47zTFzUsFIL2gjB091KaXiUSqJouCX7QpFr1FB4RqshoAtZIM-N0sdHFvwntQyvu9_qcNCq2kzEile5OttvOZI6vAOXOtUL?type=png" alt="Mermaid Diagram" /></p>
<p><strong>3. Check before you write.</strong> Add a version column, and make the update conditional on it:</p>
<pre><code class="language-sql">UPDATE accounts SET balance = 150, version = version + 1
WHERE id = 1 AND version = 7;
</code></pre>
<p>If this changes zero rows, someone else got there first, so reload and try again. This approach works in any database and holds no locks while a user is thinking.</p>
<p><strong>4. Let the database enforce the rule.</strong> Unique, foreign key and check constraints are correct under any timing. PostgreSQL also has exclusion constraints, which can stop two bookings from overlapping. If the rule is a constraint, there is no race to lose.</p>
<p><strong>5. Use Serializable, and retry.</strong> Write each transaction as if it were alone, and let the database abort the ones that conflict. Your code must catch the error and rerun the <em>whole</em> transaction. In PostgreSQL the error is a serialization failure (SQLSTATE <code>40001</code>).</p>
<p>Here is the doctors example again at Serializable in PostgreSQL, where the database aborts one of the two transactions:</p>
<p><img src="https://mermaid.ink/img/pako:eNqVUtFK60AQ_ZVxn1Q20IpPeShkmxZFewtp730qyGQzqUs3O7m7G7VK__2ShCuoIPh4Zs6ZM2eYN6G5IpGKQH87cppyg3uPzc4BAGAX2XVNSX7ELfpotGnRRcgAA2w9uoA6GnaQfeXkqiflGLHEQF_76vMMNXKyZDbLVQqbxf1ivgXNnYvnlxewLNYrYPeg0dqRmaskmc2yFK5GrH6oVO_K_55bPBBk1mgCrmvoBRLm69XqdvvJcn330XNQKi6_1akUFkWxLiCQN2jNKw7BazS28wTn15PJZHrxcXBB0R9_lGk64l8cCfiJPKgUFDh-hkAUgB1BxTqyB3bDqoCuggNRG8YEDqouHoUUDfkGTSXSNxEfqelfpaIaOxuFHCt_0BssLYWeU7OLS2yMPYpUJNi2lpJwDJEaCcoad1ih3gx4yS5K2IkN7Zng9-1OSCi45MgSbsg-UTQaJWT9lSQEdCHpT1YLOZhszGu_y_S6fRGnkxTlfs6WvUjF2fOjiSRO_wALSuL6?type=png" alt="Mermaid Diagram" /></p>
<p><strong>6. Keep transactions short.</strong> Never wait for user input or a slow network call inside a transaction. Long transactions hold locks and snapshots, which leads to more conflicts.</p>
<p>Which fix to reach for depends on the bug:</p>
<table>
<thead>
<tr>
<th>Bug</th>
<th>Simplest fix</th>
</tr>
</thead>
<tbody><tr>
<td>Dirty read</td>
<td>Read Committed or higher, which is the default almost everywhere</td>
</tr>
<tr>
<td>Non-repeatable read</td>
<td>Read everything you need in one statement, or use snapshot reads</td>
</tr>
<tr>
<td>Phantom read</td>
<td>Snapshot reads (Repeatable Read in PostgreSQL), a constraint, or Serializable</td>
</tr>
<tr>
<td>Lost update</td>
<td>Atomic UPDATE, a row lock, or a version check</td>
</tr>
<tr>
<td>Write skew</td>
<td>Lock the rows you read, a constraint, or Serializable</td>
</tr>
</tbody></table>
<h2>Reproduce a lost update</h2>
<p>You can trigger a lost update on your laptop. This script uses SQLite and Python threads. SQLite runs writers one at a time, so a barrier forces the unlucky timing from the lost update diagram above:</p>
<pre><code class="language-python">import sqlite3, threading, os

DB = 'bank.db'

def reset():
    if os.path.exists(DB):
        os.remove(DB)
    con = sqlite3.connect(DB)
    con.execute('CREATE TABLE accounts(id INTEGER PRIMARY KEY, balance INTEGER)')
    con.execute('INSERT INTO accounts VALUES (1, 100)')
    con.commit()
    con.close()

def balance():
    con = sqlite3.connect(DB)
    return con.execute('SELECT balance FROM accounts WHERE id = 1').fetchone()[0]

def unsafe_deposit(amount, both_have_read, a_has_written, is_b):
    con = sqlite3.connect(DB, timeout=5, isolation_level=None)  # autocommit
    current = con.execute('SELECT balance FROM accounts WHERE id = 1').fetchone()[0]
    both_have_read.wait()           # force the bad timing: both have seen 100
    if is_b:
        a_has_written.wait()        # B writes last, after A has finished
    con.execute('UPDATE accounts SET balance = ? WHERE id = 1', (current + amount,))
    if not is_b:
        a_has_written.set()

def safe_deposit(amount):
    con = sqlite3.connect(DB, timeout=5, isolation_level=None)
    con.execute('UPDATE accounts SET balance = balance + ? WHERE id = 1', (amount,))

def run(*jobs):
    threads = [threading.Thread(target=job) for job in jobs]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

reset()
both_have_read, a_has_written = threading.Barrier(2), threading.Event()
run(lambda: unsafe_deposit(50, both_have_read, a_has_written, False),
    lambda: unsafe_deposit(30, both_have_read, a_has_written, True))
print('read, compute, write in app code:', balance())   # 130, not 180

reset()
run(lambda: safe_deposit(50), lambda: safe_deposit(30))
print('single UPDATE statement         :', balance())   # 180
</code></pre>
<p>Running it prints:</p>
<pre><code>read, compute, write in app code: 130
single UPDATE statement         : 180
</code></pre>
<p>The database did nothing wrong here. The bug is in the application’s read, compute, write pattern. I ran the script 20 times, and the app-code version always ended at 130.</p>
<h2>Wrapping up</h2>
<p>Most concurrency bugs come from a short window between reading a value and writing a decision based on it. Each fix above closes that window in a different way, and Serializable with a retry loop is the fallback when the logic is too complex for the others.</p>
<p>Whatever you choose, test it with two real sessions to see what your database actually does.</p>
<hr />
<p><em>Published via <a href="https://zyvop.com/database-transaction-isolation-5-concurrency-bugs-and-how-to-fix-them-76877?utm_source=hashnode&amp;utm_medium=crosspost&amp;utm_campaign=syndication">ZyVOP</a> — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium &amp; Hashnode in 1 click.</em></p>
]]></content:encoded></item></channel></rss>