https://news.ycombinator.com/Links for the intellectually curious, ranked by readers.https://blog.dochia.dev/blog/idempotency/Thu, 07 May 2026 11:02:28 +0000https://news.ycombinator.com/item?id=48047930<a href="https://news.ycombinator.com/item?id=48047930">Comments</a><div class="prose prose-lg max-w-none" style="max-width: 100%;" data-astro-cid-bvzihdzo="" morss_own_score="3.0" morss_score="664.0"> <hr> <p>People talk about idempotency like it is a solved problem:</p> <blockquote> <p>Put an <code>Idempotency-Key</code> on the request. Store the response. Replay it on retry.</p> </blockquote> <p>And yes, that is doable. For the happy path, it is even fairly small.</p> <p>The client sends:</p> <pre><code><span>POST</span><span> /payments</span> <span>Idempotency-Key</span><span>:</span><span> abc-123</span> <span>Content-Type</span><span>:</span><span> application/json</span></code></pre> <pre><code><span>{</span> <span> "accountId"</span><span>: </span><span>"acc_1"</span><span>,</span> <span> "amount"</span><span>: </span><span>"10.00"</span><span>,</span> <span> "currency"</span><span>: </span><span>"EUR"</span><span>,</span> <span> "merchantReference"</span><span>: </span><span>"invoice-7781"</span> <span>}</span></code></pre> <p>The server checks whether it has seen <code>abc-123</code>. If not, it creates the payment. If yes, it returns the previous response.</p> <p>That version survives the demo.</p> <p>The part I contest is that this is the hard part. It is not. The hard part starts with the second request, because the second request is not always a clean replay of the first one.</p> <p>Maybe it is a completed replay. Fine. Return the stored result.</p> <p>Maybe it arrives while the first request is still running. Now your idempotency layer is part of your concurrency control.</p> <p>Maybe the first request created a local payment but crashed before publishing an event. Now the local row and the external side effects are out of step.</p> <p>Maybe the first request called a payment provider, the provider accepted it, and your process died before recording the result. Now your database cannot infer whether money moved.</p> <p>Or maybe the second request has the same key and different content:</p> <pre><code><span>{</span> <span> "accountId"</span><span>: </span><span>"acc_1"</span><span>,</span> <span> "amount"</span><span>: </span><span>"100.00"</span><span>,</span> <span> "currency"</span><span>: </span><span>"EUR"</span><span>,</span> <span> "merchantReference"</span><span>: </span><span>"invoice-7781"</span> <span>}</span></code></pre> <p>Same key. Different amount.</p> <p>This is the case that makes idempotency interesting. Is it a retry? Is it a client bug? Is it a new operation? Should the server replay the old response, reject the request, or treat <code>(key + content)</code> as a new identity?</p> <p>You can pick any of those policies if you document it clearly. But the server should have an opinion. Not necessarily my opinion, but a clear one.</p> <p>My bias for side-effecting APIs is: same scoped key plus different canonical command should be a hard error. It catches client bugs early. A client that believes it is safely retrying a 10 EUR payment should not have the server silently interpret the second request as something else.</p> <p>The cases that matter are the ones a replay cache does not explain:</p> <ul> <li>completed replay</li> <li>concurrent retry</li> <li>partial local success</li> <li>downstream unknown state</li> <li>same key with a different canonical command</li> <li>duplicate operation without a key</li> <li>retry after expiry</li> <li>retry after deploy, schema change, service hop, or region failover</li> </ul> <p>If your design only handles completed same-command retries, it is a replay cache. That might be enough for some endpoints. But it is not the whole problem.</p> <h2>Idempotency is about the effect</h2> <p>An operation is idempotent if applying it once or many times has the same intended effect.</p> <p>That definition is simple. The word doing all the work is “effect”.</p> <p>HTTP gives you method-level semantics. A <code>PUT /users/123/email</code> can be idempotent if sending the same representation repeatedly leaves the resource in the same state. A <code>DELETE /sessions/456</code> can be idempotent if deleting an already-deleted session still means “session does not exist”. Repeating the <code>DELETE</code> might return <code>404</code>; the effect can still be idempotent.</p> <p>But your handler can still produce repeated side effects the business cares about: duplicate audit records, duplicate domain events, duplicate emails, duplicate provider calls, or duplicate metrics that affect billing or fraud logic.</p> <p><code>POST</code> is usually not idempotent by default, but it can be made idempotent if the server stores and enforces the right behavior. The key identifies a claimed operation. It does not define request equivalence, replay policy, or downstream deduplication.</p> <p>A uniqueness constraint can prevent one class of duplicate. It does not, by itself, give the client a correct retry result.</p> <p>For example, <code>unique(account_id, merchant_reference)</code> might prevent two payment rows, but if the retry gets a generic <code>500</code>, the client still does not know whether the payment succeeded. If the row exists but the response is different, or the event is published twice, or the ledger entry is duplicated, the operation is not idempotent in the way the caller cares about.</p> <h2>What you need to remember</h2> <p>For <code>POST /payments</code>, the durable idempotency record needs to answer three questions:</p> <ol> <li>Who owns this key?</li> <li>What did the first command mean?</li> <li>What outcome can be replayed?</li> </ol> <p>In PostgreSQL-ish SQL, a minimal table might look like this:</p> <pre><code><span>create</span><span> table</span><span> idempotency_requests</span> <span>(</span> <span> tenant_id </span><span>text</span><span> not null</span><span>,</span> <span> operation_name </span><span>text</span><span> not null</span><span>,</span> <span> idempotency_key </span><span>text</span><span> not null</span><span>,</span> <span> request_hash </span><span>text</span><span> not null</span><span>,</span> <span> status</span><span> text</span><span> not null</span><span>,</span> <span> response_status </span><span>int</span><span>,</span> <span> response_body jsonb,</span> <span> resource_type </span><span>text</span><span>,</span> <span> resource_id </span><span>text</span><span>,</span> <span> error_code </span><span>text</span><span>,</span> <span> created_at </span><span>timestamptz</span><span> not null</span><span>,</span> <span> updated_at </span><span>timestamptz</span><span> not null</span><span>,</span> <span> expires_at </span><span>timestamptz</span><span> not null</span><span>,</span> <span> locked_until </span><span>timestamptz</span><span>,</span> <span> primary key</span><span> (tenant_id, operation_name, idempotency_key)</span> <span>);</span></code></pre> <p>The key is not globally unique unless you deliberately make it global. Usually it should not be. A broken client generating <code>abc-123</code> should only collide with itself, not with another tenant.</p> <p>Scope might be tenant, user, account, merchant, API client, or some combination. Pick it deliberately.</p> <p>The operation name prevents accidental reuse across different operations. A key used for <code>create_payment</code> should not automatically mean the same thing for <code>create_refund</code>.</p> <p>The <code>request_hash</code> is the server’s memory of the first command. Without it, same key plus different body becomes ambiguous. You either replay the first response for a different command, or you execute a new operation under an old key. Both are bad if the client thinks it is retrying.</p> <p><code>IN_PROGRESS</code> is not an internal detail. A retry can arrive while the first request still owns execution.</p> <p>The behavior needs to be explicit:</p> <table><thead><tr><th>Existing record</th><th>Same canonical command?</th><th>Suggested behavior</th></tr></thead><tbody><tr><td>none</td><td>yes</td><td>insert <code>IN_PROGRESS</code> and execute</td></tr><tr><td><code>COMPLETED</code></td><td>yes</td><td>replay stored response or documented equivalent</td></tr><tr><td>any existing record</td><td>no</td><td>reject with idempotency conflict</td></tr><tr><td><code>IN_PROGRESS</code>, fresh</td><td>yes</td><td>wait, return <code>202</code>, or return <code>409</code> + <code>Retry-After</code></td></tr><tr><td><code>IN_PROGRESS</code>, stale</td><td>yes</td><td>recover ownership; do not blindly execute again</td></tr><tr><td><code>FAILED_REPLAYABLE</code></td><td>yes</td><td>replay stored failure</td></tr><tr><td><code>FAILED_RETRYABLE</code></td><td>yes</td><td>allow retry according to policy</td></tr><tr><td><code>UNKNOWN_REQUIRES_RECOVERY</code></td><td>yes</td><td>trigger reconciliation or return pending/recovery status</td></tr><tr><td>expired/deleted</td><td>unknown</td><td>follow documented expiry behavior</td></tr></tbody></table> <p>The response fields exist because idempotency is not just about preventing duplicate writes. The client needs an answer.</p> <p>You can store the full response body, or store a reference to the created resource and reconstruct the response. Both choices are annoying in different ways.</p> <p>Storing full responses gives faithful replay. It can also retain PII, signed URLs, one-time tokens, cardholder-related data, or fields you never intended to keep in a retry table.</p> <p>Reconstructing from a resource reference saves space, but it can return a different representation if the resource changed after creation.</p> <p>This is a contract decision. “Replay the creation response” and “return the current payment” are both valid API designs. They are not the same design.</p> <h2>Same key, different command</h2> <p>This is the bug the idempotency layer should catch loudly.</p> <p>First request:</p> <pre><code><span>{</span> <span> "accountId"</span><span>: </span><span>"acc_1"</span><span>,</span> <span> "amount"</span><span>: </span><span>"10.00"</span><span>,</span> <span> "currency"</span><span>: </span><span>"EUR"</span><span>,</span> <span> "merchantReference"</span><span>: </span><span>"invoice-7781"</span> <span>}</span></code></pre> <p>Second request:</p> <pre><code><span>{</span> <span> "accountId"</span><span>: </span><span>"acc_1"</span><span>,</span> <span> "amount"</span><span>: </span><span>"100.00"</span><span>,</span> <span> "currency"</span><span>: </span><span>"EUR"</span><span>,</span> <span> "merchantReference"</span><span>: </span><span>"invoice-7781"</span> <span>}</span></code></pre> <p>Same <code>Idempotency-Key: abc-123</code>. Different amount.</p> <p>Returning the original response anyway is simple. It also hides a serious client bug. The client asked for a 100 EUR payment and got back a 10 EUR payment. If the caller does not compare the response carefully, it may believe the 100 EUR payment succeeded.</p> <p>That is not idempotency. That is reinterpretation.</p> <p>For side-effecting APIs, a scoped key reused with a different canonical command should be a hard error, regardless of whether the first operation completed, failed, or is still running.</p> <pre><code><span>HTTP</span><span>/</span><span>1.1</span><span> 409</span><span> Conflict</span> <span>Content-Type</span><span>:</span><span> application/json</span></code></pre> <pre><code><span>{</span> <span> "errorCode"</span><span>: </span><span>"IDEMPOTENCY_KEY_REUSED_WITH_DIFFERENT_REQUEST"</span><span>,</span> <span> "message"</span><span>: </span><span>"This idempotency key was already used with a different request."</span> <span>}</span></code></pre> <p><code>409 Conflict</code> is a defensible default because the request conflicts with the server’s remembered meaning for that scoped key. Some APIs use <code>400</code> or <code>422</code>; the important part is a stable machine-readable error and no silent replay for a different command.</p> <p>A common client bug looks like this:</p> <pre><code><span>bad:</span> <span> idempotencyKey = cartId</span> <span>POST /payments amount=10.00 key=cart_123</span> <span>POST /payments amount=15.00 key=cart_123</span> <span>better:</span> <span> idempotencyKey = paymentAttemptId</span></code></pre> <p>The server should not guess which payment the cart key was supposed to represent.</p> <p>You can design an API where <code>(key + content hash)</code> defines the operation identity. That is a valid policy. But then the key is no longer an idempotency key in the usual retry sense. It is part of a composite operation identifier. That needs to be obvious to the client.</p> <p>The dangerous version is the middle ground, where the client thinks it is safely retrying one operation and the server silently interprets the second request as another.</p> <h2>Hash the command, not the bytes</h2> <p>Raw byte comparison is usually too strict for JSON APIs. These two bodies should normally be equivalent:</p> <pre><code><span>{</span> <span> "amount"</span><span>: </span><span>"10.00"</span><span>,</span> <span> "currency"</span><span>: </span><span>"EUR"</span> <span>}</span></code></pre> <pre><code><span>{</span> <span> "currency"</span><span>: </span><span>"EUR"</span><span>,</span> <span> "amount"</span><span>: </span><span>"10.00"</span> <span>}</span></code></pre> <p>Field order and whitespace should not matter.</p> <p>Defaults are less obvious:</p> <pre><code><span>{</span> <span> "accountId"</span><span>: </span><span>"acc_1"</span><span>,</span> <span> "amount"</span><span>: </span><span>"10.00"</span><span>,</span> <span> "currency"</span><span>: </span><span>"EUR"</span> <span>}</span></code></pre> <p>versus:</p> <pre><code><span>{</span> <span> "accountId"</span><span>: </span><span>"acc_1"</span><span>,</span> <span> "amount"</span><span>: </span><span>"10.00"</span><span>,</span> <span> "currency"</span><span>: </span><span>"EUR"</span><span>,</span> <span> "channel"</span><span>: </span><span>"web"</span> <span>}</span></code></pre> <p>If <code>channel: "web"</code> is the server default, are these the same logical command? Maybe. Decide before hashing.</p> <p>Unknown fields are another trap. Suppose your API ignores unknown JSON fields. If the first request includes <code>"foo": "bar"</code> and the second does not, do you consider them the same? If unknown fields are truly ignored, perhaps yes. If they might become meaningful after a deploy, perhaps no.</p> <p>The practical rule is: hash the validated command, not the raw HTTP body.</p> <p>A reasonable flow is:</p> <ol> <li>Parse the request into a versioned request DTO or command.</li> <li>Normalize values your API treats as equivalent: amounts, enum casing, default fields, timestamp precision.</li> <li>Exclude transport-only metadata.</li> <li>Include path parameters and operation name.</li> <li>Include semantic headers if they affect the operation, such as API version.</li> <li>If a header only affects response shape, such as <code>Prefer: return=minimal</code>, decide whether it belongs in the command hash, the replay contract, or neither.</li> <li>Exclude <code>Authorization</code> and the idempotency key itself.</li> <li>Serialize canonically.</li> <li>Hash with a stable algorithm.</li> </ol> <p>For the payment example, the fingerprint might include:</p> <pre><code><span>operation: create_payment</span> <span>accountId: acc_1</span> <span>amount: 10.00</span> <span>currency: EUR</span> <span>merchantReference: invoice-7781</span> <span>channel: web</span> <span>apiVersion: 2026-05-01</span></code></pre> <p>Be careful with amounts, timestamps, generated defaults, locale-sensitive formatting, and fields added during deploys. The request hash is a contract. If you change how it is computed, old retries can start looking different.</p> <h2>The first insert decides who owns execution</h2> <p>Two identical requests hit two API instances at nearly the same time:</p> <pre><code><span>POST</span><span> /payments</span> <span>Idempotency-Key</span><span>:</span><span> abc-123</span></code></pre> <p>Same canonical command. Same tenant. Same endpoint.</p> <p>This implementation is broken even if every single-threaded test passes:</p> <pre><code><span>existing = find_by_key(key)</span> <span>if existing does not exist:</span> <span> create_payment()</span> <span> insert_idempotency_record()</span></code></pre> <p>Both requests can observe no existing row. Both can execute the side effect.</p> <p>If there is no atomic insert or unique constraint on the scoped key, two instances can both decide they own execution.</p> <p>The insert-first shape is:</p> <pre><code><span>insert into</span><span> idempotency_requests (tenant_id,</span> <span> operation_name,</span> <span> idempotency_key,</span> <span> request_hash,</span> <span> status</span><span>,</span> <span> created_at,</span> <span> updated_at,</span> <span> expires_at,</span> <span> locked_until)</span> <span>values</span><span> (:tenant_id,</span> <span> 'create_payment'</span><span>,</span> <span> :idempotency_key,</span> <span> :request_hash,</span> <span> 'IN_PROGRESS'</span><span>,</span> <span> now</span><span>(),</span> <span> now</span><span>(),</span> <span> now</span><span>() </span><span>+</span><span> interval </span><span>'24 hours'</span><span>,</span> <span> now</span><span>() </span><span>+</span><span> interval </span><span>'30 seconds'</span><span>) </span><span>on</span><span> conflict do nothing;</span></code></pre> <p>The exact syntax is database-specific. The important property is atomic ownership acquisition for <code>(tenant_id, operation_name, idempotency_key)</code>.</p> <p>Then:</p> <pre><code><span>if rows_inserted == 1:</span> <span> this request owns execution</span> <span>else:</span> <span> existing = load idempotency row</span> <span> if existing.request_hash != request_hash:</span> <span> return 409 IDEMPOTENCY_KEY_REUSED_WITH_DIFFERENT_REQUEST</span> <span> if existing.status == COMPLETED:</span> <span> return replay(existing.response_status, existing.response_body)</span> <span> if existing.status == IN_PROGRESS and existing.locked_until &gt; now():</span> <span> return 202 or 409 + Retry-After</span> <span> if existing.status == IN_PROGRESS and existing.locked_until &lt;= now():</span> <span> attempt recovery ownership</span> <span> # this must be atomic too</span> <span> if existing.status == UNKNOWN_REQUIRES_RECOVERY:</span> <span> trigger reconciliation or return pending/recovery response</span></code></pre> <p>Recovery ownership has to be acquired atomically too. Otherwise two retries can both decide the old owner is dead and both start recovery.</p> <p>In the simple local case, the owner can create the payment and complete the idempotency record in one transaction:</p> <pre><code><span>begin transaction</span> <span>insert idempotency row as IN_PROGRESS</span> <span>insert payment row pay_789</span> <span>insert outbox event PaymentCreated(pay_789)</span> <span>update idempotency row:</span> <span> status = COMPLETED</span> <span> resource_type = payment</span> <span> resource_id = pay_789</span> <span> response_status = 201</span> <span> response_body = {...}</span> <span>commit</span></code></pre> <p>That is the nice version: one database transaction covers the idempotency row, the business row, and the outbox event.</p> <p>External side effects change the shape. Holding a database transaction open while calling a provider is usually a bad idea. Committing before the provider call means your local state may say <code>IN_PROGRESS</code> while execution continues outside the transaction. If the process crashes there, a retry has to recover. This is where you need an operation state machine and a recovery worker, not just a request table.</p> <p>Redis <code>SET NX EX</code> is often proposed as the whole solution. At best, it is an execution guard:</p> <pre><code><span>SET idempotency:tenant_1:create_payment:abc-123 value NX EX 30</span></code></pre> <p>It can reduce duplicate concurrent execution. It is not durable memory of the operation outcome. If the Redis lock expires while the provider call is still running, another request can enter. If the process dies after the provider succeeds but before storing the response, the lock does not help the retry know what happened. Redis locks also need fencing or durable ownership if they protect downstream resources.</p> <p>Redis can be useful. It is not a substitute for remembering the operation outcome.</p> <h2>The provider timeout is where your guarantee ends</h2> <p>The failure path that matters is not exotic:</p> <ol> <li>API receives <code>POST /payments</code>.</li> <li>It inserts an idempotency row as <code>IN_PROGRESS</code>.</li> <li>It creates local payment <code>pay_789</code>.</li> <li>It calls a downstream payment provider.</li> <li>The provider receives the request and succeeds.</li> <li>The API times out, crashes, or loses the provider response.</li> <li>The client retries with the same key.</li> </ol> <p>If the provider received your request and your process died before recording the result, your database cannot infer whether money moved.</p> <p>A local state machine might look like this:</p> <pre><code><span>RECEIVED</span> <span>LOCAL_PAYMENT_CREATED</span> <span>PROVIDER_REQUEST_SENT</span> <span>PROVIDER_CONFIRMED</span> <span>COMPLETED</span> <span>UNKNOWN_REQUIRES_RECOVERY</span></code></pre> <p>The retry behavior depends on the state.</p> <p>If the retry finds <code>COMPLETED</code>, replay.</p> <p>If it finds a fresh <code>PROVIDER_REQUEST_SENT</code>, return <code>202 Accepted</code>, <code>409 Conflict</code> with <code>Retry-After</code>, or block briefly and wait for completion. Pick one behavior and document it. Clients need to know whether to retry, poll, or wait.</p> <p>If it finds stale <code>PROVIDER_REQUEST_SENT</code>, do not create <code>pay_790</code>. Do not call the provider with a new identity. Recover using the stable downstream operation ID:</p> <pre><code><span>payment id: pay_789</span> <span>provider idempotency key: provider_payment_pay_789</span></code></pre> <p>A recovery worker or retrying request can then:</p> <ol> <li>acquire recovery ownership for <code>pay_789</code></li> <li>query the provider by <code>provider_payment_pay_789</code>, if the provider supports it</li> <li>if confirmed, mark the provider operation confirmed</li> <li>mark the idempotency record <code>COMPLETED</code></li> <li>store or reconstruct the response</li> <li>replay the response or return a documented final status</li> <li>if the provider cannot answer, mark <code>UNKNOWN_REQUIRES_RECOVERY</code></li> </ol> <p>If the provider has no idempotency key and no query API, your system has an operational gap. You may still choose to accept it, but the local idempotency table is not protecting the external effect. It only prevents duplicate local request handling.</p> <p>For payment-like operations, the client’s idempotency key is often not the exact key sent downstream. The downstream call needs a stable identity that survives retries, crashes, and reconciliation. Otherwise the second local attempt is just a second provider attempt.</p> <p>I would avoid <code>425 Too Early</code> unless your API already has a specific reason to use it. Most clients will not handle it specially. <code>202 Accepted</code>, <code>409 Conflict</code> with <code>Retry-After</code>, or an operation-status endpoint are easier to explain.</p> <h2>Replay is a contract, not a convenience</h2> <p>For a completed idempotent request, replaying the same status and body is the least surprising behavior:</p> <pre><code><span>HTTP</span><span>/</span><span>1.1</span><span> 201</span><span> Created</span> <span>Idempotent-Replayed</span><span>:</span><span> true</span> <span>Content-Type</span><span>:</span><span> application/json</span></code></pre> <pre><code><span>{</span> <span> "paymentId"</span><span>: </span><span>"pay_789"</span><span>,</span> <span> "status"</span><span>: </span><span>"PENDING"</span><span>,</span> <span> "accountId"</span><span>: </span><span>"acc_1"</span><span>,</span> <span> "amount"</span><span>: </span><span>"10.00"</span><span>,</span> <span> "currency"</span><span>: </span><span>"EUR"</span><span>,</span> <span> "merchantReference"</span><span>: </span><span>"invoice-7781"</span> <span>}</span></code></pre> <p>A custom response header such as <code>Idempotent-Replayed: true</code> can help debugging. I would not make clients depend on it.</p> <p>Reconstructing responses from current resource state is tempting:</p> <pre><code><span>load payment pay_789</span> <span>return current representation</span></code></pre> <p>But suppose the first response was:</p> <pre><code><span>{</span> <span> "paymentId"</span><span>: </span><span>"pay_789"</span><span>,</span> <span> "status"</span><span>: </span><span>"PENDING"</span> <span>}</span></code></pre> <p>and the retry happens ten minutes later, after settlement:</p> <pre><code><span>{</span> <span> "paymentId"</span><span>: </span><span>"pay_789"</span><span>,</span> <span> "status"</span><span>: </span><span>"SETTLED"</span> <span>}</span></code></pre> <p>That may be useful, but it is not a replay. It is a fresh read of the resource. If your API contract says idempotent retries return the original creation result, you need to store enough to do that.</p> <p>Schema changes make this worse.</p> <p>Version 2 response:</p> <pre><code><span>{</span> <span> "paymentId"</span><span>: </span><span>"pay_789"</span><span>,</span> <span> "status"</span><span>: </span><span>"PENDING"</span> <span>}</span></code></pre> <p>Version 3 response:</p> <pre><code><span>{</span> <span> "id"</span><span>: </span><span>"pay_789"</span><span>,</span> <span> "state"</span><span>: </span><span>"PENDING"</span><span>,</span> <span> "createdAt"</span><span>: </span><span>"2026-05-07T10:00:00Z"</span> <span>}</span></code></pre> <p>If a generated client retries after a deploy, should it receive the stored v2 response or a reconstructed v3 response? Both can be defensible. They are different contracts.</p> <p>A common compromise is to store:</p> <pre><code><span>resource_type = payment</span> <span>resource_id = pay_789</span> <span>response_status = 201</span> <span>response_schema_version = v2</span></code></pre> <p>and store full response bodies only for endpoints where exact replay matters. If you store bodies, treat the idempotency table like sensitive data storage, not like a harmless cache.</p> <h2>Your queue consumer has the same bug</h2> <p>HTTP gets most of the attention because the header is visible. A lot of duplicate side effects happen later, in consumers, outbox publishers, inbox processors, and notification workers.</p> <p>Suppose the payment service publishes:</p> <pre><code><span>{</span> <span> "eventId"</span><span>: </span><span>"evt_100"</span><span>,</span> <span> "type"</span><span>: </span><span>"PaymentCreated"</span><span>,</span> <span> "paymentId"</span><span>: </span><span>"pay_789"</span><span>,</span> <span> "accountId"</span><span>: </span><span>"acc_1"</span><span>,</span> <span> "amount"</span><span>: </span><span>"10.00"</span><span>,</span> <span> "currency"</span><span>: </span><span>"EUR"</span> <span>}</span></code></pre> <p>A consumer receives it twice. That should not send two emails, create two ledger entries, or notify a provider twice.</p> <p>The dedupe key might be the event ID, message ID, operation ID, aggregate ID plus version, or a business key such as <code>ledger_payment_pay_789</code>. The right answer depends on the side effect.</p> <p>A consumer inbox table might be:</p> <pre><code><span>consumer_inbox</span> <span>- consumer_name</span> <span>- message_id</span> <span>- status</span> <span>- processed_at</span> <span>- error_code</span> <span>unique(consumer_name, message_id)</span></code></pre> <p>But marking the message processed is not trivial.</p> <p>If you mark it processed before sending the email and then crash, the retry skips the email forever. If you send the email before marking it processed and then crash, the retry may send it again. The usual answer is to make the side effect durable before sending it: insert an email notification row with a unique key, then have a sender process that row.</p> <p>Ledger entries often have a natural idempotency key:</p> <pre><code><span>unique(ledger_entry_type, source_payment_id)</span></code></pre> <p>Processing <code>PaymentCreated(pay_789)</code> twice attempts to create the same ledger entry twice, and the second attempt resolves to the existing entry.</p> <p>Many production queue integrations are effectively at-least-once from the consumer’s point of view. Even when the broker advertises stronger delivery semantics, your business side effects still need deduplication. Exactly-once delivery is not exactly-once business effect. The latter usually comes from durable operation IDs, unique constraints, idempotent writes, and recovery paths.</p> <p>Outbox/inbox is the usual shape:</p> <pre><code><span>same database transaction:</span> <span> insert payment row pay_789</span> <span> insert outbox event PaymentCreated(pay_789)</span> <span>publisher:</span> <span> reads unpublished outbox event</span> <span> publishes event with eventId</span> <span> marks outbox event published</span> <span>consumer:</span> <span> deduplicates by eventId or business operation key</span> <span> writes side effect behind a unique constraint</span></code></pre> <p>Idempotency prevents some duplicates. It does not remove poison messages, broken providers, dead-letter handling, or recovery work.</p> <h2>Expiry is part of the API contract</h2> <p>Idempotency records cannot usually live forever.</p> <p>If the server promises a 24-hour idempotency window, then a retry after 25 hours may create a new operation. That may be acceptable. It may also surprise clients that queue retries for days. The replay window is a product/API decision, not just a cleanup setting.</p> <p>A completed record might be:</p> <pre><code><span>created_at: 2026-05-07T10:00:00Z</span> <span>expires_at: 2026-05-08T10:00:00Z</span> <span>status: COMPLETED</span></code></pre> <p>After expiry, you might delete the response body but retain metadata longer:</p> <pre><code><span>idempotency_key</span> <span>scope</span> <span>operation_name</span> <span>request_hash</span> <span>resource_id</span> <span>created_at</span> <span>expires_at</span></code></pre> <p>That supports diagnostics without retaining sensitive response payloads.</p> <p>Stale <code>IN_PROGRESS</code> needs separate handling:</p> <pre><code><span>status: IN_PROGRESS</span> <span>resource_id: pay_789</span> <span>updated_at: 2026-05-07T10:00:00Z</span> <span>locked_until: 2026-05-07T10:00:30Z</span> <span>now: 2026-05-07T10:45:00Z</span></code></pre> <p>A retry that sees this should not blindly execute again. It should acquire recovery ownership, inspect <code>pay_789</code>, query downstream if needed, and move the operation to <code>COMPLETED</code>, <code>FAILED_RETRYABLE</code>, or <code>UNKNOWN_REQUIRES_RECOVERY</code>.</p> <p>Cleanup jobs should not remove in-progress records just because they are old. An old in-progress row may mean a stuck worker, a process crash, or an operation waiting for reconciliation. Deleting it can allow a duplicate side effect.</p> <p>Bad cleanup:</p> <pre><code><span>delete</span> <span>from</span><span> idempotency_requests</span> <span>where</span><span> expires_at </span><span>&lt;</span><span> now</span><span>();</span></code></pre> <p>Better options include deleting in small batches, partitioning by <code>expires_at</code>, dropping old time partitions after the replay window, and keeping separate retention policies for response bodies and metadata.</p> <p>Replay count is mostly capacity planning. Different-body reuse, stale <code>IN_PROGRESS</code> rows, expired retries, and unknown states are the metrics that find bugs.</p> <pre><code><span>idempotency.replay.count</span> <span>idempotency.conflict.different_request.count</span> <span>idempotency.in_progress.age.max</span> <span>idempotency.expired_retry.count</span> <span>idempotency.unknown_state.count</span></code></pre> <h2>Failure replay is a policy decision</h2> <p>The dangerous mistake is treating every failure as either “safe to retry” or “completed”.</p> <p>Pure syntactic validation failures usually do not need idempotency storage. If the JSON is malformed or a required field is missing, repeating the request will fail again.</p> <p>Business rejections are different. If the decision depends on mutable state, such as balance, inventory, account status, or fraud rules, decide whether the first decision is binding for that idempotency key or whether the client must retry with a new key.</p> <p>A deterministic rejection might be replayable:</p> <pre><code><span>{</span> <span> "errorCode"</span><span>: </span><span>"INSUFFICIENT_FUNDS"</span><span>,</span> <span> "message"</span><span>: </span><span>"The account has insufficient funds for this payment."</span> <span>}</span></code></pre> <p>But if the account balance changes five seconds later, replaying that rejection may or may not be what your API intends.</p> <p>Authentication failures should not create idempotency records. For authorization failures, be careful: a retry must still resolve to the same scope/principal that created the original record. Do not let one caller use another caller’s idempotency key to discover whether an operation happened. Whether later permission changes block replay of an already completed authorized operation is a product and security decision.</p> <p>Rate limits usually should not be recorded as completed idempotent outcomes. A retry later might be allowed.</p> <p>Server error before side effects can often allow retry. Server error after side effects is dangerous. If you created the payment but failed to serialize the response, the retry should not create another payment. If you called a provider and lost the response, the retry needs recovery state, not optimism.</p> <p>A practical internal status set might be:</p> <pre><code><span>IN_PROGRESS</span> <span>COMPLETED</span> <span>FAILED_REPLAYABLE</span> <span>FAILED_RETRYABLE</span> <span>UNKNOWN_REQUIRES_RECOVERY</span> <span>EXPIRED</span></code></pre> <p>Do not expose every internal state directly. But internally, pretending every failure is either “done” or “not done” makes recovery harder.</p> <h2>When one transaction cannot cover the operation</h2> <p>The useful distinction is not monolith versus microservices. It is whether one durable transaction can cover the operation.</p> <p>If one database transaction can cover the idempotency row, payment row, and outbox record, the local part is straightforward:</p> <pre><code><span>insert idempotency row</span> <span>insert payment row</span> <span>insert outbox event</span> <span>mark idempotency completed</span> <span>commit</span></code></pre> <p>The publisher can retry outbox delivery. Consumers deduplicate by event ID or business operation key. The local write path is much easier to reason about.</p> <p>When side effects cross boundaries, every boundary that can repeat work needs its own duplicate-suppression rule.</p> <p>An upstream API accepting <code>Idempotency-Key: abc-123</code> can prevent duplicate HTTP payment creation requests at the edge. It does not automatically prevent duplicate ledger entries, duplicate notifications, duplicate provider calls, or duplicate read-model updates.</p> <p>A better model is to maintain stable operation identities:</p> <pre><code><span>client idempotency key: abc-123</span> <span>payment operation id: payop_456</span> <span>payment id: pay_789</span> <span>ledger entry id: ledger_payment_pay_789</span> <span>email dedupe key: receipt_payment_pay_789</span> <span>provider idempotency key: provider_payment_pay_789</span></code></pre> <p>The names do not matter. The point is that each side effect has a durable identity appropriate to that side effect.</p> <p>In active-active multi-region deployments, a region-local idempotency table only protects retries that land in the same region. You either need to route all requests for the same scoped key to a home region, use a strongly consistent shared store for idempotency records, or rely on downstream business constraints that survive cross-region races. Async replication alone can allow two regions to accept the same key before either sees the other write.</p> <p>For high-throughput APIs, the idempotency table can become a hot path. Response bodies can become expensive. Cleanup can compete with traffic. Partition by tenant, hash, or time if needed. Know your replay window. Do not make a global table the bottleneck unless the duplicate harm justifies it.</p> <h2>When not to build a general idempotency layer</h2> <p>The cost is not the header. The cost is the durable memory and recovery behavior behind it.</p> <p>Do not build a payment-grade idempotency layer for an admin action where a duplicate is harmless and visible.</p> <p>For read-only operations, idempotency keys usually add noise.</p> <p>If a duplicate analytics event costs almost nothing and can be corrected downstream, a heavy idempotency table may be the wrong trade.</p> <p>For some operations, a business key is better than a random key:</p> <pre><code><span>unique(account_id, merchant_reference)</span></code></pre> <p>If the business rule is “there can be only one payment per merchant reference per account,” that constraint catches duplicates even when the client retries with a new random key by mistake. Random idempotency keys only help when the client reuses the same key for retries.</p> <p>For other operations, change the resource model:</p> <pre><code><span>PUT</span><span> /accounts/acc_1/settings/default-currency</span></code></pre> <pre><code><span>{</span> <span> "currency"</span><span>: </span><span>"EUR"</span> <span>}</span></code></pre> <p>Repeating that request leaves the setting as EUR. You still need to think about side effects, but the operation shape is helping you.</p> <p>Client-generated keys are useful when the client can identify a retry of the same operation. Properly generated random keys are usually enough; timestamp-only keys, counters, and keys derived from sensitive data are not. Scope the key to the caller and operation, for example <code>(tenant_id, operation_name, idempotency_key)</code>, so a bad client only collides with itself. If clients generate a new key on every attempt, you need a business key or a server-created operation resource.</p> <p>Use the amount of harm caused by duplicate side effects, the likelihood of retries, and the difficulty of detecting duplicates after the fact to decide how much machinery you need.</p> <p>If duplicates move money, notify humans, call providers, consume scarce inventory, or corrupt accounting, spend the design effort. If duplicates are harmless, rare, and easy to clean up, use a smaller mechanism.</p> <h2>Failure modes worth testing</h2> <p>Here are tests I would rather see than a dozen happy-path unit tests.</p> <h3>Same key, same canonical command, completed</h3> <p>First request creates the payment:</p> <pre><code><span>POST</span><span> /payments</span> <span>Idempotency-Key</span><span>:</span><span> abc-123</span></code></pre> <p>returns:</p> <pre><code><span>201 Created</span></code></pre> <p>with <code>paymentId = pay_789</code>.</p> <p>Second request with the same canonical command and key returns the same stored result or documented equivalent. It does not create <code>pay_790</code>. It does not publish a second <code>PaymentCreated</code> event.</p> <h3>Same key, different canonical command</h3> <p>First request:</p> <pre><code><span>{</span> <span> "amount"</span><span>: </span><span>"10.00"</span><span>,</span> <span> "currency"</span><span>: </span><span>"EUR"</span> <span>}</span></code></pre> <p>Second request:</p> <pre><code><span>{</span> <span> "amount"</span><span>: </span><span>"100.00"</span><span>,</span> <span> "currency"</span><span>: </span><span>"EUR"</span> <span>}</span></code></pre> <p>Same key.</p> <p>Expected behavior: reject with a stable machine-readable idempotency conflict. Log and count it.</p> <h3>Two concurrent identical requests</h3> <p>Start two requests at the same time with the same key and same command.</p> <p>Expected behavior: one wins execution. The other sees <code>IN_PROGRESS</code>, waits and replays, or returns a retry-later response. The side effect executes once.</p> <p>If this test passes without a unique constraint or atomic insert, be suspicious of the test.</p> <h3>Timeout after downstream success</h3> <p>Simulate provider success and then crash before the client receives the response.</p> <p>Expected behavior: the retry should not call the provider with a new operation identity. It should find local completed state, query provider idempotent state, or move into recovery.</p> <h3>Duplicate message from a queue</h3> <p>Deliver <code>PaymentCreated(pay_789)</code> twice.</p> <p>Expected behavior: one ledger entry, one email notification, one provider notification. If the first attempt fails halfway through, the retry should complete missing durable work without duplicating completed work.</p> <h3>Expired or stale state</h3> <p>Retry after the idempotency record expired. Retry while the record is stale <code>IN_PROGRESS</code>. Retry after response schema changed. Retry from another region if your deployment allows it.</p> <p>These are not exotic cases. They are the normal edges of retrying over networks.</p> <h2>Checklist before shipping</h2> <ul> <li>Reject same scoped key plus different canonical command.</li> <li>Use a unique constraint or atomic insert on the scoped key.</li> <li>Hash the validated command, not raw JSON bytes.</li> <li>Treat <code>IN_PROGRESS</code> as API-visible behavior.</li> <li>Define fresh, stale, completed, retryable failure, replayable failure, and unknown states.</li> <li>Store enough response data to satisfy your replay contract.</li> <li>Make downstream calls idempotent too, or have reconciliation.</li> <li>Use outbox/inbox patterns where events and queues are involved.</li> <li>Do not mark messages processed before their durable side effects exist.</li> <li>Define the idempotency window as part of the API contract.</li> <li>Retain metadata separately from sensitive response bodies if needed.</li> <li>Test concurrent duplicates, timeout after downstream success, partial failure, expiry, and schema-change replay.</li> <li>Monitor different-body reuse, stale <code>IN_PROGRESS</code>, expired retries, unknown states, and replay rates.</li> </ul> <h2>The second request is not a repeat until proven</h2> <p>The easy version of idempotency remembers that a key was seen.</p> <p>The useful version remembers what the key meant.</p> <p>For <code>POST /payments</code>, that means remembering the scoped operation, the canonical command, the execution state, the resulting resource or response, the expiry window, and enough failure state to avoid turning uncertainty into duplicate side effects.</p> <p>The second request may be a retry. It may be a different operation wearing the same key. It may be racing the first request. It may arrive after the provider succeeded but your process failed. It may arrive after your cleanup job deleted the only memory of what happened.</p> <p>The server has to prove which case it is.</p> <p>The key is not the guarantee. The guarantee is that the server remembers the first operation precisely enough to replay it, reject a mismatch, or recover instead of guessing.</p> </div> https://twitter.com/jarredsumner/status/2053047748191232310Sat, 09 May 2026 10:12:55 +0000https://news.ycombinator.com/item?id=48073680<a href="https://news.ycombinator.com/item?id=48073680">Comments</a>https://www.amusingplanet.com/2026/05/emerich-juettner-one-dollar.htmlThu, 07 May 2026 12:40:35 +0000https://news.ycombinator.com/item?id=48048684<a href="https://news.ycombinator.com/item?id=48048684">Comments</a><article class="post postContainer" data-postid="6312347561510666639" morss_own_score="9.848139711465452" morss_score="15.793791885378495"> <h1> Emerich Juettner: The One Dollar Counterfeiter </h1> <a href="https://www.blogger.com/profile/15000427721236718033" title="author profile"> Kaushik Patowary </a> <span title="2026-05-04T20:28:00+05:30"> May 4, 2026 </span> <div id="adsense-target" morss_own_score="2.977920883164673" morss_score="83.51792088316468"><p>In fiction as well as in real life, counterfeiters have always been portrayed as master forgers and artists who reproduced banknotes with astonishing precision. They were often shown as vast criminal enterprises and gangsters who destabilized economies with fake currency. Then there was Emerich Juettner, a frail old immigrant living alone in a shabby New York apartment, quietly printing one-dollar bills on a cheap hand press. He was, by almost every conventional standard, terrible at counterfeiting.</p> <p>And yet he succeeded for nearly a decade. By the time the United States Secret Service finally caught Juettner in 1948, he had become kind of folk hero. The press adored him and the public sympathized with him. A Hollywood film would soon immortalize him under the nickname “Mister 880,” a reference to his Secret Service case number.</p> <p><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwXpxWbUXh8JQ77qOz95jEMSz1Om4dBMMX9KkNU6q9VOgoSd0c9x09Iz2QFuGHTrW30gZQw-LIkr0LenTF5plIieA_W3peYZAD5JgtXMCAprBF11XeeREsHq8OU1tj5-6Ruz8CYFPEZXmbe-RzX6CvogO8X9ylGDDFCr-iATAGOOUlkSgsuCDx-WCo2ig/s1600/edward-juettner-2%20.jpg"></p><p>Emerich Juettner (also known as Edward Mueller) hardly looked like the sort of person who would trouble federal agents. Born in Austria-Hungary in 1876, he emigrated to the United States and spent most of his life drifting through modest occupations. Initially he worked as a picture frame gilder before marrying Florence LeMein in 1902 at the age of 26. After the birth of his son and daughter, Juettner began working as a maintenance man and building superintendent in New York's Upper East Side. His job allowed him and his family to live rent free in the basement of the building where he worked.</p> <p>In 1937, Juettner’s wife died and the sixty-year-old suddenly found himself living alone in New York City. He then became a junk collector.</p> <p>Juettner bought a used, two-wheel pushcart and spent long days ambling about the streets of New York picking up the discarded goods of city dwellers and selling off the occasional find to a wholesale dealer. But Juettner’s earnings were sporadic and he was barely putting food in his mouth. This forced him to look into another way of making a living.</p> <p>In his youth, Juettner had learned metal engraving. He had also dabbled in photography. Combining these two skills, in November 1938, he began to make counterfeit one-dollar bills. He snapped pictures of a $1 bill, transferred the images to a pair of zinc plates, and then meticulously filled in small details of the bill by hand. </p> <p>Juettner’s fake notes were laughably crude. He produced them using inexpensive materials and primitive techniques in his apartment kitchen. The paper was wrong. The ink was poor. The engraving lacked detail. The bills often appeared slightly blurred or uneven. Some notes even had spelling errors.</p> <p><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZqL7bBTv2XNe1qnHmCuYJ1uH_2UK91pjMAMT3Eca7zf4devM7AJOw-Sr-QRz4Yb6n01nj1K37dwL4zJJllXJTJI86qakq2-CpKwF_BCY5Og7_0MQFJexX9IUV1sA8hEjdmHXsDjAxbLvs4rCeshdc30MQyL8cPmWskwp8mm29mjvtP6kaQeDzrM4eqd0/s1600/edward-juettner-1%20.jpg"><br><em>Scenes from the movie Mister 880 dramatizing Juettner’s counterfeit operation.</em></p> <p>They were not the kind of counterfeit currency that could fool bankers or cashiers under careful inspection. But Juettner understood that almost nobody examines a one-dollar bill closely.</p> <p>Large-denomination counterfeits attract scrutiny because people expect fraud where substantial sums are involved. A suspicious twenty-dollar bill may be held to the light, checked for texture, or compared against a real note. But a worn one-dollar bill passing quickly between customers in a busy shop? Few people cared enough to look carefully.</p> <p>Juettner exploited that indifference brilliantly. He also operated on an extremely small scale. Instead of flooding the market, he released only a handful of fake bills at a time. The notes circulated quietly through diners, bars, street vendors, and small stores, disappearing into the immense ocean of American currency.</p> <p>The Secret Service became aware that poor-quality counterfeit dollar bills were appearing in New York, but the case puzzled investigators. Professional counterfeiters usually aimed for large profits. These bills, by contrast, were amateurish and limited in number. Whoever was making them seemed content merely to survive.</p> <p>The United States Secret Service, which investigates counterfeiting, opened a file on the mysterious forger. The investigation was assigned case number 880. Over the years, agents tried in vain to locate the culprit. They handed out some 200,000 warning placards at 10,000 stores. They tracked down dozens of folks who’d spent the bills and interviewed them.</p> <p>But Juettner remained elusive. He was careful not to draw attention. He never attempted large transactions. He worked alone, and he printed only tiny amounts.</p> <p>10 years went by and the search for Mister 880 turned into the largest and most expensive counterfeit investigation in Secret Service history.</p> <p>Then in January of 1948, some schoolboys playing around in a vacant lot in the Upper West Side uncovered a couple of zinc engraving plates buried in the snow. They also found “30 funny-looking dollar bills.”</p> <p>A week later, one boy’s father caught him playing poker with a strange bill and turned it into the police, who handed it over to the Secret Service. They soon tracked down the lot where the boys found the plates and learned that a few weeks earlier, there had been a fire in a bordering apartment. The firefighters had entered to find the place full to the brim with junk, and had thrown them out the window into the alley to make room. The Secret Service had found their mystery man, Mister 880.</p> <p><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWHgUVP1LlN78c3IR8Oo-eK99g2d70kXAuzSC5zXC_0GiET_4_98x0Sdub7iWd_nVjfk6JA2HJrJOwHOucUEHJxiUE6-gga5sbxgKnkXzGQeUgDUXtqQ6lrJL2afeNGgzsjIT3KCLMTx_qvY8LlwgmWagUMUxcKhTMoZ5iW7jhxV_dvxt9GVmkPD2JJDk/s1600/edward-juettner-3%20.jpg"><br><em>Emerich Juettner is interrogated by authorities after his arrest.</em></p> <p>Emerich Juettner was arrested. When questioned about his crimes, he nonchalantly admitted to them. Juettner said that he had been making them for 9 to 10 years, and that he never gave more than one bill to any one person, “so nobody ever lost more than $1.”</p> <p>Under ordinary circumstances, a federal counterfeiting arrest would have generated little sympathy. But the story of Emerich Juettner struck the public imagination immediately. Here was an old man surviving in poverty by printing crude one-dollar bills one at a time. He was not violent, greedy, or glamorous.</p> <p>At trial, Juettner admitted his activities openly. The judge sentenced him to only a year and a day in prison, and he was paroled after 4 months. He was also made to pay a fine of $1. It has been agreed that Juettner’s complete lack of greed was the rationale behind the light sentence.</p> <p>After his release, Juettner briefly achieved celebrity status. His notoriety became so widespread that Hollywood adapted the story into the 1950 film <em>Mister 880</em>, directed by Edmund Goulding. Eventually, Juettner made more money from the release of <em>Mister 880</em> than he had made by counterfeiting.</p> <p>Juettner returned to a life of normalcy, and lived out the rest of his days in the suburbs of Long Island, where he died in 1955, at the age of 79.</p> <p><strong>References:</strong></p> <p> # The 70-year-old retiree who became America’s worst counterfeiter. <a href="https://thehustle.co/worst-counterfeiter-in-history-mr-880">Hustle</a></p><p> # Finding ‘Mr. 880’: The case of the $1 counterfeit. <a href="https://www.nydailynews.com/2011/04/03/finding-mr-880-the-case-of-the-1-counterfeit/">NY Daily News</a></p></div> </article> https://github.com/imtomt/ymawkySun, 10 May 2026 03:01:44 +0000https://news.ycombinator.com/item?id=48080587<a href="https://news.ycombinator.com/item?id=48080587">Comments</a><article class="markdown-body entry-content container-lg" itemprop="text" morss_own_score="9.934640522875817" morss_score="84.50964052287583"><p><a href="https://github.com/imtomt/ymawky/blob/main/docs/ymawky.png"><img src="https://github.com/imtomt/ymawky/raw/main/docs/ymawky.png"></a></p> <h1><em>ymawky</em> -- web server in ARM assembly</h1> <p>This is <em>ymawky</em> (yuh maw kee), a web server written entirely in ARM64 assembly. ymawky is a syscall-only, no libc, fork-per-connection web server written by hand. While it is developed for MacOS, I've tried to make it as portable as possible -- <em>however</em>, it's likely you will still need to make some <del>(hopefully minor)</del> Significant tweaks to get this to run on Linux/other Unix systems. See <a href="https://github.com/imtomt/ymawky#implementation-notes">Implementation Notes</a> for more details.</p> <h2>Building</h2> <p>Requires Xcode Command Line Tools. Install with <code>xcode-select --install</code>. ymawky only runs on apple silicon (arm64).</p> <p>Run <code>make</code> to build.</p> <p>Ensure there is a <code>www/</code> directory next to the <code>ymawky</code> executable. That's the document root where <em>ymawky</em> searches for files. <code>GET</code> with an empty filename (<code>GET /</code>) will search for <code>www/index.html</code>, so you might want to make sure there's an <code>index.html</code> as well.</p> <p><em>ymawky</em> will try to serve static error pages when a client's request results in error, eg 404. The pages it searches for in <code>err/(code).html</code>, so ensure <code>err/</code> exists alongisde <code>ymawky</code> and <code>www/</code>. See <a href="https://github.com/imtomt/ymawky#configuration">Configuration</a> to modify the default file and docroot.</p> <h2>Running</h2> <ul> <li><code>./ymawky</code> to start running the web server on <code>127.0.0.1:8080</code>.</li> <li><code>./ymawky [port]</code> to start running the web server on <code>127.0.0.1:[port]</code></li> <li><code>./ymawky [literally-any-character-other-than-0-9]</code> to start running the web server on 127.0.0.1:8080 in debug mode. Debug mode disables forking, and makes ymawky only handle one request. (<em>I needed to do this because <code>lldb</code> wasn't letting me debug the children, ugh.</em>)</li> </ul> <p>Unfortunately, while custom ports are supported, custom addresses are not. as of right now, ymawky can only run on <code>127.0.0.1</code>. This is solely because I haven't implemented it -- but if you'd like to consider this a safety feature, then I guess it could be intentional.</p> <p>To see ymawky in action, start running ymawky with <code>./ymawky [port]</code>. Then open your web browser of choice (or use curl), and visit <code>127.0.0.1:8080/</code> or <code>127.0.0.1:8080/pretty/index.html</code>. Bask in the warmth of assembly.</p> <h2>What can it do?</h2> <p>ymawky is a static-file web server. It doesn't support server-side code to generate content on-the-fly, or more advanced URL parsing, such as <code>/search?query=term</code>. That's not to say it's non-functional, though.</p> <ul> <li>Supported HTTP methods: <ul> <li>GET</li> <li>PUT</li> <li>DELETE</li> <li>OPTIONS</li> <li>HEAD</li> </ul> </li> <li>Basic protection from slowloris-like Denial of Service attacks</li> <li>Decodes % hex encoding, eg, <code>%20</code> decodes to a space in filenames, and <code>%61</code> decodes to <code>a</code></li> <li>Smart path traversal detection and prevention. Blocks <code>..</code> from traversing paths, while not disallowing multiple periods when they're part of a file: <ul> <li><code>GET /../../../etc/passwd</code> -&gt; <code>403 Forbidden</code></li> <li><code>GET /ohwell...txt</code> -&gt; <code>200 OK</code></li> <li><code>GET /../src/ymawky.S</code> -&gt; <code>403 Forbidden</code></li> <li><code>GET /hehe..txt</code> -&gt; <code>200 OK</code></li> </ul> </li> <li>Automatically prepends <code>www/</code> to requested files. <code>GET /index.html</code> will retrieve <code>www/index.html</code></li> <li>Empty <code>GET /</code> requests default to <code>GET www/index.html</code></li> <li><code>PUT</code> requests support uploads of up to 1GiB, though this can be configured for larger files</li> <li><code>PUT</code> is atomic due to writing to a temporary file then renaming, allowing concurrent <code>PUT</code> requests without leaving partially-written files</li> <li><code>Content-Length:</code> parsing and verification in <code>PUT</code> requests</li> <li>MIME type detection, giving <code>Content-Type</code> in the response header with the corresponding MIME type</li> <li>Accepts <code>Range: bytes=</code> ranges in GET requests, supporting full ranges <code>bytes=X-N</code>, suffix ranges <code>bytes=-N</code>, and open-ended ranges <code>bytes=X-</code>. Video scrubbing is well supported</li> <li>Basic HTTP version parsing. Requests need to specify <code>HTTP/1.1</code> or <code>HTTP/1.0</code>, and if requesting <code>HTTP/1.1</code>, a <code>Host:</code> field needs to be present in the header. Currently, ymawky doesn't do anything with Host, but per RFC 9112 Section 3.2, the Header must be sent</li> <li>Serves custom HTML pages for error codes, such as 404, or 500. Look in the <code>err/</code> directory for an example</li> <li>If the requested resource is a directory, list all files and subdirs in the directory. Note that this excludes www/ (or whatever your docroot is): GET / will always search for index.html if no file is given.</li> </ul> <h2>"Safety"</h2> <p>This is a web server written entirely by-hand in ARM64 assembly as a fun project. It's probably got a lot of vulnerabilities I'm unaware of. However, I did do my best to make it safer. Here are some safety precautions ymawky takes.</p> <ul> <li>Rejects paths &gt;= PATH_MAX (4096 bytes)</li> <li>Reject any paths that include path traversal -- <code>/../..</code></li> <li>Reject any requests that do not contain a path within 16 bytes</li> <li>Confined to <code>www/</code>. Any path requested gets <code>www/</code> prepended to it</li> <li>Rejects any path containing symlinks, with O_NOFOLLOW_ANY</li> <li>PUT writes to a temporary file, <code>www/.ymawky_tmp_&lt;pid&gt;</code>. Upon successfully receiving the whole file, this temporary file is then renamed to the requested filename. This prevents partial or corrupted PUT requests from overwriting existing files.</li> <li>Reject any requests whose path starts with <code>www/.ymawky_tmp_</code>. This prevents someone from <code>GET</code>ing a temporary file, and prevents someone from sending <code>PUT /.ymawky_tmp_4533</code> or something.</li> <li>Must receive data within 10 seconds. If it's slower, the connection will close. If the entire header is not received within 10 seconds total, the connection will be closed. This is to prevent slowloris-like attacks.</li> </ul> <h2>HTTP Status Codes</h2> <p>ymawky currently supports and can reply with the following status codes:</p> <ul> <li><code>200 OK</code></li> <li><code>201 Created</code></li> <li><code>204 No Content</code></li> <li><code>206 Partial Content</code></li> <li><code>400 Bad Request</code></li> <li><code>403 Forbidden</code></li> <li><code>404 Not Found</code></li> <li><code>408 Request Timeout</code></li> <li><code>409 Conflict</code></li> <li><code>411 Length Required</code></li> <li><code>413 Content Too Large</code></li> <li><code>414 URI Too Long</code></li> <li><code>416 Range Not Satisfiable</code></li> <li><code>418 I'm a teapot</code></li> <li><code>431 Request Header Fields Too Large</code></li> <li><code>500 Internal Server Error</code></li> <li><code>501 Not Implemented</code></li> <li><code>503 Service Unavailable</code></li> <li><code>505 HTTP Version Not Supported</code></li> <li><code>507 Insufficient Storage</code></li> </ul> <p>Custom HTML pages will be served alongside the error codes (400+). These HTML files are located in <code>err/(code).html</code>. You can use <code>build_err_pages.sh</code> to create a page for each code, with different text at your leisure. Edit the source code of <code>build_err_pages.sh</code> to modify the text per-page, and modify <code>err/template.html</code> to modify the base template. In <code>err/template.html</code>:</p> <ul> <li><code>{{CODE}}</code> - HTTP Code: eg, 404</li> <li><code>{{TITLE}}</code> - Title text: eg, "Not Found"</li> <li><code>{{MSG}}</code> - Custom message: eg, "the rats ate this page"</li> </ul> <h2>MIME Types</h2> <p>MIME types are detected by analyzing the file extension. The following MIME types are recognized.</p> <p>Web-related files:</p> <ul> <li><code>.html</code> -&gt; <code>text/html; charset=utf-8</code></li> <li><code>.htm</code> -&gt; <code>text/html; charset=utf-8</code></li> <li><code>.css</code> -&gt; <code>text/css; charset=utf-8</code></li> <li><code>.csv</code> -&gt; <code>text/csv; charset=utf-8</code></li> <li><code>.xml</code> -&gt; <code>text/xml; charset=utf-8</code></li> <li><code>.js</code> -&gt; <code>text/javascript; charset=utf-8</code></li> <li><code>.json</code> -&gt; <code>application/json</code></li> <li><code>.wasm</code> -&gt; <code>application/wasm</code></li> <li><code>.mjs</code> -&gt; <code>text/javascript; charset=utf-8</code></li> <li><code>.map</code> -&gt; <code>application/json</code></li> </ul> <p>Image files:</p> <ul> <li><code>.png</code> -&gt; <code>image/png</code></li> <li><code>.jpg</code> -&gt; <code>image/jpeg</code></li> <li><code>.jpeg</code> -&gt; <code>image/jpeg</code></li> <li><code>.gif</code> -&gt; <code>image/gif</code></li> <li><code>.svg</code> -&gt; <code>image/svg+xml</code></li> <li><code>.ico</code> -&gt; <code>image/x-icon</code></li> <li><code>.webp</code> -&gt; <code>image/webp</code></li> <li><code>.avif</code> -&gt; <code>image/avif</code></li> <li><code>.bmp</code> -&gt; <code>image/bmp</code></li> <li><code>.tiff</code> -&gt; <code>image/tiff</code></li> <li><code>.apng</code> -&gt; <code>image/apng</code></li> </ul> <p>Font files:</p> <ul> <li><code>.woff</code> -&gt; <code>font/woff</code></li> <li><code>.woff2</code> -&gt; <code>font/woff2</code></li> <li><code>.ttf</code> -&gt; <code>font/ttf</code></li> <li><code>.otf</code> -&gt; <code>font/otf</code></li> </ul> <p>Document files:</p> <ul> <li><code>.txt</code> -&gt; <code>text/plain; charset=utf-8</code></li> <li><code>.pdf</code> -&gt; <code>application/pdf</code></li> <li><code>.doc</code> -&gt; <code>application/msword</code></li> <li><code>.docx</code> -&gt; <code>application/vnd.openxmlformats-officedocument.wordprocessingml.document</code></li> <li><code>.epub</code> -&gt; <code>application/epub+zip</code></li> <li><code>.rtf</code> -&gt; <code>application/rtf</code></li> </ul> <p>Video files:</p> <ul> <li><code>.mp4</code> -&gt; <code>video/mp4</code></li> <li><code>.webm</code> -&gt; <code>video/webm</code></li> <li><code>.mkv</code> -&gt; <code>video/x-matroska</code></li> <li><code>.avi</code> -&gt; <code>video/x-msvideo</code></li> <li><code>.mov</code> -&gt; <code>video/quicktime</code></li> </ul> <p>Audio files:</p> <ul> <li><code>.mp3</code> -&gt; <code>audio/mpeg</code></li> <li><code>.ogg</code> -&gt; <code>audio/ogg</code></li> <li><code>.wav</code> -&gt; <code>audio/wav</code></li> <li><code>.flac</code> -&gt; <code>audio/flac</code></li> <li><code>.aac</code> -&gt; <code>audio/aac</code></li> <li><code>.m4a</code> -&gt; <code>audio/mp4</code></li> <li><code>.opus</code> -&gt; <code>audio/opus</code></li> </ul> <p>Archive files:</p> <ul> <li><code>.zip</code> -&gt; <code>application/zip</code></li> <li><code>.gz</code> -&gt; <code>application/gzip</code></li> <li><code>.tar</code> -&gt; <code>application/x-tar</code></li> <li><code>.7z</code> -&gt; <code>application/x-7z-compressed</code></li> <li><code>.bz2</code> -&gt; <code>application/x-bzip2</code></li> <li><code>.rar</code> -&gt; <code>application/vnd.rar</code></li> </ul> <h2>Configuration</h2> <p>You can configure ymawky with the <code>config.S</code> file. The options are documented here.</p> <ul> <li><code>#define DEFAULT_DIR "www/"</code> -- This is the docroot. Change it to wherever your HTML files are, relative to ymawky, or use an absolute path: <ul> <li><code>#define DEFAULT_DIR "www/"</code></li> <li><code>#define DEFAULT_DIR "/Library/WebServer/Documents</code></li> <li><code>#define DEFAULT_DIR "./"</code></li> </ul> </li> <li><code>#default ERR_DIR "err/"</code> -- This is the directory in which ymawky will search for custom error HTML pages, eg, <code>err/404.html</code> or <code>err/500.html</code></li> <li><code>#define DEFAULT_FILE "index.html"</code> -- This is the default file ymawky will serve when it receives an empty <code>GET / HTTP/1.1</code> request</li> <li><code>.equ RECV_TIMEOUT, 10</code> -- Number of seconds ymawky will wait to receive datta before closing the connection. If it's more than <code>RECV_TIMEOUT</code> seconds between <code>read()</code>s, ymawky will close the connection with <code>408 Request Timed Out</code></li> <li><code>.equ HEADER_REQ_TIMEOUT_SECS, 10</code> -- Maximum number of seconds ymawky will wait to receive the full header before timing out. If it takes, longer than this to receive the header, ymawky will close the connection with <code>408 Request Timed Out</code></li> <li><code>.equ PUT_GRACE_SECS, 5</code> -- ymawky dynamically calculates a max-time-per-PUT based on <code>Content-Length</code>. The max time is defined as <code>PUT_GRACE_SECS + Content-Length / PUT_MIN_BPS</code>. This is the minimum grace period allowed if it calculates a file should take &lt;1 second to upload</li> <li><code>.equ PUT_MIN_BPS, 1024 * 16</code> -- Minimum bytes-per-second. Higher if you want to be stricter, smaller if you want to be more lenient. Since this uses the <code>.equ</code> directive, arithmetic is supported, and <code>1024 * 16</code> gets calculated at assembly time becoming <code>16384</code> or 16KB</li> <li><code>.equ MAX_BODY_SIZE, 1024 * 1024 * 1024</code> -- Maximum bytes PUT allows for Content-Length. By default, 1GB (1024<em>1024</em>1024 = 1073741824 bytes). Files with a larger Content-Length larger than this will be rejected with <code>413 Content Too Large</code></li> <li><code>.equ MAX_PROCS, 256</code> -- Maximum number of concurrent proccesses ymawky is allowed to run. Since ymawky is a fork-per-connection server, you want to ensure ymawky doesn't exhaust your PID space. ymawky will reply with <code>503 Service Unavailable</code></li> </ul> <h2>Implementation Notes</h2> <p>ymawky is written for MacOS (sorry...). There are a few (well, more than a <em>few</em>) things that are MacOS-specific in this code that won't be portable.</p> <ul> <li>Syscalls on MacOS use <code>x16</code> for the number and <code>svc #0x80</code> to call it. Linux uses <code>x8</code> and <code>svc #0</code>.</li> <li>Error reporting is different. MacOS sets the carry flag on error, and puts <code>errno</code> in <code>x0</code>. Linux returns a negative value in <code>x0</code>, like <code>-ENOENT</code>. Ever <code>b.cs</code> would need to be replaced with <code>cmp x0, #0</code> / <code>b.lt ...</code>, and you'd negate <code>x0</code> to get errno.</li> <li><code>fork()</code> works differently, MacOS puts 1 in <code>x1</code> in the child process, whereas Linux puts <code>0</code> in <code>x0</code>.</li> <li><code>SO_NOSIGPIPE</code> doesn't exist on Linux.</li> <li><code>O_NOFOLLOW_ANY</code> is also MacOS-specific.</li> <li><code>renameatx_np()</code> is also MacOS-specific. Linux has <code>renameat2()</code>, with different flag values.</li> <li>Struct layouts and offsets will differ. The <code>stat64</code> struct, <code>itimerval</code> struct, and <code>sockaddr_in</code> struct, will all need to be reconsidered.</li> <li><code>adr xN, foo@PAGE</code> / <code>add xN, xN, foo@PAGEOFF</code> are Mach-O relocation operators. Linux ELF uses different syntax, like <code>:pg_hi21:</code> and <code>:lo12:</code>. The <code>adr_l</code>, <code>ldr_l</code> and <code>str_l</code> macros would need to be rewritten or replaced.</li> <li>My personal favorite :3 Signal handling works differently on Linux and MacOS. MacOS's <code>sigaction</code> struct contains a <code>sa_tramp</code> field that the kernel jumps to before your handler. ymawky utilizes <code>sa_tramp</code> directly <em>as the handler itself</em>, skipping the libc trampoline and <code>sigreturn</code> entirely. Since the handler only sends a 408 and exits, without needing to return, that's fine and works wonderfully without libc. The <code>sigaction</code> call would need to be rewritten for POSIX systems.</li> </ul> <h3>Special Thanks:</h3> <ul> <li><em>Bob Johnson</em></li> <li><em>Bob Johnson's Therapist</em></li> </ul> </article>https://lemire.me/blog/2025/12/04/we-see-something-that-works-and-then-we-understand-it/Wed, 06 May 2026 15:15:23 +0000https://news.ycombinator.com/item?id=48037223<a href="https://news.ycombinator.com/item?id=48037223">Comments</a><article id="post-22357" class="post-22357 post type-post status-publish format-standard has-post-thumbnail hentry category-84" morss_own_score="9.59059893858984" morss_score="15.01091143858984"> <img src="https://lemire.me/blog/wp-content/uploads/2025/12/Capture-decran-le-2025-12-04-a-10.40.38-825x510.png"> <h1>We see something that works, and then we understand it</h1> <div class="entry-content" morss_own_score="5.840624999999999" morss_score="52.26484491084695"> <p>“We see something that works, and then we understand it.” (Thomas Dullien)</p> <p>It is a deeper insight than it seems.</p> <p>Young people spend years in school learning the reverse: understanding happens before progress. That is the linear theory of innovation.</p> <p>So Isaac Newton comes up with his three laws of mechanics, and we get a clockmaking boom. Of course, that’s not what happened: we get the pendulum clock in 1656, then Hooke (1660) and Newton (1665–1666) get to think about forces, speed, motion, and latent energy.</p> <p>The linear model of innovation makes as much sense as the waterfall model in software engineering. In the waterfall model, you are taught that you first need to design every detail of your software application (e.g., using a language like UML) before you implement it. To this day, half of the information technology staff members at my school are made up of “analysts” whose main job is supposedly to create such designs based on requirements and supervise execution.</p> <p>Both the linear theory and the waterfall model are forms of thinkism, a term I learned from Kevin Kelly. Thinkism sets aside practice and experience. It is the belief that given a problem, you should just think long and hard about it, and if you spend enough time thinking, you will solve it.</p> <p>Thinkism works well in school. The teacher gives you all the concepts, then gives you a problem that, by a wonderful coincidence, can be solved just by thinking with the tools the same teacher just gave you.</p> <p>As a teacher, I can tell you that students get really angry if you put a question on an exam that requires a concept not explicitly covered in class. Of course, if you work as an engineer and you’re stuck on a problem and you tell your boss it cannot be solved with the ideas you learned in college… you’re going to look like a fool.</p> <p>If you’re still in school, here’s a fact: you will learn as much or more every year of your professional life than you learned during an entire university degree—assuming you have a real engineering job.</p> <p>Thinkism also works well in other limited domains beyond school. It works well in bureaucratic settings where all the rules are known and you’re expected to apply them without question. There are many jobs where you first learn and then apply. And if you ever encounter new conditions where your training doesn’t directly apply, you’re supposed to report back to your superiors, who will then tell you what to do.</p> <p>But if you work in research and development, you always begin with incomplete understanding. And most of the time, even if you could read everything ever written about your problem, you still wouldn’t understand enough to solve it. The way you make discoveries is often to either try something that seems sensible, or to observe something that happens to work—maybe your colleague has a practical technique that just works—and then you start thinking about it, formalizing it, putting it into words… and it becomes a discovery.</p> <p>And the reason it often works this way is that “nobody knows anything.” The world is so complex that even the smartest individual knows only a fraction of what there is to know, and much of what they think they know is slightly wrong—and they don’t know which part is wrong.</p> <p>So why should you care about how progress happens? You should care because…</p> <p> 1. It gives you a recipe for breakthroughs: spend more time observing and trying new things… and less time thinking abstractly.</p><p> 2. Stop expecting an AI to cure all diseases or solve all problems just because it can read all the scholarship and “think” for a very long time. No matter how much an AI “knows,” it is always too little.</p><p><strong>Further reading</strong>: Godin, Benoît (2017). Models of innovation: The history of an idea. MIT press.</p> <p>Daniel Lemire, "We see something that works, and then we understand it," in <em>Daniel Lemire's blog</em>, December 4, 2025, <a href="https://lemire.me/blog/2025/12/04/we-see-something-that-works-and-then-we-understand-it/">https://lemire.me/blog/2025/12/04/we-see-something-that-works-and-then-we-understand-it/</a>. <br> <a href="https://lemire.me/blog/2025/12/04/we-see-something-that-works-and-then-we-understand-it/"> [BibTeX] </a> </p> </div> <img src="https://secure.gravatar.com/avatar/a0c6c34cf4a45ff5083d5ea541e7f27c5e4457ac393889e077af10688f6b3831?s=56&amp;d=mm&amp;r=g"> <h3>Daniel Lemire</h3> <p> A computer science professor at the University of Quebec (TELUQ). <a href="https://lemire.me/blog/author/lemire/"> View all posts by Daniel Lemire </a> </p> </article>