Creating links idempotently in automated systems

Why duplicates appear

A workflow posts a create request, the response is lost to a timeout, the workflow retries, and now two links point at the same destination. Both accumulate clicks. Neither report is complete. This happens most often in queue-based systems with at-least-once delivery, which is nearly all of them.

Deterministic slugs are the simplest fix

If you can derive the slug from the input, creation becomes naturally idempotent: the second attempt tries to create a slug that already exists and gets a conflict, which your code treats as success. A hash of the destination plus the campaign identifier, truncated and encoded, works well and gives you a stable, reproducible slug for a given input.

The trade-off is that the slug is opaque. For internal automation that is fine; for anything a person reads, use a meaningful slug and a different mechanism.

Idempotency keys where the API supports them

Send a client-generated key with each create request and reuse the same key on retries. The server recognises the repeat and returns the original result instead of creating a second resource. Generate the key from the logical unit of work — the order id, the message id — not randomly, or a retry after a process restart will generate a new key and defeat the purpose.

Check-then-create is a race, not a solution

Searching for an existing link and creating one if absent looks idempotent and is not: two workers can both find nothing and both create. If you must use this pattern, rely on a unique constraint on the slug to make one of them fail, and handle the conflict rather than trusting the check.

Store the mapping on your side

Keep a table from your business key to the short link that was created for it. Look there first, and treat the link platform as the system of record for clicks rather than for existence. This makes reprocessing an entire batch safe, and it means you can reconcile after an outage without querying the API for every record.

Make cleanup part of the design

If duplicates do slip through, you need a way to find them: group by destination and creation window, and keep the one with clicks. Build the query before you need it. An automation running for a year will produce duplicates at some point, and finding them by hand across ten thousand links is not a pleasant afternoon.

Test the retry path deliberately

Simulate a timeout after the server has processed the request. Most integrations are never tested this way and only discover their behaviour during a real incident. Kill the connection mid-flight in a staging environment and confirm the retry produces one link, not two.