Integrations do not break loudly — design for the silent failure

Engineers plan for the integration returning an error. That case is easy: you see it, you handle it, you retry. The failures that actually cause damage are quieter — a timeout after the remote side already committed the write, a webhook delivered twice, a provider silently changing a field name, a queue that stops being consumed.

None of these throw an exception you would notice. All of them corrupt data over time. So the design goal is not preventing failure, which you cannot do across a network you do not control. It is making failure visible and recovery boring.

Every retry you add is a duplicate-execution risk unless the operation is idempotent. For outbound calls, that means sending a stable idempotency key the provider honours, so a retried charge or message does not happen twice. For inbound webhooks, it means recording the event identifier and ignoring one you have already processed — providers explicitly promise at-least-once delivery, so duplicates are normal traffic, not an anomaly.

Get this right before adding retries. Retries on a non-idempotent operation do not improve reliability, they multiply the damage.

Retry immediately and you hammer a service that is already struggling. Exponential backoff with jitter spreads the load and avoids every client in the world retrying in lockstep after an outage.

Equally important is a terminal state. An operation that retries forever is a queue that never drains and an alert that never fires. After a bounded number of attempts it should land in a dead-letter state that a human can inspect and replay — which means building the replay path, not just the failure state.

Distinguish error classes too. A 429 means slow down and retry. A 400 means your payload is wrong and retrying will never help; that should fail fast and loudly rather than consuming the retry budget.

This is the piece most integrations lack. Error-rate alerting catches the integration that is failing. It does not catch the integration that has stopped entirely — zero requests produces zero errors, and every dashboard looks healthy.

So alert on expected volume. If a nightly sync normally moves several hundred records and moves none, that is the alert. If a webhook endpoint normally receives traffic every hour and has been silent since Tuesday, that is the alert. In practice these caught more real incidents than error alerting ever did.

Most integration work involves at least one provider whose documentation is wrong, outdated or absent. The productive approach is to treat their sandbox as the specification: probe it, record what actually comes back, and write that down as part of the delivery.

That document is worth more to the next developer than anything the provider publishes, and it is the difference between an integration your team can maintain and one only its author understands.