A Hanging Dedup Fetch Killed Our Daily Blog for a Week

The Week Our Blog Went Silent

For seven days, our automated blog pipeline produced nothing. No posts, no errors in Slack, no pages. The cron job showed as running. The logs showed it starting. And then: nothing. The process just sat there, alive but useless, until the next scheduled run kicked off another zombie alongside it.

This is the story of how a single hanging HTTP fetch inside a Promise.all took down our entire content automation system, why no alert fired, and exactly what we changed to make sure it never happens again.

What the Pipeline Actually Does

Our blog automation is a TypeScript service running on a containerized Next.js backend. Each day, a cron triggers a content generation job that runs three gate checks before writing anything to MongoDB:

• Cannibalization check, queries existing posts to make sure the new topic doesn't overlap with something we already rank for
• Semantic dedup check, hits an external embedding API to compare the candidate post against recent content
• Base keyword check, validates the target keyword against our keyword strategy rules

All three ran concurrently inside a single Promise.all:

const [cannibalizationResult, dedupResult, baseKeyResult] = await Promise.all([
  checkCannibalization(topic),
  checkSemanticDedup(topic),
  checkBaseKeyword(topic),
]);

This is clean, fast, and completely correct under normal conditions. The problem was what happened when conditions were not normal.

The Actual Failure: EAI_AGAIN on Flaky Container DNS

Our container environment started throwing intermittent EAI_AGAIN errors on DNS resolution. EAI_AGAIN is a temporary DNS failure, the resolver saying "try again later." Node's fetch (and most HTTP clients) will retry internally on these, but if the DNS never resolves, the fetch just hangs. No rejection, no timeout, no error. It waits.

The semantic dedup check calls an external embedding API. That fetch hung indefinitely on a DNS failure. Because it was inside Promise.all with no per-fetch timeout, the entire Promise.all also hung. The outer async handler never returned. The cron job never reached its COMPLETE state.

Here is the part that made it invisible: our alerting was wired to the cron's success grep. The monitoring script looked for a "status": "complete" string in the job's output. If the job never returned, that string never appeared. But the monitor didn't treat "no output" as a failure. It treated it as "still running." So no page fired.

Seven days. Fourteen missed blog posts. Zero alerts.

The Fix: Promise.race With a Named Timeout

The fix has two parts. First, wrap every gate check in a Promise.race against a named timeout. Second, make sure the outer handler returns a non-success JSON response when any check fails or times out, so the cron's success-grep has something to catch.

Here is the timeout wrapper we added:

function withTimeout<T>(promise: Promise<T>, ms: number, label: string): Promise<T> {
  const timeout = new Promise<never>((_, reject) =>
    setTimeout(() => reject(new Error(`${label} timed out after ${ms}ms`)), ms)
  );
  return Promise.race([promise, timeout]);
}

And here is how the gate checks now run:

try {
  const [cannibalizationResult, dedupResult, baseKeyResult] = await Promise.all([
    withTimeout(checkCannibalization(topic), 30000, 'cannibalization_check_timeout'),
    withTimeout(checkSemanticDedup(topic), 30000, 'dedup_check_timeout'),
    withTimeout(checkBaseKeyword(topic), 30000, 'base_key_check_timeout'),
  ]);

  // proceed with generation
} catch (err) {
  logger.error({ err, topic }, 'Gate check failed or timed out');
  return res.status(200).json({ status: 'error', reason: err.message });
}

A few things worth noting about this implementation:

• The timeout is 30 seconds per check, named dedup_check_timeout in logs so we can grep for it specifically in Datadog.
• We return status: 200 with an error body rather than a 5xx, because our cron runner treats non-200 as a network failure and retries silently. The success-grep now looks for "status": "complete" and pages on anything else.
• We fail closed. If the dedup check times out, we do not generate the post. A missed post is recoverable. A duplicate or cannibalizing post is not.

Why Fail-Closed Is the Right Call Here

Some teams would argue for fail-open: if the dedup check is unavailable, skip it and generate anyway. That logic makes sense for non-critical gates. For content quality gates, it does not.

The cannibalization check exists because publishing a post that competes with an existing ranking page actively hurts SEO. The semantic dedup check exists because publishing near-duplicate content is a quality signal Google penalizes. Skipping either check because of a transient DNS failure trades a short-term miss for a long-term ranking problem.

Fail-closed means we miss a day of content. Fail-open means we potentially publish something that damages the site. The math is straightforward.

The key rule we wrote into our runbook: a silent hang is never acceptable, even if the failure mode is conservative. The system must always return, and it must always return something the monitoring layer can evaluate.

What We Added to Monitoring

Beyond the code change, we made three monitoring updates:

• Explicit timeout log lines: every dedup_check_timeout or cannibalization_check_timeout error writes a structured log entry that triggers a Datadog monitor.
• Cron heartbeat check: the job now writes a {"status": "heartbeat"} line every 60 seconds while running. If the monitor sees no heartbeat and no completion within 5 minutes, it pages.
• Dead-letter queue for skipped posts: when a gate check fails, the topic goes into a MongoDB collection called pending_retry so the next day's run picks it up automatically.

The retry collection is what actually recovered the seven missed posts. Once the DNS issue was resolved and the fix was deployed, the next cron run processed the backlog automatically.

The Broader Pattern

This failure is not specific to blog automation. Any Promise.all that calls external services without per-promise timeouts is vulnerable to this exact hang. The pattern shows up in:

• Parallel API enrichment pipelines
• Multi-source data aggregation jobs
• Webhook processors that fan out to third-party services

The fix is always the same: Promise.race with a named timeout, structured error logging, and a monitoring layer that treats "no response" as a failure, not as "still running."

At Savage Digital Solutions, this incident became the basis for a standard we now apply to every async pipeline we build: no Promise.all over external calls without an explicit per-promise timeout and a fail-closed error path.

Key Takeaways

• A single hanging fetch inside Promise.all with no timeout will hang the entire Promise.all indefinitely.
• EAI_AGAIN DNS errors in containerized environments do not reject promises. They hang them.
• Wrap every external call in Promise.race with a named timeout (we use 30 seconds, named per-check for log grep).
• Fail closed on content quality gates. A missed post is recoverable; a cannibalizing post is not.
• Your monitoring must treat "no response" as a failure. Success-grep on a process that never returns will never fire.
• Write a structured log line on every timeout so your observability layer has something to alert on.
• Use a pending_retry collection or equivalent dead-letter queue so skipped jobs are not permanently lost.