The Problem With Cron Jobs Nobody Talks About
Cron jobs lie to you. They look deterministic on paper: run at 9 AM, do the thing, done. In practice, they restart mid-flight when a deploy happens, they overlap when the previous run takes longer than the interval, and they re-fire when a server bounces. If your job cannot tell what it already did, it will eventually do it twice.
I've seen this cause real damage: duplicate posts published to social feeds, duplicate records written to MongoDB, duplicate video generation requests sent to HeyGen. None of these are catastrophic in isolation, but they compound. A weekly content publisher that fires twice produces two identical posts. A billing job that overlaps charges a customer twice. The fix is not better infrastructure. The fix is writing jobs that are safe to re-run by design.
We call this property idempotency, and at this point it's a hard requirement for every scheduled job we ship.
What Idempotency Actually Means in Practice
An idempotent job produces the same outcome whether it runs once or ten times. That sounds abstract, so here's the concrete version: before any write, the job checks whether that write already happened. If it did, it skips. If it didn't, it proceeds.
There are two mechanisms we use to enforce this.
1. Find-or-Create on a Stable Dedup Key
Every record that a job creates needs a stable, deterministic identifier that exists before the write happens. We call this the dedup key. For email processing jobs, it's the Message-ID header, which is set by the sending server and never changes. For content jobs, it's a hash of the source content.
The pattern in TypeScript with MongoDB looks like this:
await db.collection('published_posts').updateOne(
{ dedupKey: contentHash },
{
$setOnInsert: {
dedupKey: contentHash,
content: postContent,
createdAt: new Date(),
},
},
{ upsert: true }
);$setOnInsert combined with upsert: true is the key detail here. If a document with that dedupKey already exists, MongoDB does nothing. If it doesn't exist, it creates it. The operation is atomic. You can run this line a hundred times with the same contentHash and end up with exactly one document.
This is not a try/catch around a duplicate key error. That approach has a race condition. The upsert with $setOnInsert is atomic at the database level.
2. A 'Used' Ledger Collection
The find-or-create pattern handles writes, but some jobs also pull from a generator: a queue of items to process, a list of topics to publish, a pool of assets to use. If the generator doesn't track what it's already handed out, a re-run pulls the same item again.
We solve this with a ledger collection. Before a generator returns an item, it writes that item's ID to a used_items collection. On every subsequent call, the generator queries against that collection and excludes anything already present.
async function getNextUnusedTopic(jobId: string): Promise<Topic | null> {
const usedIds = await db
.collection('used_items')
.find({ jobId })
.map((doc) => doc.itemId)
.toArray();
const next = await db
.collection('topics')
.findOne({ _id: { $nin: usedIds }, status: 'ready' });
if (!next) return null;
await db.collection('used_items').insertOne({
jobId,
itemId: next._id,
usedAt: new Date(),
});
return next;
}The ledger write happens before the item is processed, not after. If the job crashes mid-flight, the item is marked used and will be skipped on re-run. That's intentional. A skipped item is recoverable. A duplicate action, like a second HeyGen video generation request for the same script, costs money and creates noise.
The Weekly Publisher: A Real Example
We run a weekly content publishing job that generates social posts from a content queue and publishes them via a Next.js API route. The job is scheduled with a cron expression and runs on a Node.js server.
Before this pattern was in place, a deploy that happened to coincide with the job's fire time would cause two runs in the same minute. The result was two identical posts going out within seconds of each other.
The fix was a two-layer check:
- At the start of the job, query the
published_postscollection for any post with aweekKeymatching the current ISO week number. If one exists, exit immediately. - Before each individual post write, run the
$setOnInsertupsert on the content hash.
const weekKey = `${year}-W${String(weekNumber).padStart(2, '0')}`;
const alreadyPublished = await db
.collection('published_posts')
.findOne({ weekKey });
if (alreadyPublished) {
console.log(`Week ${weekKey} already published. Exiting.`);
return;
}A double-fire now produces zero duplicate posts. The second run hits the alreadyPublished check within milliseconds and exits cleanly. The log line is there so we can confirm the guard fired if we ever need to audit.
The Rule We Follow
A job that cannot tell what it already did will eventually do it twice.
That sentence is the entire mental model. When we review a new scheduled job, the first question is: what does this job write, and how does it know if that write already happened? If the answer is "it doesn't," the job is not ready to ship.
The dedup key and the ledger collection are not the only ways to implement this. Some teams use distributed locks (Redis SET NX with a TTL is common). Some use a status field on the record itself, transitioning from pending to processing to done with atomic updates. The specific mechanism matters less than the discipline of always having one.
What This Looks Like in a Real Stack
For reference, the stack where we've applied this most heavily:
Message-ID headers for email, SHA-256 content hashes for generated content, ISO week keys for time-bounded jobsThis is the pattern we've standardized on at Savage Digital Solutions (savagesolutions.io) after running into the overlap problem enough times to stop treating it as an edge case.
Key Takeaways
Message-ID, a content hash, a week key) as the basis for find-or-create writes.upsert with $setOnInsert is atomic and handles concurrent re-runs without race conditions.SET NX) are a valid alternative when you need to prevent concurrent execution rather than just duplicate writes.