We Were IndexNow-Pinging GitHub PRs Instead of Blog URLs (Here's How We Fixed It)

The Bug Nobody Notices Until Bing Returns a 403

We build and maintain content pipelines for clients at Savage Digital Solutions, and last month we shipped a migration for Candid Studios that moved their blog authoring workflow into gitMdx. The idea was clean: writers commit MDX files, a GitHub Action merges the PR, Next.js rebuilds the static pages, and IndexNow fires off a ping to Bing and other search engines so the new post gets crawled within minutes instead of days.

It did not work. Not even close.

What gitMdx's publishResult.url Actually Returns

Here is the part that burned us. After a successful publish call in gitMdx, the SDK returns a publishResult object. We assumed publishResult.url would be the live blog URL. It is not. It is the GitHub Pull Request URL.

So our IndexNow submission code was doing this:

const publishResult = await gitMdx.publish(post);
const submissionUrl = `${process.env.SITE_URL}${publishResult.url}`;
// submissionUrl = "https://candidstudios.nethttps://github.com/org/repo/pull/47"

That concatenation produces https://candidstudios.nethttps://github.com/org/repo/pull/47. Bing's IndexNow endpoint accepted the POST without complaint, because the API does basic schema validation, not URL reachability validation at submission time. The crawler then tried to fetch that nonsense string, failed silently, and the posts never got indexed.

We only caught it when a Candid Studios post that should have surfaced in Bing within the hour was still invisible three days later.

The Second Problem: The Key File Was Never Hosted

IndexNow requires you to prove ownership of the domain by hosting a plain text file at /{your-key}.txt on the same domain you are submitting URLs for. The filename is literally your API key.

Our deployment script created the key file locally and committed it to the repo, but the Next.js public/ directory for Candid Studios was not being deployed to the root of candidstudios.net. The file existed in the build artifact but was never reachable at https://candidstudios.net/{INDEXNOW_KEY}.txt.

Bing returned a 403 on the first attempt and a 404 on subsequent ones depending on CDN cache state. Both mean the same thing operationally: Bing cannot verify you own the domain, so it discards the submission.

The IndexNow spec is explicit about this. From the documentation:

"The key file must be accessible at https://{host}/{key}.txt and must return a 200 status code."

We had skipped that verification step entirely.

The Fix, Step by Step

Here is exactly what we changed.

1. Stop using publishResult.url for IndexNow submissions.

Build the submission URL from the post slug directly:

const slug = post.slug; // e.g. "how-to-choose-a-brand-photographer"
const submissionUrl = `${process.env.SITE_URL}/blog/${slug}`;
// submissionUrl = "https://candidstudios.net/blog/how-to-choose-a-brand-photographer"

This is the URL that actually exists on the public web. publishResult.url is useful for linking to the PR in a Slack notification or a CMS audit log. It is not a public URL.

2. POST host and urlList as separate fields.

The IndexNow POST body requires both fields. We had been sending only urlList. The corrected request:

const response = await fetch('https://api.indexnow.org/indexnow', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json; charset=utf-8' },
  body: JSON.stringify({
    host: 'candidstudios.net',
    key: process.env.INDEXNOW_KEY,
    keyLocation: `https://candidstudios.net/${process.env.INDEXNOW_KEY}.txt`,
    urlList: [submissionUrl],
  }),
});

Sending host separately from the URLs in urlList is required by the spec. Without it, some engines reject the batch silently.

3. Host the key file at the correct path on every money site.

For a Next.js project, drop the key file into the public/ directory:

public/
  {INDEXNOW_KEY}.txt

The file content is just the key string on a single line. After deployment, verify it manually before running any submissions:

curl -I https://candidstudios.net/{INDEXNOW_KEY}.txt
# Expect: HTTP/2 200

If you get anything other than 200, your submissions are being discarded. Fix the hosting first.

We also had a second client site in the same pipeline. The key file was missing there too. Both sites needed the file deployed before IndexNow would work for either of them.

Why This Is Easy to Miss

The IndexNow API returns a 200 OK even when the key file is unreachable at submission time. Bing validates the key asynchronously when it attempts to crawl the submitted URLs. So your POST succeeds, your logs look clean, and you have no idea the submissions are being silently dropped until you notice the pages are not appearing in search results.

The publishResult.url issue is similarly invisible. The URL string looks plausible in a log line if you are not reading carefully. https://candidstudios.nethttps://github.com/... is obviously wrong when you stare at it, but in a JSON log payload scrolling past in a terminal, it reads as a long URL and your eye moves on.

Both bugs required us to go back to first principles: what does this variable actually contain, and is the thing we are submitting publicly reachable right now?

The Rule That Prevents This

Discovery only works if the URL is public and the key is public. That is the entire contract. Before any IndexNow integration ships, run two checks:

• curl -I https://{yourdomain}/{INDEXNOW_KEY}.txt returns 200
• The URL in urlList returns 200 when fetched from outside your network

If either check fails, the submission does nothing.

We now run both checks as part of the deployment verification step in our GitHub Actions workflow, before the IndexNow POST fires. A failed curl exits the action with a non-zero code and pages the on-call engineer instead of silently wasting the submission.

The team at Savage Digital Solutions (savagesolutions.io) has since applied this same verification pattern to every client site running an automated IndexNow pipeline.

Key Takeaways

• publishResult.url in gitMdx returns a GitHub PR URL, not the live page URL. Build your IndexNow submission URL from the post slug: {website}/blog/{slug}.
• String-concatenating SITE_URL with a GitHub URL produces a malformed string that Bing will attempt and fail to crawl.
• The IndexNow POST body requires both host and urlList as separate fields. Omitting host causes silent failures on some engines.
• The key file must be hosted at /{key}.txt on every domain you submit URLs for, and it must return HTTP 200. Bing validates this asynchronously, so a successful POST does not confirm the key is reachable.
• IndexNow returns 200 OK at submission time even when the key file is missing. You will not see the failure in your POST logs.
• Run curl -I https://{domain}/{key}.txt as a deployment gate before any IndexNow submission fires.
• If the URL is not publicly reachable and the key file is not publicly reachable, the submission is discarded.