onBrokenLinks incorrectly handles "antennapod-subscribe" scheme

[wparad]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

Prerequisites

• I'm using the latest version of Docusaurus.
• I have tried the npm run clear or yarn clear command.
• I have tried rm -rf node_modules yarn.lock package-lock.json and re-installing packages.
• I have tried creating a repro with https://new.docusaurus.io.
• I have read the console error message carefully (if applicable).

Description

links with antennapod-subscribe, and I'm sure any scheme that contain a url path that matches the docusaurus host path are incorrectly assumed to be the site itself.

For instance, I don't control a scheme, that's registered somewhere else. And I have a page in the website that I'm dynamically generating, for instance the rss.xml feed url.

onBrokenLinks doesn't take in generated pages into consideration, even if that file is listed in pages/.

Reproducible demo

No response

Steps to reproduce

Create a element with a to=antennapod-subscribe://site-url/rss.xml ensure that site-url is the docusaurus site, and that rss.xml exists in the pages directory

Run the build, and see it fail.

Expected behavior

It should correctly include pages in broken link validation

Actual behavior

It ignores the file and assumes there is a broken link.

Your environment

• Public source code: https://github.com/AdventuresInDevops/Website
• Public site URL: https://adventuresindevops.com
• Uncomment this line: https://github.com/AdventuresInDevops/Website/blob/main/src/components/socialButtons/index.js#L60

Self-service

• I'd be willing to fix this bug myself.

[slorber]

If you want help, please help us help you:

• provide a minimal repro where the bug reproduces by simply running yarn start
• share the full and exact error message and not a vague "incorrectly handles" term. I don't know what "incorrectly" means here
• share clearly the expected behavior and the actual behavior

antennapod-subscribe::/ seems wrong.

Protocols are more likely antennapod-subscribe://, and using :: may not be recognized by our Link component as an external link to automatically opt out of broken link checks. If really needed, you can pass a data-noBrokenLinkCheck attribute to force the opt out and avoid the build error.

Our broken link checker doesn't know about external files like a RSS feed XML file, and cannot check whether a link to it is valid or not.

onBrokenLinks doesn't take in generated pages into consideration, even if that file is listed in pages/.

I have no idea what you mean

---

My intuition is that we may have a bug in this code:

export function hasProtocol(url: string): boolean {
  return /^(?:\w*:|\/\/)/.test(url);
}

export default function isInternalUrl(url?: string): boolean {
  return typeof url !== 'undefined' && !hasProtocol(url);
}

It seems to assume a protocol cannot contain a dash

[wparad]

antennapod-subscribe::/ seems wrong.

That's a typo in the bug report, I've updated the original post.

provide a minimal repro where the bug reproduces by simply running yarn start

onBrokenLinks seems only run yarn build.

antennapod-subscribe:// may not be recognized by our Link component as an external link to automatically opt out of broken link checks.

Given I'm actually pointing to an "internal file", I would actually prefer for it to not opt out. I think the current functionality is actually fine.

Okay, hmm there are actually two problems:

1. as you pointed out the dash isn't handle in the scheme, I'm not actually sure that's even a valid scheme, but /shrug

2. Something is weird with pages directory.

Our broken link checker doesn't know about external files like a RSS feed XML file, and cannot check whether a link to it is valid or not.
I have no idea what you mean

Docusaurus supports a /pages directory, here is the documentation on that, if you are unaware: https://docusaurus.io/docs/creating-pages

I would expect that creating a rss.xml file in that directory, (which is being copied to the output directory would get correctly picked up by the broken link checker, but it is not.

---

And here is your minimal repro using npm init docusaurus@latest my-website classic

step 1: run npm init docusaurus@latest my-website classic
step 2: update the src/pages/index.js Link to and point it at /rss.xml
step 3: create a src/pages/rss.xml file
step 4: run yarn && yarn build

Weirdly in my version of docusaurus, I actually get a /rss.xml file in my build directory, but in this repo, the rss.xml isn't being copied to the output. I can't actually get any non-js, non-ts, non-md* file copied to the build output directory. Did that c

[slorber]

onBrokenLinks seems only run yarn build.

My mistake, I meant you should provide a repro that triggers when running yarn build

Given I'm actually pointing to an "internal file", I would actually prefer for it to not opt out. I think the current functionality is actually fine.

Docusaurus broken links checker only checks links to internal routes. A rss.xml file is not an internal route and will never be checked. It can be provided by putting a file in static, generated it in a plugin postBuild(), generated as part of your CI script, or provided through CDN options and other means.

Docusaurus supports a /pages directory, here is the documentation on that, if you are unaware: docusaurus.io/docs/creating-pages

I'm the main Docusaurus maintainer and wrote this page 😅 If I ask for clarifications, it's probably because you were not clear, and not because I don't know how Docusaurus works.

I would expect that creating a rss.xml file in that directory, (which is being copied to the output directory would get correctly picked up by the broken link checker, but it is not.

The pages plugin doesn't handle xml files, it only handles js/ts/jsx/tsx/md/mdx files.

Other file extensions are not copied to the output directory by the plugin itself, so if this happens, it's for other reasons (you copy it though other means, or your code imports the file as an asset with CSJ/ESM require/import statements)

The presence of a random file in pages, docs or blog does not create an internal route, and the broken link checker only validates internal links against internal routes at build time. It doesn't validate links to static files that are not routes, even if those files are part of your website.

[wparad]

If putting a file in pages is not enough to avoid the broken link checker, what's the correct solution? Put the absolute url in the <Link to="">?

Re: data-noBrokenLinkCheck I don't see any docs on that, is that just an html attribute?

Edit: Hmm, looks like the answer is yes to that.

[slorber]

data-noBrokenLinkCheck is an internal, undocumented prop of the <Link> component. We may document it publicly later, but ideally, I'd prefer to share this escape hatch on a case-by-case basis to understand problem first and see if we can figure out a solution that is more "automatic".

Here we may have a bug, so it's better to make me aware of the bug with a minimal repro, rather than silently applying a documented escape hatch that you wouldn't even need if there were no bug.

---

If putting a file in pages is not enough to avoid the broken link checker, what's the correct solution? Put the absolute url in the <Link to="">?

I don't know what you mean by "avoid the broken link checker".

The broken links checker verifies links that "look internal" (without a protocol) created through the <Link> component (also used under the hood for some themeConfig elements).

You can avoid checking a link through the secret data-noBrokenLinkCheck.

The presence of a static XML file anywhere in the codebase doesn't enable/disable the check of a given link; only the link configuration does.

The links to check are checked against known internal Docusaurus routes, which are usually created by js/jsx/ts/tsx/md/mdx files.

Unless you use fancy third-party plugins, the presence of an XML file anywhere doesn't create an internal route, and it won't have any visible impact when checking the links.

The broken link checker is meant to check "navigation links", and it's not made to check links to assets.