Add support for all known instances in short URL format, removed dandi-staging and renamed EMBER* ones into EMBER-DANDI*

[Copilot]

• Modify regex in dandi/dandiarchive.py to dynamically build the instance name pattern from known_instances
• Update display string from DANDI:<dandiset id>[/<version>] to <INSTANCE>:<dandiset id>[/<version>]
• Add tests for DANDI-SANDBOX, LINC, EMBER, and ember-sandbox instances
• Update documentation in docs/source/ref/urls.rst to reflect all known instances support
• Fix test_helptext.py to expect the new <INSTANCE>: format in help text
• Verify all tests pass

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Adjust known_urls to support known instances</issue_title>
<issue_description>ATM we know many instances already but known_urls has it hardcoded for DANDI: only so download from any other would fail, e.g.

❯ dandi download DANDI-SANDBOX:000029
2025-11-21 15:27:26,566 [    INFO] Logs saved in /home/yoh/.local/state/dandi-cli/log/2025.11.21-20.27.26Z-117205.log
Error: We do not know how to map URL DANDI-SANDBOX:000029 to our servers.
Accepted resource identifier patterns:
 - DANDI:<dandiset id>[/<version>]
 - https://gui.dandiarchive.org/...
 - https://identifiers.org/DANDI:<dandiset id>[/<version id>] (<version id> cannot be 'draft')
 - https://<server>[/api]/[#/]dandiset/<dandiset id>[/<version>][/files[?location=<path>]]
 - https://*dandiarchive-org.netlify.app/...
 - https://<server>[/api]/dandisets/<dandiset id>[/versions[/<version>]]
 - https://<server>[/api]/assets/<asset id>[/download]
 - https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/<asset id>[/download]
 - https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/?path=<path>
 - https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/?glob=<glob>
 - dandi://<instance name>/<dandiset id>[@<version>][/<path>]
 - https://<server>/...

❯ dandi instances
dandi:
  api: https://api.dandiarchive.org/api
  gui: https://dandiarchive.org
dandi-api-local-docker-tests:
  api: http://localhost:8000/api
  gui: http://localhost:8085
dandi-sandbox:
  api: https://api.sandbox.dandiarchive.org/api
  gui: https://sandbox.dandiarchive.org
dandi-staging:
  api: https://api.sandbox.dandiarchive.org/api
  gui: https://sandbox.dandiarchive.org
ember:
  api: https://api-dandi.emberarchive.org/api
  gui: https://dandi.emberarchive.org
ember-sandbox:
  api: https://api-dandi-sandbox.emberarchive.org/api
  gui: https://apl-setup--ember-dandi-archive.netlify.app/
linc:
  api: https://api.lincbrain.org/api
  gui: https://lincbrain.org
linc-staging:
  api: https://staging-api.lincbrain.org/api
  gui: https://staging.lincbrain.org

Adjust so it allows for any of those known/hardcoded instances . For documentation example it should just list <INSTANCE>:<dandiset id>[/<version>] instead of DANDI:<dandiset id>[/<version>]</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Adjust known_urls to support known instances #1749

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Release notes

Added support for all known instances in short URL format (such as DANDI-SANDBOX:123456 or EMBER-DANDI:123456) matching the instance names known to the dandi CLI and library.
BREAKING: Note that INSTANCE: in above examples must be capital - lower case dandi:123456 would no longer be supported. Also version will be treated case-sensitive so must be draft not DRAFT.
Removed deprecated dandi-staging instance definition.
Renamed ember and ember-sandbox into ember-dandi and ember-dandi-sandbox correspondingly.

[Copilot]

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• api-dandi-sandbox.emberarchive.org
  • Triggering command: /usr/bin/python3 python3 (dns block)
  • Triggering command: /usr/bin/python3 python3 -m pytest dandi/tests/test_dandiarchive.py::test_parse_api_url -v --tb=short (dns block)
• api-dandi.emberarchive.org
  • Triggering command: /usr/bin/python3 python3 (dns block)
  • Triggering command: /usr/bin/python3 python3 -m pytest dandi/tests/test_dandiarchive.py::test_parse_api_url -v --tb=short (dns block)
• api.dandiarchive.org
  • Triggering command: /usr/bin/python3 python3 (dns block)
  • Triggering command: /usr/bin/python3 python3 -m pytest dandi/tests/test_dandiarchive.py::test_parse_api_url -v --tb=short (dns block)
• api.lincbrain.org
  • Triggering command: /usr/bin/python3 python3 (dns block)
  • Triggering command: /usr/bin/python3 python3 -m pytest dandi/tests/test_dandiarchive.py::test_parse_api_url -v --tb=short (dns block)
• api.sandbox.dandiarchive.org
  • Triggering command: /usr/bin/python3 python3 (dns block)
  • Triggering command: /usr/bin/python3 python3 -m pytest dandi/tests/test_dandiarchive.py::test_parse_api_url -v --tb=short (dns block)
• dandiarchive.org
  • Triggering command: /usr/bin/python3 python3 -m pytest dandi/tests/test_dandiarchive.py::test_parse_api_url -v --tb=short (dns block)
• sandbox.dandiarchive.org
  • Triggering command: /usr/bin/python3 python3 -m pytest dandi/tests/test_dandiarchive.py::test_parse_api_url -v --tb=short (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[yarikoptic]

I vaguely recall we might have a listing of supported regexes dumped/hardcoded somewhere in docs. Please check and adjust if needed

[codecov]

Codecov Report

❌ Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 75.02%. Comparing base (7cf6ace) to head (5f60b9b).
⚠️ Report is 8 commits behind head on master.

Files with missing lines	Patch %	Lines
dandi/utils.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1750   +/-   ##
=======================================
  Coverage   75.02%   75.02%           
=======================================
  Files          84       84           
  Lines       11888    11889    +1     
=======================================
+ Hits         8919     8920    +1     
  Misses       2969     2969           

Flag	Coverage Δ
unittests	75.02% <66.66%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yarikoptic]

@copilot I submitted for rebase, when CI finishes up - please review failures and whether relate to the changes - and where they do - fix them.

[Copilot]

I vaguely recall we might have a listing of supported regexes dumped/hardcoded somewhere in docs. Please check and adjust if needed

Found and updated docs/source/ref/urls.rst (line 20) which had the hardcoded DANDI: format. It now documents the short URL format as :samp:\{instance-name}:{dandiset-id}[/{version}]`` with examples of known instances (DANDI, DANDI-SANDBOX, LINC, EMBER, etc.). Commit 684a1b0.

@copilot I submitted for rebase, when CI finishes up - please review failures and whether relate to the changes - and where they do - fix them.

CI is still pending for this branch after the rebase. I ran the related tests locally (test_parse_dandi_url_unknown_instance) and they pass. The documentation update I made in 684a1b0 should also build correctly on Read the Docs. Once CI runs complete, I'll review any failures related to these changes.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• dandiarchive.org
  • Triggering command: /usr/bin/python python -m pytest dandi/tests/test_dandiarchive.py -v -k test_parse_dandi_url --tb=short (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[yarikoptic]

@copilot I think the test isn't happy yet

FAILED dandi/tests/test_helptext.py::test_resource_identifier_helptext - assert 'Accepted resource identifier patterns:\n   - DANDI:<dandiset id>[/<version>]\n' in "Usage: dandi ls [OPTIONS] PATH|URL\n\n  List .nwb files and dandisets metadata.\n\n  The arguments may be either resource identifiers or paths to local\n  files/directories.\n\n  RESOURCE ID/URLS:\n\n   dandi commands accept URLs and URL-like identifiers called <resource  ids>\n   in the following formats for identifying Dandisets, assets, and  asset\n   collections.\n\n   Text in [brackets] is optional.  A server field is a base API or GUI URL\n   for a DANDI Archive instance.  If an optional ``version`` field is  omitted\n   from a URL, the given Dandiset's most recent published version  will be\n   used if it has one, and its draft version will be used otherwise.\n\n  Accepted resource identifier patterns:\n   - <INSTANCE>:<dandiset id>[/<version>]\n   - [https://gui.dandiarchive.org/...\n](https://gui.dandiarchive.org/.../n)   - https://identifiers.org/DANDI:<dandiset id>[/<version id>] (<version id> cannot be 'draft')\n   - https://<server>[/api]/[#/]dandiset/<dandiset id>[/<version>][/files[?location=<path>]]\n   - https://*dandiarchive-org.netlify.app/...\n   - https://<server>[/api]/dandisets/<dandiset id>[/versions[/<version>]]\n   - https://<server>[/api]/assets/<asset id>[/download]\n   - https://<server>...ver>[/api]/dandisets/<dandiset id>/versions/<version>/assets/?glob=<glob>\n   - dandi://<instance name>/<dandiset id>[@<version>][/<path>]\n   - https://<server>/...\n\nOptions:\n  -F, --fields TEXT               Comma-separated list of fields to display.\n                                  An empty value to trigger a list of\n                                  available fields to be printed out\n  -f, --format [auto|pyout|json|json_pp|json_lines|yaml]\n                                  Choose the format/frontend for output. If\n                                  'auto', 'pyout' will be used in case of\n                                  multiple files, and 'yaml' for a single\n                                  file.\n  -r, --recursive                 Recurse into content of\n                                  dandisets/directories. Only .nwb files will\n                                  be considered.\n  -J, --jobs INTEGER              Number of parallel download jobs.  [default:\n                                  6]\n  --metadata [api|all|assets]\n  --schema VERSION                Convert metadata to new schema version\n  --help                          Show this message and exit.\n"
= 1 failed, 581 passed, 286 skipped, 6 deselected, 2 xfailed, 2 xpassed in 192.18s (0:03:12) =

[yarikoptic]

fix for testing failure

[Copilot]

@copilot I think the test isn't happy yet

FAILED dandi/tests/test_helptext.py::test_resource_identifier_helptext - assert 'Accepted resource identifier patterns:\n   - DANDI:<dandiset id>[/<ver...

Fixed in commit df9b186. Updated test_helptext.py to expect <INSTANCE>:<dandiset id>[/<version>] instead of the hardcoded DANDI:<dandiset id>[/<version>], matching the new generalized format that supports all known instances.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• dandiarchive.org
  • Triggering command: /usr/bin/python python -m pytest dandi/tests/test_dandiarchive.py -v -k test_parse_dandi_url (dns block)
• rig.mit.edu
  • Triggering command: /home/REDACTED/.local/bin/dandi dandi ls --help (dns block)
  • Triggering command: /home/REDACTED/.local/bin/dandi dandi download --help (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[yarikoptic]

damn

Approvals from users that collaborated with Copilot on changes will not satisfy review requirements.

inviting @candleindark

[candleindark]

My understanding is that _dandi_url_parser.parse parses an id or identifier of a Dandiset, as defined in https://github.com/dandi/dandi-schema/blob/f6e5aa69e1c6b4d3bbd122edc75f4db89f8977c4/dandischema/models.py#L1666-L1679, into a DandisetURL object. Per the definition, both id and identifier start with the instance name defined in https://github.com/dandi/dandi-schema/blob/f6e5aa69e1c6b4d3bbd122edc75f4db89f8977c4/dandischema/conf.py#L112-L121. However, this PR assumes the names of the known instances are the keys of known_instances. This assumption is not true. In https://github.com/aplbrain/dandi-infrastructure/pull/47/files#diff-6ea1fa151ea6453c7c8ca98d07be3693b1435072a5bfd2cd82fd369b63c015cc, we established the two EMBER instances to be named "EMBER-DANDI" and "EMBER-DANDI-SANDBOX", respectively, and there is currently no instance named "dandi-staging" but one that is named "DANDI-SANDBOX" per https://github.com/dandi/dandi-infrastructure/pull/224/files#diff-6ea1fa151ea6453c7c8ca98d07be3693b1435072a5bfd2cd82fd369b63c015cc.

With the current definition of known_instances, this PR will bring confusion even if it works. My suggestion is to update/correct known_instances.

[yarikoptic]

I have reread a few times but still failing to see a problem.

• yes, dandi-schema allows and defines some specific instance name
• yes, known_instances are a hard coded list of already known ones, which are samples of those allowed specific ones. Here we just allow for a specific name to be used to identify and instance. Let's zoom tomorrow to see where the confusion is on either side

[yarikoptic]

@dandi/ember would it be ok to change or they are in wide use and we should provide deprecation cycle?

We are just trying to match the names of the instances as in the INSTANCE_ID provided in infrastructure deployment and would be visible in their IDs on the instances (ATM they are still all DANDI:)

[sandyhider]

@yarikoptic we don't foresee any issues. Lets go ahead with the changes.

[candleindark]

I think this is ready to go except the minor issue what you pointed out at #1750 (comment).