Fix pip-compile handling of `-r` paths which are not subpaths of the cwd #2260

sirosen · 2025-11-04T23:12:06Z

resolves #2231

When -r is used to pass a path, we are normalizing with pathlib.Path.relative_to. This fails when the input is not a subpath of the current working directory.

In Python 3.12+ pathlib supports this usage (walk_up=True), but on older Pythons we need to fallback to using os.path.relpath.

In order to support this usage and give a clear indication of how we should upgrade when older versions are dropped, a small compat wrapper is here defined, _relative_to_walk_up which uses relative_to(..., walk_up=True) when it is available, and only uses os.path.relpath when necessary. After we drop Python 3.11 in a few years, the helper can be removed.

A new regression test reproduces #2231 and is fixed with this patch.
As part of writing that test, I wanted to leverage the TestFilesCollection helper, so this was extended in a small way to support usage without an explicit name.

Contributor checklist

Included tests for the changes.
A change note is created in changelog.d/ (see changelog.d/README.md for instructions) or the PR text says "no changelog needed".

Maintainer checklist

If no changelog is needed, apply the skip-changelog label.
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

webknjaz · 2025-11-05T01:05:36Z

changelog.d/2231.bugfix.md

@@ -0,0 +1,2 @@
+Fix `pip-compile` handling of relative path includes which are not subpaths of


non-imperative mood plz

I need to break this habit of writing changelogs like commit messages. I even updated docs at work to remind people to use past tense for changelogs, and then I go around forgetting it myself! 😆

Thanks for the save.

I wonder if we can get an LLM help out with this. I've been looking into configuring some LLM behaviors (CLAUDE.md / AGENTS.md) recently and this should be a func exercise.. Although, we could probably also try Vale.

I'm interested in Vale, I think it looks very cool.

Less interested in using an LLM for this, personally. I'm waiting for the hype to die down, and plan to be a late adopter of the tech (after some not-great experiences trying it so far) assuming I can square it with my ethics.

I think it's important to have these files in the repo because people are going to use agentic help regardless of what we tell them. And if there's some guidelines, the chance of them producing rubbish would be decreased. With that, they wouldn't waste as much compute retrying or generating nonsense. So in my book helping such tools be efficient is definitely positive if we're talking about the environmental impact.

We just got such configuration in ansible/ansible#85841 just now and I'm interested in exploring making such configs for projects better.

@sirosen re:vale — I have this old issue where I collect various prose checkers @ aio-libs/multidict#276. If we succeed in integating any of them, I'd like to scale that to other projects.

changelog.d/2231.bugfix.md

webknjaz · 2025-11-05T01:06:38Z

piptools/_compat/pip_compat.py

+def _relative_to_walk_up(path: pathlib.Path, start: pathlib.Path) -> pathlib.Path:
+    """
+    Compute a relative path allowing for the input to not be a subpath of the start.
+
+    This is a compatibility helper for ``pathlib.Path.relative_to(..., walk_up=True)``
+    on all Python versions. (``walk_up: bool`` is Python 3.12+)
+    """
+    # prefer `pathlib.Path.relative_to` where available
+    if sys.version_info >= (3, 12):
+        return path.relative_to(start, walk_up=True)
+
+    str_result = os.path.relpath(path, start=start)
+    return pathlib.Path(str_result)


Perhaps this should go into a separate pathlib_compat.py?

My only concern would be that it might only ever have this one function.
What about a path_compat.py so that it's not only scoped to pathlib?

I wouldn't move other things over right away, but any helpers for handling drive letters and such could go there.

I snooped around and immediately found a second helper to put in here!

I still prefer path_compat.py, so I'll go with that for the moment, but I'm happy to rename as well.

I unfortunately ended up with a little bit of extra change needed ( 3df3efa ) because my desire to put two related helpers together created an import cycle. I can revert if this feels like too much extraneous change.

@sirosen circular imports is not a new problem. They've been annoying people trying to organize helpers for quite some time: #2108.

I think we should postpone moving other things into the new helper module as to keep the PR atomic and prevent scope creep growing infinitely.

I think you're right, especially coming back to this after a couple of days.
I'm going to partially revert, cutting it back to just this one helper in a path_compat.py module, as part of my rebase right now.

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

webknjaz · 2025-11-06T02:39:25Z

piptools/_compat/path_compat.py

+
+
+def is_path_relative_to(path1: pathlib.Path, path2: pathlib.Path) -> bool:
+    """Return True if ``path1`` is relative to ``path2``."""


The literal can also be linked.

Suggested change

"""Return True if ``path1`` is relative to ``path2``."""

"""Return :data:`True` if ``path1`` is relative to ``path2``."""

But as I said elsewhere, let's not include this helper.

Since I reverted any move of this helper, I'm not changing the docstring, but 👍 to marking literals like this when possible.

webknjaz · 2025-11-06T02:42:29Z

piptools/_compat/path_compat.py

+    # TODO: remove this function in favor of Path.is_relative_to()
+    #       when we drop support for Python 3.8


Both 3.8 and 3.9 are both EOL already: https://endoflife.date/python. And testing against pip is becoming difficult in older runtimes.

I think we could plan dropping Python 3.8 once #2257 is implemented and released. With that, there would be no need to even think about hosting this function in compat modules.

webknjaz · 2025-11-06T02:43:47Z

piptools/_compat/path_compat.py

@@ -0,0 +1,36 @@
+"""
+Compatibility helpers for working with paths and ``pathlib`` across platforms


Suggested change

Compatibility helpers for working with paths and ``pathlib`` across platforms

Compatibility helpers for working with paths and :mod:`pathlib` across platforms

piptools/_compat/path_compat.py

When `-r` is used to pass a path, we are normalizing with `pathlib.Path.relative_to`. This fails when the input is not a subpath of the current working directory. In Python 3.12+ pathlib supports this usage (`walk_up=True`), but on older Pythons we need to fallback to using `os.path.relpath`. In order to support this usage and give a clear indication of how we should upgrade when older versions are dropped, a small compat wrapper is here defined, `_relative_to_walk_up` which uses `relative_to(..., walk_up=True)` when it is available, and only uses `os.path.relpath` when necessary. After we drop Python 3.11 in a few years, the helper can be removed. A new regression test reproduces jazzband#2231 and is fixed with this patch. As part of writing that test, I wanted to leverage the `TestFilesCollection` helper, so this was extended in a small way to support usage without an explicit name.

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

`piptools._compat.path_compat` is a new module for holding any helpers dedicated to path inspection and manipulations.

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

sirosen added this to the 7.5.2 milestone Nov 4, 2025

sirosen requested review from AndydeCleyre and webknjaz November 4, 2025 23:12

sirosen added the bug Something is not working label Nov 4, 2025

psf-chronographer bot added the bot:chronographer:provided label Nov 4, 2025

webknjaz reviewed Nov 5, 2025

View reviewed changes

changelog.d/2231.bugfix.md Show resolved Hide resolved

webknjaz reviewed Nov 5, 2025

View reviewed changes

sirosen force-pushed the improve-relpath-handling branch from 4ef3b4c to 21e5b23 Compare November 5, 2025 03:58

sirosen added a commit to sirosen/pip-tools that referenced this pull request Nov 5, 2025

Minor changelog improvements for jazzband#2260

21e5b23

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

sirosen force-pushed the improve-relpath-handling branch from 3e781db to 95a824d Compare November 5, 2025 04:41

sirosen added a commit to sirosen/pip-tools that referenced this pull request Nov 5, 2025

Minor changelog improvements for jazzband#2260

d585ddd

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

sirosen force-pushed the improve-relpath-handling branch from 3df3efa to bc8d8c8 Compare November 5, 2025 14:50

webknjaz reviewed Nov 6, 2025

View reviewed changes

piptools/_compat/path_compat.py Show resolved Hide resolved

sirosen and others added 3 commits November 6, 2025 13:43

Minor changelog improvements for jazzband#2260

4e6085c

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

Move pathlib compatibility code to dedicated module

3a76d39

`piptools._compat.path_compat` is a new module for holding any helpers dedicated to path inspection and manipulations.

sirosen force-pushed the improve-relpath-handling branch from bc8d8c8 to 7c7fff5 Compare November 6, 2025 19:48

Apply review suggestions

d7da20b

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

sirosen force-pushed the improve-relpath-handling branch from 7c7fff5 to d7da20b Compare November 6, 2025 20:05

webknjaz approved these changes Nov 7, 2025

View reviewed changes

webknjaz added this pull request to the merge queue Nov 7, 2025

Merged via the queue into jazzband:main with commit 30ed588 Nov 7, 2025
49 checks passed

		@@ -0,0 +1,2 @@
		Fix `pip-compile` handling of relative path includes which are not subpaths of



		def is_path_relative_to(path1: pathlib.Path, path2: pathlib.Path) -> bool:
		"""Return True if ``path1`` is relative to ``path2``."""

	"""Return True if ``path1`` is relative to ``path2``."""
	"""Return :data:`True` if ``path1`` is relative to ``path2``."""

		# TODO: remove this function in favor of Path.is_relative_to()
		# when we drop support for Python 3.8

		@@ -0,0 +1,36 @@
		"""
		Compatibility helpers for working with paths and ``pathlib`` across platforms

Uh oh!

Fix pip-compile handling of -r paths which are not subpaths of the cwd #2260

Fix pip-compile handling of -r paths which are not subpaths of the cwd #2260

Uh oh!

Conversation

sirosen commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contributor checklist

Maintainer checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Fix pip-compile handling of `-r` paths which are not subpaths of the cwd #2260

Fix pip-compile handling of `-r` paths which are not subpaths of the cwd #2260

sirosen commented Nov 4, 2025 •

edited

Loading