pytest 9 is not consistently loading conftest.py , which then fails because I'm adding custom options

[zzzeek]

Hey there -

very difficult to reproduce this consistently, as the behavior changes arbitrarily. see my comment below for reproducer

The bad behavior looks like this:

[classic@framework sqlalchemy:try_pytest_9]$ .nox/tests-py313-sqlite-cext-backendonly/bin/python -m pytest -v  --db sqlite --write-idents ident.txt -s 
ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --db --write-idents
  inifile: /home/classic/dev/sqlalchemy/pyproject.toml
  rootdir: /home/classic/dev/sqlalchemy

Where the file https://github.com/sqlalchemy/sqlalchemy/blob/main/test/conftest.py is not being loaded before pytest parses the arguments. If I make random changes to the command, like the exact filename in my custom --write-idents option, then it works. or maybe it fails after a few runs. the behavior seems to be non-deterministic.

If I start pytest and specify test on the CLI as the folder to look in, then it will work every time, which makes sense OK:

[classic@framework sqlalchemy:try_pytest_9]$ .nox/tests-py313-sqlite-cext-backendonly/bin/python -m pytest -v  test/ --db sqlite --write-idents ident.txt -s 

but if I use the python_files option to do the same thing, that does not work:

[tool.pytest.ini_options]
addopts = "--tb native -v -r sfxX --maxfail=250 -p warnings -p logging --strict-markers"
# will still fail to read test/conftest.py at startup, non-deterministically
testpaths = ["test"]
python_files = "test_*.py"

reproducer (updated):

git clone https://github.com/sqlalchemy/sqlalchemy.git
cd sqlalchemy/
virtualenv .venv
.venv/bin/pip install pytest>=9 pytest-xdist typing_extensions
touch idents.txt
.venv/bin/python -m pytest --db sqlite --write-idents idents.txt

the --db and --write-idents options are custom arguments. it seems to complain more often when --write-idents is part of the CLI.

but mostly if we can look at what has changed about how conftest.py is loaded, mine is not getting loaded consistently. Here's part of a sample Jenkins run showing the problem

10:08:29  nox > Will store junit xml in junit-mssql-cext-backendonly.xml
10:08:29  nox > python -m pytest -n2 --max-worker-restart=5 --dburi 'mssql+pyodbc://scott:tiger^5HHH@mssql2022:1433/test?driver=ODBC+Driver+18+for+SQL+Server&trustservercertificate=yes&Encrypt=Optional' --dbdriver pyodbc --dbdriver pymssql --dbdriver aioodbc -m backend --write-idents db_idents.txt --junitxml junit-mssql-cext-backendonly.xml
10:08:32  =========================== sqlalchemy installation ============================
10:08:32  SQLAlchemy 2.1.0b1 (no user site)
10:08:32  Path: /home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/sqlalchemy/__init__.py
10:08:32  compiled extension enabled, e.g. /home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/sqlalchemy/engine/_util_cy.cpython-313-x86_64-linux-gnu.so 
10:08:32  ============================= test session starts ==============================
10:08:32  platform linux -- Python 3.13.7, pytest-9.0.0, pluggy-1.6.0 -- /home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/bin/python
10:08:32  cachedir: .pytest_cache
10:08:32  rootdir: /home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy
10:08:32  configfile: pyproject.toml
10:08:32  plugins: xdist-3.8.0
10:08:32  created: 2/2 workers
10:08:32  [gw1] node down: Traceback (most recent call last):
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/execnet/gateway_base.py", line 1291, in executetask
10:08:32      exec(co, loc)
10:08:32      ~~~~^^^^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/xdist/remote.py", line 420, in <module>
10:08:32      config = _prepareconfig(args, None)
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/config/__init__.py", line 358, in _prepareconfig
10:08:32      config: Config = pluginmanager.hook.pytest_cmdline_parse(
10:08:32                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
10:08:32          pluginmanager=pluginmanager, args=args
10:08:32          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10:08:32      )
10:08:32      ^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
10:08:32      return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
10:08:32             ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
10:08:32      return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
10:08:32             ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
10:08:32      raise exception
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
10:08:32      teardown.throw(exception)
10:08:32      ~~~~~~~~~~~~~~^^^^^^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/helpconfig.py", line 124, in pytest_cmdline_parse
10:08:32      config = yield
10:08:32               ^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
10:08:32      res = hook_impl.function(*args)
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1155, in pytest_cmdline_parse
10:08:32      self.parse(args)
10:08:32      ~~~~~~~~~~^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1540, in parse
10:08:32      self._parser.parse(args, namespace=self.option)
10:08:32      ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/config/argparsing.py", line 145, in parse
10:08:32      return self.optparser.parse_intermixed_args(strargs, namespace=namespace)
10:08:32             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10:08:32    File "/opt/python3.13/lib/python3.13/argparse.py", line 2465, in parse_intermixed_args
10:08:32      self.error(msg)
10:08:32      ~~~~~~~~~~^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/config/argparsing.py", line 476, in error
10:08:32      raise UsageError(self.format_usage() + msg)
10:08:32  pytest.UsageError: usage: -c [options] [file_or_dir] [file_or_dir] [...]
10:08:32  -c: error: unrecognized arguments: --dburi --dbdriver --dbdriver --dbdriver --write-idents
10:08:32    inifile: /home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/pyproject.toml
10:08:32    rootdir: /home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy
10:08:32  
10:08:32  
10:08:32  replacing crashed worker gw1
10:08:32  [gw2] node down: Traceback (most recent call last):
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/execnet/gateway_base.py", line 1291, in executetask
10:08:32      exec(co, loc)
10:08:32      ~~~~^^^^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/xdist/remote.py", line 420, in <module>
10:08:32      config = _prepareconfig(args, None)
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/config/__init__.py", line 358, in _prepareconfig
10:08:32      config: Config = pluginmanager.hook.pytest_cmdline_parse(
10:08:32                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
10:08:32          pluginmanager=pluginmanager, args=args
10:08:32          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10:08:32      )
10:08:32      ^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
10:08:32      return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
10:08:32             ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
10:08:32      return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
10:08:32             ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
10:08:32      raise exception
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
10:08:32      teardown.throw(exception)
10:08:32      ~~~~~~~~~~~~~~^^^^^^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/helpconfig.py", line 124, in pytest_cmdline_parse
10:08:32      config = yield
10:08:32               ^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
10:08:32      res = hook_impl.function(*args)
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1155, in pytest_cmdline_parse
10:08:32      self.parse(args)
10:08:32      ~~~~~~~~~~^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1540, in parse
10:08:32      self._parser.parse(args, namespace=self.option)
10:08:32      ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/config/argparsing.py", line 145, in parse
10:08:32      return self.optparser.parse_intermixed_args(strargs, namespace=namespace)
10:08:32             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10:08:32    File "/opt/python3.13/lib/python3.13/argparse.py", line 2465, in parse_intermixed_args
10:08:32      self.error(msg)
10:08:32      ~~~~~~~~~~^^^^^
10:08:32    File "/home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/.nox/tests-py313-mssql-cext-backendonly/lib/python3.13/site-packages/_pytest/config/argparsing.py", line 476, in error
10:08:32      raise UsageError(self.format_usage() + msg)
10:08:32  pytest.UsageError: usage: -c [options] [file_or_dir] [file_or_dir] [...]
10:08:32  -c: error: unrecognized arguments: --dburi --dbdriver --dbdriver --dbdriver --write-idents
10:08:32    inifile: /home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy/pyproject.toml
10:08:32    rootdir: /home/jenkins/workspace/sqlalchemy/gerrit_21@3/py313-mssql-cext-backendonly/sqlalchemy/sqlalchemy
10:08:33  
10:08:33  

[bluetech]

Thanks for the report, I will try to see if I can reproduce it.

It will be interesting to know if this only happens with xdist.

[zzzeek]

the bug is the filename for my custom option is being interpreted as a test path. full repro:

git clone https://github.com/sqlalchemy/sqlalchemy.git
cd sqlalchemy/
virtualenv .venv
.venv/bin/pip install pytest>=9 pytest-xdist typing_extensions
touch idents.txt
.venv/bin/python -m pytest --db sqlite --write-idents idents.txt

this patch would fix:

diff --git a/src/_pytest/config/__init__.py b/src/_pytest/config/__init__.py
index 9b2afe3e8..6646f3b67 100644
--- a/src/_pytest/config/__init__.py
+++ b/src/_pytest/config/__init__.py
@@ -595,7 +595,7 @@ class PytestPluginManager(PluginManager):
 
             # Ensure we do not break if what appears to be an anchor
             # is in fact a very long option (#10169, #11394).
-            if safe_exists(anchor):
+            if safe_exists(anchor) and anchor.is_dir():
                 self._try_load_conftest(
                     anchor,
                     importmode,

[bluetech]

Thanks for the reproducer. Bisected to 115b870. Given the invocation touch idents.txt && pytest --db sqlite --write-idents idents.txt, before this commit Parser.parse_known_and_unknown_args would give (non-intermixed parse):

file_or_dir = ['sqlite']
unknown = ['--db', '--write-idents', 'idents.txt']

and after (intermixed parse):

file_or_dir = ['sqlite', 'idents.txt']
unknown = ['--db', '--write-idents']

The new behavior is more sensible (I cannot immediately explain the previous behavior). But it incidentally causes the problem in this way (relevant code here):

• Because idents.txt is now considered a possible arg during pre-parsing, it is considered for initial conftest discovery.
• It doesn't find anything (there is no conftest.py in any of its parents). But it is still considered foundanchor = True.
• Because foundanchor = True, we don't look for initial conftests in the invocation dir, which means the sqlalchemy/test* dirs and hence sqlalchemy/test/conftest.py conftest is not loaded, which means the --db and --write-idents don't get registered.

We need to consider how to help with this. The problem with pre-parse config flags interpreted as args is well known (there are some existing issues about it), but maybe not this exact variant. I'm thinking we should maybe have some reliable (conventional) location from which initial conftests are loaded without being affected by args. But not sure yet.

If you want a workaround, you can change the invocation to pytest --db=sqlite --write-idents=idents.txt (use equals instead of spaces).

---

Small aside: I recommend including the entries from the default norecursedirs in your config norecursedirs setting, otherwise there is some wasted effort looking through .git and whatnot.

[zzzeek]

OK, those are all good points, but can I ask about this code:

diff --git a/src/_pytest/config/__init__.py b/src/_pytest/config/__init__.py
index 9b2afe3e8..6646f3b67 100644
--- a/src/_pytest/config/__init__.py
+++ b/src/_pytest/config/__init__.py
@@ -595,7 +595,7 @@ class PytestPluginManager(PluginManager):
 
             # Ensure we do not break if what appears to be an anchor
             # is in fact a very long option (#10169, #11394).
-            if safe_exists(anchor):
+            if safe_exists(anchor) and anchor.is_dir():
                 self._try_load_conftest(
                     anchor,
                     importmode,

What is the valid pytest use case for self._try_load_conftest(anchor, ...) where anchor is not a directory ? there seems to be some logic in _loadconftestmodules that given a file, locates the parent directory, but in that case how is passing a filename "dbidents.txt" not just resolving to the current directory in this case, where it should then continue into test/ and locate the conftest.py file?

[bluetech]

Given pytest mytests/test_it.py the arg mytests/test_it.py is the anchor and is not a directory. So applying this diff as is won't work.

how is passing a filename "dbidents.txt" not just resolving to the current directory in this case, where it should then continue into test/ and locate the conftest.py file?

I'm not sure if it was done intentionally or not. It's possible this makes sense. This diff doesn't break any pytest tests at least:

diff --git i/src/_pytest/config/__init__.py w/src/_pytest/config/__init__.py
index 9b2afe3e8..319dba91e 100644
--- i/src/_pytest/config/__init__.py
+++ w/src/_pytest/config/__init__.py
@@ -634,6 +634,7 @@ class PytestPluginManager(PluginManager):
         *,
         consider_namespace_packages: bool,
     ) -> None:
+        anchor = self._get_directory(anchor)
         self._loadconftestmodules(
             anchor,
             importmode,

I think it will make your case work again (didn't test), but still incidentally (e.g. if the file was not in the root directory it would still fail). Would be good to have a principled fix (based on "source roots" or "workspaces" or such in the long term, or something simpler for the shorter term).

[zzzeek]

I guess the argument could be the name of an actual test script:

pytest test_things.py

in which case it wants to look in ./ for a conftest.py, and it does not want to look in test/ for such file, since test_things.py is in the root.

so instead of "argument can't be a file" it would need to say "this is a file but it's not a test script"

I think providing a way to force a conftest.py to be loaded would be best.

for now I am ending my test command with test/ so it uses that directory as the location for tests (but what makes it choose test/ and not ./ for db_idents.txt? )

[RonnyPfannschmidt]

we probably ought to consider testpaths for initial condtests, possibly even if thats not part of the command

the testroots proposal could also be used (using the testsroots as valid paths of initial conftests