OSError: Error reading file '/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml': failed to load external entity "/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml"

[HelloYLee]

HI, I didn't deploy s3fs or cvmfs but I use aliyun cloud oss bucket to create refdata pv & pvc, but some pods failed to start and I check pod logs found "
OSError: Error reading file '/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml': failed to load external entity "/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml"
"
I found that /cvmfs/data.galaxyproject.org/ mounted to refdata pvc(oss bucket), but I have no idea how to get this data, please help!

[afgane]

Thanks for reporting this. We noticed a similar issue yesterday and are looking into it but haven't gotten to the bottom of it yet. Will update here as things progress.

[ksuderman]

@HelloYLee could you please try again? This may have been a transient error somewhere that seems to have resolved itself, at least my test cases are working again.

Nevermind, the resolution seems to have been a transient anomaly and /cvmfs/data.galaxyproject.org is broken again...

[iadiadiad]

Hello. I'm seeing the same issue. Is there a known workaround or fix?

Traceback (most recent call last):
  File "/galaxy/server/lib/galaxy/webapps/galaxy/buildapp.py", line 60, in app_pair
    app = galaxy.app.UniverseApplication(global_conf=global_conf, is_webapp=True, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/app.py", line 731, in __init__
    self._configure_tool_data_tables(from_shed_config=False)
  File "/galaxy/server/lib/galaxy/app.py", line 396, in _configure_tool_data_tables
    self.tool_data_tables: BaseToolDataTableManager = ToolDataTableManager(
                                                      ^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/tool_util/data/__init__.py", line 943, in __init__
    self.load_from_config_file(single_config_filename, self.tool_data_path, from_shed_config=False)
  File "/galaxy/server/lib/galaxy/tool_util/data/__init__.py", line 995, in load_from_config_file
    tree = util.parse_xml(config_filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/util/__init__.py", line 352, in parse_xml
    tree = cast(ElementTree, etree.parse(str(fname), parser=parser))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/lxml/etree.pyx", line 3548, in lxml.etree.parse
  File "src/lxml/parser.pxi", line 1879, in lxml.etree._parseDocument
  File "src/lxml/parser.pxi", line 1905, in lxml.etree._parseDocumentFromURL
  File "src/lxml/parser.pxi", line 1808, in lxml.etree._parseDocFromFile
  File "src/lxml/parser.pxi", line 1180, in lxml.etree._BaseParser._parseDocFromFile
  File "src/lxml/parser.pxi", line 618, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 728, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 655, in lxml.etree._raiseParseError
OSError: Error reading file '/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml': failed to load external entity "/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml"

[afgane]

What are the versions of your setup? And where are you running the chart? We've seen this earlier this week in one environment but have not been able to replicate it yet elsewhere.

[iadiadiad]

Using chart v5.19.0 and deploying to AWS EKS.

[afgane]

When does it happen (eg, when installing the chart, running a job, installing a tool, etc)?

[iadiadiad]

It seems to happen as galaxy is initializing for the first time.

[iadiadiad]

I tried using v6.0.0 of the helm chart and received the same error. Galaxy just won't initialize, and the pod remains in CrashLoopBackOff. Kubernetes version is 1.32

Traceback (most recent call last):
  File "/galaxy/server/lib/galaxy/webapps/galaxy/buildapp.py", line 68, in app_pair
    app = galaxy.app.UniverseApplication(global_conf=global_conf, is_webapp=True, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/app.py", line 756, in __init__
    self._configure_tool_data_tables(from_shed_config=False)
  File "/galaxy/server/lib/galaxy/app.py", line 420, in _configure_tool_data_tables
    self.tool_data_tables: BaseToolDataTableManager = ToolDataTableManager(
                                                      ^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/tool_util/data/__init__.py", line 943, in __init__
    self.load_from_config_file(single_config_filename, self.tool_data_path, from_shed_config=False)
  File "/galaxy/server/lib/galaxy/tool_util/data/__init__.py", line 995, in load_from_config_file
    tree = util.parse_xml(config_filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/galaxy/server/lib/galaxy/util/__init__.py", line 365, in parse_xml
    with source.open("rb") as f:
         ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/pathlib.py", line 1013, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml'

[nuwang]

Can you check the logs of the cvmfs-csi related pods? Does everything seem to be in order?

[mapk-amazon]

I encounter the same issue. The mount works for me in the galaxy-celery pods. The files are also available on the pod there.

Update: /cvmfs/data.galaxyproject.org/data.galaxyproject.org/.. this is how the bucket is accessed as directory with s3-csi on the celery pod. My guess is that this commit broke it: f23b51c

Update 2: I changed as follows and it starts successfully.

        - mountPath: /cvmfs/
          name: refdata-gxy