Collection of Galaxy Admin Training Path Issues, Workarounds and Solutions

[rhiniodontypus]

Galaxy Admin Training - Issues

These are the issues I encountered during my Galaxy Admin Training Path tutorial runs between April and August 2025 on an Ubuntu 22.04 VM. This collection documents problems, workarounds, and suggestions for improvement across the complete Galaxy Admin Training learning pathway.

System Configuration:

Operating System: Ubuntu 22.04.5 LTS (GNU/Linux 5.15.0-143-generic x86_64)
VM Environment: openstack Nova
VM Specifications: 8 CPU cores, 15 GB RAM
Python Version: Python 3.10.12
Ansible Version: ansible [core 2.17.12]

---

STEP 1: ansible-galaxy

• ISSUE 1: Galaxy v23.2 requires miniconda version 25.3
• ISSUE 2: Failing to load welcome page in center panel
• ISSUE 3: "404 Not Found" error when using the display function ("eye icon")

STEP 2: backup-cleanup

• "Couldn't resolve module/action 'community.general.system.make'"

STEP 4: tus

• ISSUE 1: Set a fixed tusd version in order to avoid using a github token
• ISSUE 2: Galaxy v23.2 does not work with latest tusd version (v2.2.2)

STEP 9: data-library

• Library Permission Issues

STEP 12: job-destinations

• "Unhandled exception while caching job destination dynamic rule"

STEP 15: gxadmin

• "psql: error: connection to server [...] failed"

STEP 17: monitoring

• ISSUE 1: "ansible.builtin.include has been removed"
• ISSUE 2: "Failed to update apt cache"
• ISSUE 3: "ansible.builtin.include has been removed"
• ISSUE 4: "Timeout when waiting for 0.0.0.0:3000"

STEP 18: tiaas

• ISSUE 1: "Source /srv/galaxy/server/static/style/base.css not found"
• ISSUE 2: Server Error (500) when accessing database
• ISSUE 3: "Unable to run this job due to a cluster error"
• ISSUE 4: Order of roles
• ISSUE 5: Incomplete link

STEP 19: sentry

• ISSUE 1: "'AnsibleUndefinedVariable: ''sentry_domain'' is undefined'"
• ISSUE 2: "Configuration error - kwargs_from_env() got an unexpected keyword argument 'ssl_version'"
• ISSUE 3: "TypeError: kwargs_from_env() got an unexpected keyword argument 'ssl_version'"
• ISSUE 4: "'vault_sentry_password'' is undefined'"
• ISSUE 5: "Error: User: admin@example.com exists, use --force-update to force."
• ISSUE 6: "SSL: no alternative certificate subject name matches target host name 'sentry."
• ISSUE 7: No <my-domain>/error logs in Sentry
• ISSUE 8: Test Job Properties tool does not appear in the tool list

STEP 20: ftp

• ISSUE 1: "ansible.builtin.include has been removed."
• ISSUE 2: "Unable to start service proftpd"

STEP 21: beacon

• ISSUE 1: "msg: No package matching 'python39' is available"
• ISSUE 2: "msg: No package matching 'cronie' is available"
• ISSUE 3: "msg: 'No such file or directory: b''/usr/bin/pip3.9'''"
• ISSUE 4: "msg: 'Could not find the requested service crond: host'"
• ISSUE 5: Calling <my-domain>/beacon --> 404 Not found

---

---

Step 1 ansible-galaxy

ISSUE 1

ERROR

LibMambaUnsatisfiableError: Encountered problems while solving:  
  - package conda-23.9.0-py310hff52083_0 requires python >=3.10,<3.11.0a0, but none of the providers can be installed"

SOLUTION

Switch to galaxy_commit_id: release_23.2 in ./group_vars/galaxyservers.yml

also ref. to Step 4 / Issue 2

ISSUE 2

also documented here:
Failing to load welcome page in center panel after Ansible playbook deployment (404 Not Found) #6009

ERROR
After deploying Galaxy via the official Galaxy Admin tutorial playbook (Revision: 102, Last modification: Apr 8, 2025), the welcome page fails to load inside the main panel of the Galaxy UI. Instead, the browser displays a 404 Not Found error:

SOLUTION

./galaxy/roles/galaxyproject.nginx/templates/nginx/galaxy.j2

Switch from alias {{ galaxy_server_dir }}/static/welcome.html.sample; to alias {{ galaxy_server_dir }}/static/welcome.sample.html;

ISSUE 3

Also documented here
404 Not Found" error when displaying datasets #6212

ERROR
404 Not Found error (e.g. when trying to display a dataset using the "eye icon" in the history column)

404 Not Found
The resource could not be found.

No route for /_x_accel_redirect/data/datasets/1/3/e/dataset_xyz.dat

SOLUTION

Replace the following block in ./templates/nginx/galaxy.j2:

	location /_x_accel_redirect {
		internal;
		alias /;
	}

with

    location /_x_accel_redirect/ {
        internal;
        alias /;
        # Add upstream response headers that would otherwise be omitted
        add_header Access-Control-Allow-Origin $upstream_http_access_control_allow_origin;
        add_header Access-Control-Allow-Methods $upstream_http_access_control_allow_methods;
    }

RESOURCES
https://help.galaxyproject.org/t/404-not-found-error-no-route-for-x-accel-redirect/7507
https://docs.galaxyproject.org/en/master/admin/nginx.html#sending-files-with-nginx

Step 2 backup-cleanup

ERROR

ERROR! couldn't resolve module/action 'community.general.system.make'. This often indicates a misspelling, missing collection, or incorrect module path.

The error appears to be in '/home/ubuntu/galaxy/roles/galaxyproject.gxadmin/tasks/main.yml': line 16, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- name: Compile gxadmin when there are updates
  ^ here

SOLUTION

Switch to community.general.make in galaxy/roles/galaxyproject.gxadmin/tasks/main.yml': line 16, column 3

Step 4 tus

ISSUE 1

ERROR
GitHub has rate limited the automated access and requires now a token authentification access:

SOLUTION

The tusd role contains

when: tusd_version == 'latest'

so the playbook alway tries to get the latest version of tusd and therefore has to access Github. This can be avoided by overwriting the tusd version in group_vars/galaxyservers.yml:

 galaxy_tusd_port: 1080
 galaxy_tus_upload_store: /data/tus
 tusd_version: "v1.7.3"

ISSUE 2

ERROR
Already described here:
Galaxy 23.2 does not work with latest tusd (v2.2.2) #17570

WORKAROUND
Ref to. Issue 1:
Set a known version to work (tusd v1.7.3)

(Potential) TO DO
Figure out the latest combination of Galaxy and tusd to work.

Step 9 data-library

ERROR
Executing
set-library-permissions -g http(s)://<your_host_IP> -a <api-key> LIBRARY --roles ROLES role1,role2,role3
throws the error:

Traceback (most recent call last):
File "/home/ubuntu/ephemeris_venv/bin/set-library-permissions", line 8, in <module>
  sys.exit(main())
File "/home/ubuntu/ephemeris_venv/lib/python3.10/site-packages/ephemeris/set_library_permissions.py", line 119, in main
  args.roles = [r.strip() for r in args.roles.split(",")]
AttributeError: 'list' object has no attribute 'split'

SOLUTION
?

Step 12 job-destinations

ERROR
Job Resource Selectors error. The job executed with personal parameter setting Cores = 2 fails:

`Encountered an unhandled exception while caching job destination dynamic rule.`

WORKAROUND
Uncommented rule in files/galaxy/config/tpv_rules_local.yml:

  .*testing.*:
  cores: 2
  mem: cores * 4
  rules:
    - id: admin_only_testing_tool
      if: |
        # Only allow the tool to be executed if the user is an admin
        admin_users = app.config.admin_users
        # last line in block must evaluate to a value - which determines whether the TPV if conditional matches or not
        not user or user.email not in admin_users
      fail: Unauthorized. Only admins can execute this tool.
    # - id: resource_params_defined
    #   if: |
    #     param_dict = job.get_param_values(app)
    #     param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
    #   cores: int(job.get_param_values(app)['__job_resource']['cores'])
    #   params:
    #     walltime: "{int(job.get_param_values(app)['__job_resource']['time'])}"

---

Step 15 gxadmin

ERROR

$ gxadmin query old-histories 10  

psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  role "ubuntu" does not exist

SOLUTION
Use sudo -u galaxy gxadmin query latest-users

---

Step 17 monitoring

ISSUE 1

ERROR

$ ansible-playbook monitoring.yml  

ERROR! [DEPRECATED]: ansible.builtin.include has been removed. Use include_tasks or import_tasks instead. This feature was removed from ansible-core in a release after 2023-05-16. Please update your playbooks.

The error appears to be in '/home/ubuntu/galaxy/roles/usegalaxy_eu.influxdb/tasks/main.yml': line 25, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Configure collectd
  ^ here

SOLUTION
Replace - include: with include_tasks: in galaxy/roles/usegalaxy_eu.influxdb/tasks/main.yml:

- name: Configure collectd
include_tasks: collectd.yml
when: influxdb_collectd_enabled == "true"

- include_tasks: install-download.yml
  when: influxdb_install_method == "download"
  tags:
    - influxdb

- include_tasks: install-debian.yml
  when: influxdb_install_method == "repository" and ansible_distribution in ["Debian", "Ubuntu"]
  tags:
    - influxdb

- include_tasks: install-rhel.yml
  when: influxdb_install_method == "repository" and ansible_distribution in ["Enterprise Linux", "CentOS"]
  tags:
    - influxdb

ISSUE 2

ERROR

TASK [usegalaxy_eu.influxdb : Add InfluxData repository] *************************************************************************************************************************************************
fatal: [<my-domain>]: FAILED! => changed=false 
  msg: 'Failed to update apt cache: W:GPG error: https://repos.influxdata.com/ubuntu jammy InRelease: The following signatures couldn''t be verified because the public key is not available: NO_PUBKEY D8FF8E1F7DF8B07E, E:The repository ''https://repos.influxdata.com/ubuntu jammy InRelease'' is not signed., W:Updating from such a repository can''t be done securely, and is therefore disabled by default., W:See apt-secure(8) manpage for repository creation and user configuration details., W:https://apt.bi.denbi.de/repos/apt/jammy/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details., W:https://ppa.launchpadcontent.net/apptainer/ppa/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details., W:https://cvmrepo.web.cern.ch/cvmrepo/apt/dists/jammy-prod/Release.gpg: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg),
    see the DEPRECATION section in apt-key(8) for details., W:https://ppa.launchpadcontent.net/natefoo/slurm-drmaa/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.'

(TEMPORARY) SOLUTION
After manually running

$ wget -q https://repos.influxdata.com/influxdata-archive.key
$ gpg --show-keys --with-fingerprint --with-colons ./influxdata-archive.key 2>&1 | grep -q '^fpr:\+24C975CBA61A024EE1B631787C3D57159FC2F927:$' && cat influxdata-archive.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/influxdata-archive.gpg > /dev/null
$ echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive.gpg] https://repos.influxdata.com/debian stable main' | sudo tee /etc/apt/sources.list.d/influxdata.list

Source: https://repos.influxdata.com/

ISSUE 3

  $ ansible-playbook monitoring.yml 

  ERROR! [DEPRECATED]: ansible.builtin.include has been removed. Use include_tasks or import_tasks instead. This feature was removed from ansible-core in a release after 2023-05-16. Please update your playbooks.

  The error appears to be in '/home/ubuntu/galaxy/roles/cloudalchemy.grafana/tasks/main.yml': line 17, column 3, but may
  be elsewhere in the file depending on the exact syntax problem.

  The offending line appears to be:

  - include: preflight.yml
    ^ here

SOLUTION
Replace all include_tasks with import_tasks in galaxy/roles/cloudalchemy.grafana/tasks/main.yml.

ISSUE 4

ERROR
When running the monitoring notebook monitoring.yml the start up process of Grafana exits with a time out error:

TASK [cloudalchemy.grafana : Wait for grafana to start] **************************************************************************************************
fatal: [<my-domain>]: FAILED! => changed=false 
elapsed: 300
msg: Timeout when waiting for 0.0.0.0:3000

SOLUTION

Replacing

# Alerting
[alerting]
{% if grafana_alerting != {} %}
enabled = true

with

# Alerting
[unified_alerting]
{% if grafana_alerting != {} %}
enabled = true

in roles/cloudalchemy.grafana/templates/grafana.ini.j2 works.

---

Step 18 tiaas

ISSUE 1

ERROR

TASK [galaxyproject.tiaas2 : Copy Galaxy's stylesheet] ***************************************************************************************************
fatal: [<my-domain>]: FAILED! => changed=false 
msg: Source /srv/galaxy/server/static/style/base.css not found

SOLUTION

The correct path to base.css is

tiaas_galaxy_stylesheet: "{{ galaxy_server_dir }}/static/dist/base.css

for which the role should be modified in ./galaxy/roles/galaxyproject.tiaas2/defaults/main.yml

ISSUE 2

ERROR

After finishing the playboook the static website (<my-domain>/tiaas) worked but all applications using the database (e.g. when trying to login as admin (<my-domain>/tiaas/admin) or accessing the calendar) led to an Server Error (500).

WORKAROUND

1. Set tiaas_tiaas_use_sqlite: false in roles/galaxyproject.tiaas2/defaults/main.yml.
2. I also had to manually create a tiaas db in PostgreSQL.

ISSUE 3

ERROR
Executing a job directed to slurm inside the training role throws the error:

Unable to run this job due to a cluster error

WORKAROUND

I had to set --mem=1 for native_specification: --nodes=1 --ntasks=1 --mem=1 --cpus-per-task={cores} --time=00:30:00 in files/galaxy/config/tpv_rules_local.yml as set in the node configuration:

$ scontrol show nodes

NodeName=localhost Arch=x86_64 CoresPerSocket=1 
[...]
 CfgTRES=cpu=2,mem=1M,billing=2
[...]

ISSUE 4

https://training.galaxyproject.org/training-material/topics/admin/tutorials/tiaas/tutorial.html#setting-up-tiaas

"4. We need to add the `galaxyproject.tiaas2` role before the `nginx` role, as TIaaS defines variables that Nginx needs.  "
```YAML
-- a/galaxy.yml
++ b/galaxy.yml
    @ -47,6 +47,7 @@
    - galaxyproject.nginx
    - geerlingguy.docker
    - usegalaxy_eu.rabbitmqserver
    - galaxyproject.tiaas2
    - galaxyproject.gxadmin
    - galaxyproject.cvmfs
    - dj-wasabi.telegraf

The tiaas2 role in the code template is still located after the nginx role.

ISSUE 5

The link to the page in section 4. Monitor TIaaS status (Hands On: Using TIaaS) is incomplete:

/join-training/gat should read /join-training/gat/status

Step 19 sentry

ISSUE 1

ERROR
The playbook run breaks throwing the error that the sentry_domain (= sentry.<my-domain>.de) is not defined:

TASK [galaxyproject.nginx : Install SSL vhost configs] **********************************************************************************************************
ok: [<my-domain>] => (item=galaxy)
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.errors.AnsibleUndefinedVariable: 'sentry_domain' is undefined
failed: [<my-domain>] (item=sentry) => changed=false 
  ansible_loop_var: item
  item: sentry
  msg: 'AnsibleUndefinedVariable: ''sentry_domain'' is undefined'

WORKAROUND
The root cause might be:
Galaxy needs to know the sentry domain in order to deploy the nginx config. If, however, the sentry playbook is not executed before the galaxy playbook, the playbook breaks during the task galaxyproject.nginx : Install SSL vhost configs. Therefore, I had to hardcode the domain here: sentry_domain: sentry.<my-domain>.de in group_vars/galaxyservers.yml.

ISSUE 2

ERROR
The playbook run breaks with

TASK [mvdbeek.sentry_selfhosted : Start sentry self hosted docker-compose project] *********************************************************
fatal: [sentry.<my-domain>]: FAILED! => changed=false 
msg: Configuration error - kwargs_from_env() got an unexpected keyword argument 'ssl_version'

SOLUTION
The sentry task (./galaxy/roles/mvdbeek.sentry_selfhosted/tasks/main.yml --> Start sentry self hosted docker-compose project) in the tutorial is

- name: Start sentry self hosted docker-compose project
docker_compose:
project_src: "{{ sentry_docker_compose_project_folder }}"
remove_orphans: true
state: present

docker_compose is referring to docker-compose version 1 which is depricated since 2023 (Issue 3194).
Ansible release > 9.2.0 requires docker compose V2 which should be implemented as described in community.docker.docker_compose_v2 module

- name: Start sentry self hosted docker-compose project
community.docker.docker_compose_v2:
project_src: "{{ sentry_docker_compose_project_folder }}"
remove_orphans: true
state: present

ISSUE 3

ERROR
The same issue from above occurs also in the next task:

TASK [mvdbeek.sentry_selfhosted : Add sentry super users] **********************************************************************************
failed: [sentry.<my-domain>] (item={'email': 'admin@example.com', 'password': '<password>'}) => changed=true 
  [...]
  stderr: |-
    Traceback (most recent call last):
      File "/usr/local/bin/docker-compose", line 8, in <module>
        sys.exit(main())
      File "/usr/local/lib/python3.10/dist-packages/compose/cli/main.py", line 81, in main
        command_func()
      File "/usr/local/lib/python3.10/dist-packages/compose/cli/main.py", line 200, in perform_command
        project = project_from_options('.', options)
      File "/usr/local/lib/python3.10/dist-packages/compose/cli/command.py", line 60, in project_from_options
        return get_project(
      File "/usr/local/lib/python3.10/dist-packages/compose/cli/command.py", line 152, in get_project
        client = get_client(
      File "/usr/local/lib/python3.10/dist-packages/compose/cli/docker_client.py", line 41, in get_client
        client = docker_client(
      File "/usr/local/lib/python3.10/dist-packages/compose/cli/docker_client.py", line 124, in docker_client
        kwargs = kwargs_from_env(environment=environment, ssl_version=tls_version)
    TypeError: kwargs_from_env() got an unexpected keyword argument 'ssl_version'
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
...ignoring

SOLUTION
Change the docker command part in ./galaxy/roles/mvdbeek.sentry_selfhosted/tasks/main.yml --> Add sentry super users from

docker-compose run --rm web createuser

to

docker compose run --rm web createuser

(w/o hyphen) according to the new docker compose V2 release:

- name: Add sentry super users
command:
  chdir: "{{ sentry_docker_compose_project_folder }}"
  cmd: >
    docker compose run --rm web createuser
      --email "{{ item.email }}"
      --password "{{ item.password }}"
      --superuser
      --no-input
loop: "{{ sentry_superusers }}"
ignore_errors: true

ISSUE 4

ERROR

TASK [mvdbeek.sentry_selfhosted : Add sentry super users] *******************************************************************************************************
fatal: [sentry.<my-domain>]: FAILED! => 
msg: '[{''email'': ''admin@example.com'', ''password'': ''{{ vault_sentry_password }}''}]: ''vault_sentry_password'' is undefined'
...ignoring

SOLUTION
There is no reference to the group_vars/secret.yml vault file in the sentry.yml playbook.
Adding the reference solves this issue:

- hosts: sentryservers
  vars_files:
    - group_vars/secret.yml
  become: true
  pre_tasks:
    - pip:
        name: docker-compose
  roles:
    - mvdbeek.sentry_selfhosted

ISSUE 5

After running the sentry.yml playbook you might encounter the error:

TASK [mvdbeek.sentry_selfhosted : Add sentry super users] *******************************************************************************************************
failed: [sentry.<my-domain>] (item={'email': 'admin@example.com', 'password': '<my-password>'}) => changed=true 
  [...]
    Container sentry-self-hosted-redis-1  Healthy
    Container sentry-self-hosted-redis-1  Healthy
    /usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
      if key is '':
    /usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
      if key_extra_len is 0:
    Error: User: admin@example.com exists, use --force-update to force.
  stderr_lines: <omitted>
  stdout: |-
    Updating certificates in /etc/ssl/certs...
    0 added, 0 removed; done.
    Running hooks in /etc/ca-certificates/update.d...
    done.
    13:25:28 [INFO] sentry.plugins.github: apps-not-configured
  stdout_lines: <omitted>
...ignoring

SOLUTION
Add --force-update to roles/mvdbeek.sentry_selfhosted/tasks/main.yml:

- name: Add sentry super users
  command:
    chdir: "{{ sentry_docker_compose_project_folder }}"
    cmd: >
      docker compose run --rm web createuser
        --email "{{ item.email }}"
        --password "{{ item.password }}"
        --superuser
        --no-input
        --force-update
  loop: "{{ sentry_superusers }}"
  ignore_errors: true

ISSUE 6

ERROR
Calling the sentry.<my-domain> subdomain does not create a secure connection saying that there is no valid certificate.

curl -v "https://sentry.<my-domain>" -o /dev/null

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 100.113.16.120:443...
* Connected to sentry.<my-domain> (100.113.16.120) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
[...]
* Server certificate:
*  subject: CN=<my-domain>
*  start date: Jul  8 12:35:22 2025 GMT
*  expire date: Oct  6 12:35:21 2025 GMT
*  subjectAltName does not match sentry.<my-domain>
* SSL: no alternative certificate subject name matches target host name 'sentry.<my-domain>'


0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
[]
curl: (60) SSL: no alternative certificate subject name matches target host name 'sentry.<my-domain>'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

WORKAROUND
According to tutorial I don't need a new certificate for the subdomain, which did not work in my case. Although the vault_galaxy_sentry_dsn entry in group_vars/secret.yml was ...@localhost:9000/2 a secure connection was not possible with all browsers and on the terminal level using curl.

Adding the subdomain to the certbot_domains in group_vars/galaxyservers.yml

certbot_domains:
 - "{{ inventory_hostname }}"
 - "{{ sentry_domain }}"

and deleting the folder /etc/letsencrypt + rerunning the playbooks finally created a new certificate including the new subdomain sentry.<my-domain>

ISSUE 7

ERROR
Probably related to ISSUE 6:
The logs of the /error route to intentionally create a fake error cannot be seen in Sentry.

WORKAROUND
The Sentry DSN (Data Source Name) entry in group_vars/secrets.yml according to the tutorial is:
vault_galaxy_sentry_dsn: 'https://KEY@sentry.<my-domain>:9000/2'.
This is telling Galaxy to connect directly to port 9000 on sentry., but port 9000 is only listening for HTTP connections (not HTTPS) (--> nginx config: proxy_pass "http://localhost:9000";). The nginx proxy handles the SSL termination on port 443:

# Listen on port 443
      listen        *:443 ssl;
      # The virtualhost is our domain name
      server_name   "sentry.<my-domain>";

Removing the ':9000' port from the DSN vault_galaxy_sentry_dsn: 'https://KEY@sentry.<my-domain>/2' resolved the SSL error.

ISSUE 8

ERROR
The Test Job Properties tool to send failing tool errors to Sentry does not appear in the tool list.

SOLUTION

The XML code does not work. Here is a working version:

<tool id="job_properties" name="Test Job Properties" version="1.0.0">
    <stdio>
        <exit_code range="127" level="fatal" description="Failing exit code." />
    </stdio>
    <version_command>echo 'v1.1'</version_command>
    <command><![CDATA[
#if $thebool
    echo 'The bool is true' &&
    echo 'The bool is really true' 1>&2 &&
    echo 'This is a line of text.' > '$out_file1' &&
    cp '$out_file1' '$one' &&
    cp '$out_file1' '$two' &&
    sleep $sleepsecs
#else
    echo 'The bool is not true' &&
    echo 'The bool is very not true' 1>&2 &&
    echo 'This is a different line of text.' > '$out_file1' &&
    sleep $sleepsecs &&
    sh -c 'exit 2'
#end if
#if $failbool
    ; exit 127
#end if
    ]]></command>
    <inputs>
        <param name="sleepsecs" type="integer" value="0" label="Sleep this many seconds"/>
        <param name="thebool" type="boolean" label="The boolean property" />
        <param name="failbool" type="boolean" label="The failure property" checked="false" />
    </inputs>
    <outputs>
        <data name="out_file1" format="txt" />
        <collection name="list_output" type="list" label="A list output">
            <data name="one" format="txt" />
            <data name="two" format="txt" />
        </collection>
    </outputs>
    <tests>
        <test>
            <param name="thebool" value="true" />
            <output name="out_file1" file="simple_line.txt" />
            <assert_stdout>
                <has_line line="The bool is true" />
            </assert_stdout>
            <assert_stderr>
                <has_line line="The bool is really true" />
            </assert_stderr>
            <assert_command_version>
                <has_text text="v1.1" />
            </assert_command_version>
        </test>
        <test expect_exit_code="2">
            <param name="thebool" value="false" />
            <output name="out_file1" file="simple_line_alternative.txt" />
            <assert_command>
                <has_text text="very not" />
            </assert_command>
            <assert_stdout>
                <has_line line="The bool is not true" />
            </assert_stdout>
            <assert_stderr>
                <has_line line="The bool is very not true" />
            </assert_stderr>
        </test>
        <test expect_exit_code="127" expect_failure="true">
            <param name="thebool" value="true" />
            <param name="failbool" value="true" />
        </test>
    </tests>
    <help>
    </help>
</tool>

---

Step 20 FTP

ISSUE 1

ERROR

ERROR! [DEPRECATED]: ansible.builtin.include has been removed. Use include_tasks or import_tasks instead. This feature was removed from ansible-core in a release after 2023-05-16. Please update your playbooks.

The error appears to be in '/home/ubuntu/galaxy/roles/galaxyproject.proftpd/tasks/main.yml': line 7, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Include initial OS-specific tasks
  ^ here

and

The error appears to be in '/home/ubuntu/galaxy/roles/galaxyproject.proftpd/tasks/main.yml': line 58, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

and so on.

SOLUTION

Exchange the include: tasks in /galaxy/roles/galaxyproject.proftpd/tasks/main.yml role with include_tasks:, which are dynamic (the variable will be defined dynamically during the playbook run and is not predefined [ref. to https://serverfault.com/questions/875247/whats-the-difference-between-include-tasks-and-import-tasks])

- name: Include initial OS-specific tasks
  import_tasks: "{{ ansible_os_family | lower}}.yml"

and

- name: Include tasks for TLS configuration
  include_tasks: tls.yml

and

- name: Include tasks for Galaxy authentication
  include_tasks: galaxy_auth.yml

ISSUE 2

ERROR

RUNNING HANDLER [galaxyproject.proftpd : restart proftpd] **********************************************************************************************
fatal: [<my-domain>]: FAILED! => changed=false 
  msg: |-
    Unable to start service proftpd: Job for proftpd.service failed because the control process exited with error code.
    See "systemctl status proftpd.service" and "journalctl -xeu proftpd.service" for details.

SOLUTION

The mod_tls.c module has to be loaded in order to connect via TLS/SSL. This has to do with this task in the galaxyproject.proftpd role ./galaxy/roles/galaxyproject.proftpd/tasks/galaxy_auth.yml:

- name: Uncomment LoadModule directives for Galaxy
  lineinfile:
    name: "{{ proftpd_modules_config_file }}"
    regexp: '^#\s*(LoadModule\s+{{ item }}.*)$'
    backrefs: yes
    line: '\1 # uncommented by Ansible'
    backup: yes
  with_items: "{{ proftpd_galaxy_modules }}"
  notify:
    - reload proftpd

The variable proftpd_galaxy_modules is defined by the settings in ./galaxy/roles/galaxyproject.proftpd/defaults/main.yml (original):

proftpd_galaxy_modules:
  - mod_sql.c
  - mod_sql_passwd.c
  - mod_sql_postgres.c

The required module mod_tls.c is missing and therefore the task will skip to uncomment this module in /etc/proftpd/modules.conf. Adding mod_tls.c to the list proftpd_galaxy_modules will uncomment the module during the playbook run.

Step 21 beacon

ISSUE 1

ERROR

TASK [paprikant.beacon-importer : Install python] ******************************************************************************************
failed: [<my-domain>] (item=python39) => changed=false 
  ansible_loop_var: item
  item: python39
  msg: No package matching 'python39' is available
ok: [<my-domain>] => (item=python3-pip)
ok: [<my-domain>] => (item=python3-setuptools)

SOLUTION

Changed to python3 in ../galaxy/roles/paprikant.beacon-importer/tasks/main.yml:

- name: Install python
  ansible.builtin.package:
    name: "{{ item }}"
    state: present
  loop:
    - python3
    - python3-pip
    - python3-setuptools

ISSUE 2

ERROR

TASK [paprikant.beacon-importer : Install cronie] ******************************************************************************************
fatal: [<my-domain>]: FAILED! => changed=false 
  msg: No package matching 'cronie' is available

SOLUTION

cronie is the standard cron daemon for RedHat-based systems—RHEL, CentOS, Fedora, etc.) but the training platform runs on Ubuntu and cronie is not available.
We have to ensure that cron, the Debian-based cron-daemon, is present.
(Code taken from https://galaxy.ansible.com/ui/standalone/roles/buluma/netdata/documentation)

I also had to comment update_cache: true

in ./roles/paprikant.beacon-importer/tasks/main.yml:

  ansible.builtin.apt:
    name: cron
    #update_cache: true
  become: true
  when: ansible_os_family == "Debian"

ISSUE 3

ERROR

TASK [paprikant.beacon-importer : Install python deps] *************************************************************************************
fatal: [<my-domain>]: FAILED! => changed=false 
  cmd: /usr/bin/pip3.9 list --format=freeze
  msg: '[Errno 2] No such file or directory: b''/usr/bin/pip3.9'''
  rc: 2
  stderr: ''
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

SOLUTION

Switch to /usr/bin/pip3.10 in ./roles/paprikant.beacon-importer/tasks/main.yml:

- name: Install python deps
  ansible.builtin.pip:
    requirements: "{{ script_dir }}/requirements.txt"
    #executable: /usr/bin/pip3.9
    executable: /usr/bin/pip3.10

ISSUE 4

ERROR

TASK [paprikant.beacon-importer : Make sure crond is started and enabled] ******************************************************************
fatal: [<my-domain>]: FAILED! => changed=false 
  msg: 'Could not find the requested service crond: host'

SOLUTION

Switch to cron.service in ./roles/paprikant.beacon-importer/tasks/main.yml:

- name: Make sure cron.service is started and enabled
  ansible.builtin.systemd:
    #name: crond
    name: cron.service
    state: started
    enabled: yes

ISSUE 5

ERROR

Calling <my-domain>/beacon --> 404 Not found

SOLUTION

In ./galaxy/templates/nginx/galaxy.j2 the nginx entry
proxy_pass http://{{ groups['beacon_server'][0] }}:5050;
translates into
proxy_pass http://<my-domain>:5050; which does not work.

Changing the entry to
proxy_pass http://localhost:5050; or

proxy_pass http://127.0.0.1:5050; and

setting a slash after the page entry like

location /beacon/ {
(otherwise the address <my-domain>/beacon will fail or one must set the slash in the address like <my-domain>/beacon/)

[gsaudade99]

When doing the galaxy admin training step 11 on Ubuntu 24.0.4 (nobel) the PPA package is missing as mentioned by @natefoo
To resolve the issue I built the packages locally as a root user:

set -euxo pipefail

ver=1.1.5
curl -fsSL -O https://github.com/natefoo/slurm-drmaa/releases/download/${ver}/slurm-drmaa-${ver}.tar.gz
tar xzf slurm-drmaa-${ver}.tar.gz
cd slurm-drmaa-${ver}

# Build & runtime deps
sudo apt-get update
sudo apt-get install -y \
  build-essential autoconf automake libtool pkg-config \
  libslurm-dev slurm-client libmunge-dev gperf flex

# Where Ubuntu puts headers/libs
SLURM_INC=/usr/include         # contains slurm/*.h (from libslurm-dev)
SLURM_LIB=/usr/lib/x86_64-linux-gnu

# Install to the system multi-arch lib dir so loaders can find it
./configure \
  --with-slurm-inc="${SLURM_INC}" \
  --with-slurm-lib="${SLURM_LIB}" \
  --prefix=/usr \
  --libdir=/usr/lib/x86_64-linux-gnu

make -j"$(nproc)"
sudo make install
sudo ldconfig

# Sanity checks
command -v srun
test -f /usr/lib/x86_64-linux-gnu/libdrmaa.so || \
  ls -l /usr/lib/x86_64-linux-gnu | grep -i drmaa || true

Then we just need to comment the role galaxyproject.repos from the galaxy playbook.

diff --git a/galaxy.yml b/galaxy.yml
index c264091..e43abc1 100644
--- a/galaxy.yml
+++ b/galaxy.yml
@@ -22,7 +22,7 @@
         dest: /libraries/
   roles:
     - galaxyproject.tusd
-    - galaxyproject.repos
+#    - galaxyproject.repos
     - galaxyproject.slurm
     - usegalaxy_eu.apptainer
     - galaxyproject.galaxy
diff --git a/group_vars/galaxyservers.yml b/group_vars/galaxyservers.yml
index ff3585f..cf8fb1d 100644
--- a/group_vars/galaxyservers.yml
+++ b/group_vars/galaxyservers.yml
@@ -184,8 +184,8 @@ slurm_nodes:
   CPUs: 4         # Here you would need to figure out how many cores your machine has. For this training we will use 2 but in real life, look at `htop` or similar.
 slurm_config:
   SlurmdParameters: config_overrides   # Ignore errors if the host actually has cores != 2
-  SelectType: select/cons_res
-  SelectTypeParameters: CR_CPU_Memory   # Allocate individual cores/memory instead of entire node
+  SelectType: select/cons_tres
+  SelectTypeParameters: CR_Core_Memory   # Allocate individual cores/memory instead of entire node

Also, the SelectType and SelectTypeParameters was needed, because select/cons_res doesn't exist as an option in slurm docs + "By default SelectType=select/cons_tres uses CR_Core_Memory"

EDIT:
Also we don't need to this step.

EDIT2:
The library was installed but it was not recognized by under DRMAA_LIBRARY_PATH env variable. For this I added to the /etc/environment to persist across shell sessions.

EDIT3: Rabbitmq

To save the knowledge somewhere.
Because I'm running galaxy locally on a virtual machine, the rabbitmq certificates where not installed.
For this I reused the certs generated for the nginx. However rabbitmq need to run over UID/GID 999:999 instead of root:root.
Thats why I copied the .pem file to /etc/ssl/user/privkey-999:999.pem

  ssl_options:
    verify: verify_peer
    cacertfile: /etc/ssl/certs/{{ openssl_domains[0] }}.crt    # contains server cert + mkcert root
    certfile: /etc/ssl/certs/{{ openssl_domains[0] }}.crt
    keyfile: /etc/ssl/user/privkey-999:999.pem

[gsaudade99]

There is a PR to fix the monitoring issues with deprecated include tasks: usegalaxy-eu/ansible-influxdb#2

[OtimusOne]

Hello, I have also encountered an issue on Step 1 of the training path.
galaxyproject.miniconda version 0.3.1 produces an error regarding Anaconda Terms of Service. I bumped version to 0.3.4 to solve this issue, but then down the line I encountered a different error about not being able to find Mamba version 23.9, tried setting miniconda_version: 25.3 but that did not solve this.