bug: ValueError -- A absorbanceReaderType cannot be loaded into slot 6 [running in simulation mode]

[sgbaird]

Overview

When running in simulation, can't load an absorbance reader (works when running in a protocol run)

Steps to reproduce

import re
import os

SIMULATION_MODE = os.getenv("SIMULATION_MODE", "true").lower() == "true"

if SIMULATION_MODE:
    print("Running in simulation mode.")
    import opentrons.simulate

    protocol = opentrons.simulate.get_protocol_api("2.21")
else:
    print("WARNING: Running on physical hardware.")
    import opentrons.execute

    protocol = opentrons.execute.get_protocol_api("2.21")

print(f"Protocol API {protocol.api_version} loaded.")

# NOTE: 2.16 tested with https://github.com/AccelerationConsortium/ac-training-lab/blob/3988313d80d3d9b1f2a795ceb8701194af00d8e3/src/ac_training_lab/ot-2/_scripts/OT2mqtt.py # noqa: E501
# protocol = opentrons.execute.get_protocol_api("2.16")

pr_mod = protocol.load_module(module_name="absorbanceReaderV1", location="C3")

pr_mod.close_lid()
pr_mod.initialize(
    mode="single", wavelengths=[600]
)  # can add (reference_wavelength=) for normalization (reference wavelenth data will be subtracted from wavelength indicated)
pr_data = pr_mod.read()
print(pr_data)
pr_data[600]["A1"]
pr_data = pr_mod.read(export_filename="raw_absorbance_in")  # CSV file

Current behavior

Exception has occurred: ProtocolCommandFailedError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
Error 4000 GENERAL_ERROR (ProtocolCommandFailedError): PythonException: ValueError: A absorbanceReaderType cannot be loaded into slot 6
  File "/opt/anaconda3/envs/opentrons/lib/python3.12/site-packages/opentrons/protocol_engine/clients/transports.py", line 84, in execute_command
    original_error=error,

                message=f"{error.errorType}: {error.detail}",

            )
  File "/opt/anaconda3/envs/opentrons/lib/python3.12/site-packages/opentrons/protocol_engine/clients/sync_client.py", line 126, in execute_command_without_recovery
    return self._transport.execute_command(create_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/opentrons/lib/python3.12/site-packages/opentrons/protocol_api/core/engine/protocol.py", line 724, in load_module
    cmd.LoadModuleParams(

                model=EngineModuleModel(model),

                location=DeckSlotLocation(slotName=normalized_deck_slot),

            )

        )
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/opentrons/lib/python3.12/site-packages/opentrons/protocol_api/protocol_context.py", line 1035, in load_module
    model=requested_model,

            deck_slot=deck_slot,

            configuration=configuration,

        )
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/opentrons/lib/python3.12/site-packages/opentrons/protocols/api_support/util.py", line 389, in _check_version_wrapper
    return decorated_obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zeqingbao/Documents/GitHub/drug_surfactant/experiments/20250912-Plate Interpretation/absorbance_sim_reproducer.py", line 22, in <module>
    pr_mod = protocol.load_module(module_name="absorbanceReaderV1", location="C3")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/opentrons/lib/python3.12/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/opt/anaconda3/envs/opentrons/lib/python3.12/runpy.py", line 198, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
opentrons.protocol_engine.errors.error_occurrence.ProtocolCommandFailedError: Error 4000 GENERAL_ERROR (ProtocolCommandFailedError): PythonException: ValueError: A absorbanceReaderType cannot be loaded into slot 6

Expected behavior

Allow the initialization at that slot

Operating system

None

System and robot setup or anything else?

Running directly on an OT-Flex via SSH:

python absorbance_sim_reproducer.py

[sgbaird]

Update:

In addition to above, when running on the physical robot (SSH'd into a real Flex, not into a simulator device / on a local machine), simulation works, but real hardware results in OSError: [Errno 16] Device or resource busy and opentrons.hardware_control.thread_manager.ThreadManagerException: Failed to create Managed Object

root@543f62:/data# python3 opentrons_execute_absorbance_mwe.py 
Running in simulation mode.
/data/robot_settings.json not found. Loading defaults
Protocol API 2.21 loaded.
{600: {'A1': 0.0, 'A2': 0.0, 'A3': 0.0, 'A4': 0.0, 'A5': 0.0, 'A6': 0.0, 'A7': 0.0, 'A8': 0.0, 'A9': 0.0, 'A10': 0.0, 'A11': 0.0, 'A12': 0.0, 'B1': 0.0, 'B2': 0.0, 'B3': 0.0, 'B4': 0.0, 'B5': 0.0, 'B6': 0.0, 'B7': 0.0, 'B8': 0.0, 'B9': 0.0, 'B10': 0.0, 'B11': 0.0, 'B12': 0.0, 'C1': 0.0, 'C2': 0.0, 'C3': 0.0, 'C4': 0.0, 'C5': 0.0, 'C6': 0.0, 'C7': 0.0, 'C8': 0.0, 'C9': 0.0, 'C10': 0.0, 'C11': 0.0, 'C12': 0.0, 'D1': 0.0, 'D2': 0.0, 'D3': 0.0, 'D4': 0.0, 'D5': 0.0, 'D6': 0.0, 'D7': 0.0, 'D8': 0.0, 'D9': 0.0, 'D10': 0.0, 'D11': 0.0, 'D12': 0.0, 'E1': 0.0, 'E2': 0.0, 'E3': 0.0, 'E4': 0.0, 'E5': 0.0, 'E6': 0.0, 'E7': 0.0, 'E8': 0.0, 'E9': 0.0, 'E10': 0.0, 'E11': 0.0, 'E12': 0.0, 'F1': 0.0, 'F2': 0.0, 'F3': 0.0, 'F4': 0.0, 'F5': 0.0, 'F6': 0.0, 'F7': 0.0, 'F8': 0.0, 'F9': 0.0, 'F10': 0.0, 'F11': 0.0, 'F12': 0.0, 'G1': 0.0, 'G2': 0.0, 'G3': 0.0, 'G4': 0.0, 'G5': 0.0, 'G6': 0.0, 'G7': 0.0, 'G8': 0.0, 'G9': 0.0, 'G10': 0.0, 'G11': 0.0, 'G12': 0.0, 'H1': 0.0, 'H2': 0.0, 'H3': 0.0, 'H4': 0.0, 'H5': 0.0, 'H6': 0.0, 'H7': 0.0, 'H8': 0.0, 'H9': 0.0, 'H10': 0.0, 'H11': 0.0, 'H12': 0.0}}
root@543f62:/data# SIMULATION_MODE=0 python3 opentrons_execute_absorbance_mwe.py 
WARNING: Running on physical hardware.
/data/robot_settings.json not found. Loading defaults
Exception in Thread Manager build
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 314, in _build_and_start_loop
    managed_obj = loop.run_until_complete(builder(*args, loop=loop, **kwargs))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/ot3api.py", line 396, in build_hardware_controller
    backend = await OT3Controller.build(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 303, in build
    inst = cls(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 332, in __init__
    self._drivers = self._build_system_hardware(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 484, in _build_system_hardware
    gpio = OT3GPIO("hardware_control")
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/gpio/__init__.py", line 50, in __init__
    self._estop_out_line.request(
OSError: [Errno 16] Device or resource busy
Traceback (most recent call last):
  File "/data/opentrons_execute_absorbance_mwe.py", line 15, in <module>
    protocol = opentrons.execute.get_protocol_api("2.21")
  File "/opt/opentrons-robot-server/opentrons/execute.py", line 182, in get_protocol_api
    hardware_controller = _get_global_hardware_controller(robot_type)
  File "/opt/opentrons-robot-server/opentrons/execute.py", line 712, in _get_global_hardware_controller
    _THREAD_MANAGED_HW = ThreadManager(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 289, in __init__
    object.__getattribute__(self, "managed_thread_ready_blocking")()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 294, in managed_thread_ready_blocking
    raise ThreadManagerException("Failed to create Managed Object")
opentrons.hardware_control.thread_manager.ThreadManagerException: Failed to create Managed Object

root@543f62:~# ls
root@543f62:~# ls -a
.  ..  .bash_history  .cache  .python_history  .ssh
root@543f62:~# cd /data/
root@543f62:/data# ls
ODD          feature_flags.json  logs                      pipettes  start_tailscale.sh    tailscale_1.82.0_arm.tgz  tip_lengths
config.json  labware             performance_metrics_data  robot     tailscale_1.82.0_arm  testing_data            user_storage
root@543f62:/data# vi opentrons_execute_absorbance_mwe.py
root@543f62:/data# cat opentrons_execute_absorbance_mwe.py 
import re
import os

SIMULATION_MODE = os.getenv("SIMULATION_MODE", "true").lower() == "true"

if SIMULATION_MODE:
    print("Running in simulation mode.")
    import opentrons.simulate

    protocol = opentrons.simulate.get_protocol_api("2.21")
else:
    print("WARNING: Running on physical hardware.")
    import opentrons.execute

    protocol = opentrons.execute.get_protocol_api("2.21")

print(f"Protocol API {protocol.api_version} loaded.")

# NOTE: 2.16 tested with https://github.com/AccelerationConsortium/ac-training-lab/blob/3988313d80d3d9b1f2a795ceb8701194af00d8e3/src/ac_training_lab/ot-2/_scripts/OT2mqtt.py # noqa: E501
# protocol = opentrons.execute.get_protocol_api("2.16")

pr_mod = protocol.load_module(module_name="absorbanceReaderV1", location="C3")

pr_mod.close_lid()
pr_mod.initialize(
    mode="single", wavelengths=[600]
)  # can add (reference_wavelength=) for normalization (reference wavelenth data will be subtracted from wavelength indicated)
pr_data = pr_mod.read()
print(pr_data)
pr_data[600]["A1"]
pr_data = pr_mod.read(export_filename="raw_absorbance_in")  # CSV file

root@543f62:/data# python opentrons_execute_absorbance_mwe.py 
-sh: python: command not found
root@543f62:/data# python3 opentrons_execute_absorbance_mwe.py 
Running in simulation mode.
/data/robot_settings.json not found. Loading defaults
Protocol API 2.21 loaded.
{600: {'A1': 0.0, 'A2': 0.0, 'A3': 0.0, 'A4': 0.0, 'A5': 0.0, 'A6': 0.0, 'A7': 0.0, 'A8': 0.0, 'A9': 0.0, 'A10': 0.0, 'A11': 0.0, 'A12': 0.0, 'B1': 0.0, 'B2': 0.0, 'B3': 0.0, 'B4': 0.0, 'B5': 0.0, 'B6': 0.0, 'B7': 0.0, 'B8': 0.0, 'B9': 0.0, 'B10': 0.0, 'B11': 0.0, 'B12': 0.0, 'C1': 0.0, 'C2': 0.0, 'C3': 0.0, 'C4': 0.0, 'C5': 0.0, 'C6': 0.0, 'C7': 0.0, 'C8': 0.0, 'C9': 0.0, 'C10': 0.0, 'C11': 0.0, 'C12': 0.0, 'D1': 0.0, 'D2': 0.0, 'D3': 0.0, 'D4': 0.0, 'D5': 0.0, 'D6': 0.0, 'D7': 0.0, 'D8': 0.0, 'D9': 0.0, 'D10': 0.0, 'D11': 0.0, 'D12': 0.0, 'E1': 0.0, 'E2': 0.0, 'E3': 0.0, 'E4': 0.0, 'E5': 0.0, 'E6': 0.0, 'E7': 0.0, 'E8': 0.0, 'E9': 0.0, 'E10': 0.0, 'E11': 0.0, 'E12': 0.0, 'F1': 0.0, 'F2': 0.0, 'F3': 0.0, 'F4': 0.0, 'F5': 0.0, 'F6': 0.0, 'F7': 0.0, 'F8': 0.0, 'F9': 0.0, 'F10': 0.0, 'F11': 0.0, 'F12': 0.0, 'G1': 0.0, 'G2': 0.0, 'G3': 0.0, 'G4': 0.0, 'G5': 0.0, 'G6': 0.0, 'G7': 0.0, 'G8': 0.0, 'G9': 0.0, 'G10': 0.0, 'G11': 0.0, 'G12': 0.0, 'H1': 0.0, 'H2': 0.0, 'H3': 0.0, 'H4': 0.0, 'H5': 0.0, 'H6': 0.0, 'H7': 0.0, 'H8': 0.0, 'H9': 0.0, 'H10': 0.0, 'H11': 0.0, 'H12': 0.0}}

root@543f62:/data# SIMULATION_MODE=0 python3 opentrons_execute_absorbance_mwe.py 
WARNING: Running on physical hardware.
/data/robot_settings.json not found. Loading defaults
Exception in Thread Manager build
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 314, in _build_and_start_loop
    managed_obj = loop.run_until_complete(builder(*args, loop=loop, **kwargs))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/ot3api.py", line 396, in build_hardware_controller
    backend = await OT3Controller.build(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 303, in build
    inst = cls(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 332, in __init__
    self._drivers = self._build_system_hardware(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 484, in _build_system_hardware
    gpio = OT3GPIO("hardware_control")
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/gpio/__init__.py", line 50, in __init__
    self._estop_out_line.request(
OSError: [Errno 16] Device or resource busy
Traceback (most recent call last):
  File "/data/opentrons_execute_absorbance_mwe.py", line 15, in <module>
    protocol = opentrons.execute.get_protocol_api("2.21")
  File "/opt/opentrons-robot-server/opentrons/execute.py", line 182, in get_protocol_api
    hardware_controller = _get_global_hardware_controller(robot_type)
  File "/opt/opentrons-robot-server/opentrons/execute.py", line 712, in _get_global_hardware_controller
    _THREAD_MANAGED_HW = ThreadManager(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 289, in __init__
    object.__getattribute__(self, "managed_thread_ready_blocking")()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 294, in managed_thread_ready_blocking
    raise ThreadManagerException("Failed to create Managed Object")
opentrons.hardware_control.thread_manager.ThreadManagerException: Failed to create Managed Object
root@543f62:/data# SIMULATION_MODE=0 python3 opentrons_execute_absorbance_mwe.py 
WARNING: Running on physical hardware.
/data/robot_settings.json not found. Loading defaults
Exception in Thread Manager build
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 314, in _build_and_start_loop
    managed_obj = loop.run_until_complete(builder(*args, loop=loop, **kwargs))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/ot3api.py", line 396, in build_hardware_controller
    backend = await OT3Controller.build(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 303, in build
    inst = cls(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 332, in __init__
    self._drivers = self._build_system_hardware(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 484, in _build_system_hardware
    gpio = OT3GPIO("hardware_control")
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/gpio/__init__.py", line 50, in __init__
    self._estop_out_line.request(
OSError: [Errno 16] Device or resource busy
Traceback (most recent call last):
  File "/data/opentrons_execute_absorbance_mwe.py", line 15, in <module>
    protocol = opentrons.execute.get_protocol_api("2.21")
  File "/opt/opentrons-robot-server/opentrons/execute.py", line 182, in get_protocol_api
    hardware_controller = _get_global_hardware_controller(robot_type)
  File "/opt/opentrons-robot-server/opentrons/execute.py", line 712, in _get_global_hardware_controller
    _THREAD_MANAGED_HW = ThreadManager(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 289, in __init__
    object.__getattribute__(self, "managed_thread_ready_blocking")()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 294, in managed_thread_ready_blocking
    raise ThreadManagerException("Failed to create Managed Object")
opentrons.hardware_control.thread_manager.ThreadManagerException: Failed to create Managed Object
root@543f62:/data# vi opentrons_execute_absorbance_mwe.py
root@543f62:/data# SIMULATION_MODE=1 python3 opentrons_execute_absorbance_mwe.py 
WARNING: Running on physical hardware.
^CTraceback (most recent call last):
  File "/data/opentrons_execute_absorbance_mwe.py", line 13, in <module>
    import opentrons.execute
  File "/opt/opentrons-robot-server/opentrons/execute.py", line 31, in <module>
    from opentrons import protocol_api, __version__, should_use_ot3
  File "/opt/opentrons-robot-server/opentrons/protocol_api/__init__.py", line 17, in <module>
    from .protocol_context import ProtocolContext
  File "/opt/opentrons-robot-server/opentrons/protocol_api/protocol_context.py", line 26, in <module>
    from opentrons.legacy_commands.helpers import (
  File "/opt/opentrons-robot-server/opentrons/legacy_commands/helpers.py", line 3, in <module>
    from opentrons.protocol_api.labware import Well, Labware
  File "/opt/opentrons-robot-server/opentrons/protocol_api/labware.py", line 50, in <module>
    from opentrons.protocol_engine.types import LiquidTrackingType
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/__init__.py", line 10, in <module>
    from .protocol_engine import ProtocolEngine
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/protocol_engine.py", line 17, in <module>
    from .actions.actions import (
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/actions/__init__.py", line 6, in <module>
    from .action_dispatcher import ActionDispatcher
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/actions/action_dispatcher.py", line 4, in <module>
    from .action_handler import ActionHandler
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/actions/action_handler.py", line 4, in <module>
    from .actions import Action
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/actions/actions.py", line 17, in <module>
    from ..commands import (
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/commands/__init__.py", line 16, in <module>
    from . import absorbance_reader
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/commands/absorbance_reader/__init__.py", line 2, in <module>
    from .close_lid import (
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/commands/absorbance_reader/close_lid.py", line 8, in <module>
    from ..command import AbstractCommandImpl, BaseCommand, BaseCommandCreate, SuccessData
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/commands/command.py", line 25, in <module>
    from opentrons.protocol_engine.state.update_types import StateUpdate
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/state/update_types.py", line 12, in <module>
    from opentrons.protocol_engine.resources import pipette_data_provider
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/resources/__init__.py", line 9, in <module>
    from .deck_data_provider import DeckDataProvider, DeckFixedLabware
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/resources/deck_data_provider.py", line 22, in <module>
    from .labware_data_provider import LabwareDataProvider
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/resources/labware_data_provider.py", line 15, in <module>
    from opentrons.protocols.labware import get_labware_definition
  File "/opt/opentrons-robot-server/opentrons/protocols/labware.py", line 9, in <module>
    import jsonschema  # type: ignore
  File "/opt/opentrons-robot-server/jsonschema/__init__.py", line 13, in <module>
    from jsonschema._format import FormatChecker
  File "/opt/opentrons-robot-server/jsonschema/_format.py", line 281, in <module>
    import idna
  File "/opt/opentrons-robot-server/idna/__init__.py", line 2, in <module>
    from .core import (
  File "/opt/opentrons-robot-server/idna/core.py", line 1, in <module>
    from . import idnadata
KeyboardInterrupt

root@543f62:/data# SIMULATION_MODE=true python3 opentrons_execute_absorbance_mwe.py 
Running in simulation mode.
/data/robot_settings.json not found. Loading defaults
Protocol API 2.16 loaded.
Traceback (most recent call last):
  File "/data/opentrons_execute_absorbance_mwe.py", line 22, in <module>
    pr_mod = protocol.load_module(module_name="absorbanceReaderV1", location="C3")
  File "/opt/opentrons-robot-server/opentrons/protocols/api_support/util.py", line 389, in _check_version_wrapper
    return decorated_obj(*args, **kwargs)
  File "/opt/opentrons-robot-server/opentrons/protocol_api/protocol_context.py", line 901, in load_module
    raise APIVersionError(
opentrons.protocols.api_support.util.APIVersionError: Error 4011 INCORRECT_API_VERSION (APIVersionError): Module of type absorbanceReaderV1 is not available until API version 2.21. You are currently using API version 2.16.
root@543f62:/data# SIMULATION_MODE=0 python3 opentrons_execute_absorbance_mwe.py 
WARNING: Running on physical hardware.
/data/robot_settings.json not found. Loading defaults
Exception in Thread Manager build
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 314, in _build_and_start_loop
    managed_obj = loop.run_until_complete(builder(*args, loop=loop, **kwargs))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/ot3api.py", line 396, in build_hardware_controller
    backend = await OT3Controller.build(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 303, in build
    inst = cls(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 332, in __init__
    self._drivers = self._build_system_hardware(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 484, in _build_system_hardware
    gpio = OT3GPIO("hardware_control")
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/gpio/__init__.py", line 50, in __init__
    self._estop_out_line.request(
OSError: [Errno 16] Device or resource busy
Traceback (most recent call last):
  File "/data/opentrons_execute_absorbance_mwe.py", line 15, in <module>
    protocol = opentrons.execute.get_protocol_api("2.16")
  File "/opt/opentrons-robot-server/opentrons/execute.py", line 182, in get_protocol_api
    hardware_controller = _get_global_hardware_controller(robot_type)
  File "/opt/opentrons-robot-server/opentrons/execute.py", line 712, in _get_global_hardware_controller
    _THREAD_MANAGED_HW = ThreadManager(
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 289, in __init__
    object.__getattribute__(self, "managed_thread_ready_blocking")()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 294, in managed_thread_ready_blocking
    raise ThreadManagerException("Failed to create Managed Object")
opentrons.hardware_control.thread_manager.ThreadManagerException: Failed to create Managed Object
root@543f62:/data# systemctl stop opentrons-robot-server
root@543f62:/data# SIMULATION_MODE=0 python3 opentrons_execute_absorbance_mwe.py 
WARNING: Running on physical hardware.
/data/robot_settings.json not found. Loading defaults
Protocol API 2.16 loaded.
Traceback (most recent call last):
  File "/data/opentrons_execute_absorbance_mwe.py", line 22, in <module>
    pr_mod = protocol.load_module(module_name="absorbanceReaderV1", location="C3")
  File "/opt/opentrons-robot-server/opentrons/protocols/api_support/util.py", line 389, in _check_version_wrapper
    return decorated_obj(*args, **kwargs)
  File "/opt/opentrons-robot-server/opentrons/protocol_api/protocol_context.py", line 901, in load_module
    raise APIVersionError(
opentrons.protocols.api_support.util.APIVersionError: Error 4011 INCORRECT_API_VERSION (APIVersionError): Module of type absorbanceReaderV1 is not available until API version 2.21. You are currently using API version 2.16.
Expected Nodes should have been specified, Setting expected nodes to [<NodeId.gantry_x: 48>, <NodeId.gripper: 32>, <NodeId.gantry_y: 64>, <NodeId.head: 80>, <NodeId.pipette_left: 96>, <NodeId.pipette_right: 112>]
Exception in CAN send
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 238, in _send
    await self._drive.send(
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/driver.py", line 98, in send
    await self._loop.run_in_executor(self._executor, self._bus.send, m)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 821, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Expected Nodes should have been specified, Setting expected nodes to [<NodeId.gantry_x: 48>, <NodeId.gripper: 32>, <NodeId.gantry_y: 64>, <NodeId.head: 80>, <NodeId.pipette_left: 96>, <NodeId.pipette_right: 112>]
Exception in CAN send
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 238, in _send
    await self._drive.send(
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/driver.py", line 98, in send
    await self._loop.run_in_executor(self._executor, self._bus.send, m)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 821, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Exception during post-run finish steps.
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 238, in _send
    await self._drive.send(
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/driver.py", line 98, in send
    await self._loop.run_in_executor(self._executor, self._bus.send, m)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 821, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/protocol_engine.py", line 504, in finish
    await exit_stack.aclose()
  File "/usr/lib/python3.10/contextlib.py", line 656, in aclose
    await self.__aexit__(None, None, None)
  File "/usr/lib/python3.10/contextlib.py", line 714, in __aexit__
    raise exc_details[1]
  File "/usr/lib/python3.10/contextlib.py", line 697, in __aexit__
    cb_suppress = await cb(*exc_details)
  File "/usr/lib/python3.10/contextlib.py", line 608, in _exit_wrapper
    await callback(*args, **kwds)
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/execution/hardware_stopper.py", line 139, in do_stop_and_recover
    await self._hardware_api.stop(home_after=False)
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 149, in wrapper
    return await call_coroutine_threadsafe(loop, attr, *args, **kwargs)
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 52, in call_coroutine_threadsafe
    return await wrapped
  File "/opt/opentrons-robot-server/opentrons/hardware_control/ot3api.py", line 878, in stop
    await self._stop_motors()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/ot3api.py", line 863, in _stop_motors
    await self._backend.halt()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 1383, in halt
    error = await self._messenger.ensure_send(
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 271, in ensure_send
    return await self._ensure_send(node_id, message, timeout, expected_nodes)
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 300, in _ensure_send
    return await listener.send_and_verify_recieved()
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 149, in send_and_verify_recieved
    await self._can_messenger.send(self._node_id, self._message)
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 211, in send
    return await self._send(node_id, message)
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 245, in _send
    raise CanbusCommunicationError(
opentrons_shared_data.errors.exceptions.CanbusCommunicationError: Error 1001 CANBUS_COMMUNICATION_ERROR (CanbusCommunicationError): Exception in canbus.send
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-6' coro=<SubsystemManager._tool_detection_task_main() running at /opt/opentrons-robot-server/opentrons/hardware_control/backends/subsystem_manager.py:372> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Exception ignored in: <coroutine object _await_one_result at 0xffff920fdee0>
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tools/detector.py", line 44, in _await_one_result
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 470, in __anext__
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 474, in read
  File "/usr/lib/python3.10/asyncio/queues.py", line 161, in get
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
Tool reader task failed
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/subsystem_manager.py", line 372, in _tool_detection_task_main
    await self._tool_detection_task_protected()
GeneratorExit
SocketcanBus was not properly shut down
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-35' coro=<TipDetector._main_task() running at /opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py:78> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-36' coro=<TipDetector._main_task() running at /opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py:78> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Exception ignored in: <coroutine object TipDetector._main_task at 0xffff951245f0>
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 78, in _main_task
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 84, in _main_task_protected
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 91, in _debounce_task
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 470, in __anext__
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 474, in read
  File "/usr/lib/python3.10/asyncio/queues.py", line 161, in get
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
Exception ignored in: <coroutine object TipDetector._main_task at 0xffff951247b0>
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 78, in _main_task
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 84, in _main_task_protected
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 91, in _debounce_task
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 470, in __anext__
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 474, in read
  File "/usr/lib/python3.10/asyncio/queues.py", line 161, in get
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-11' coro=<Poller._poll_forever() running at /opt/opentrons-robot-server/opentrons/hardware_control/poller.py:80> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-16' coro=<Poller._poll_forever() running at /opt/opentrons-robot-server/opentrons/hardware_control/poller.py:80> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
root@543f62:/data# # note: this was with 2.16 still
root@543f62:/data# vi opentrons_execute_absorbance_mwe.py
root@543f62:/data# vi opentrons_execute_absorbance_mwe.py
root@543f62:/data# vi opentrons_execute_absorbance_mwe.py
root@543f62:/data# SIMULATION_MODE=0 python3 opentrons_execute_absorbance_mwe.py 
WARNING: Running on physical hardware.
/data/robot_settings.json not found. Loading defaults
Protocol API 2.21 loaded.
{600: {'A1': 0.0, 'A2': 0.0, 'A3': 0.0, 'A4': 0.0, 'A5': 0.0, 'A6': 0.0, 'A7': 0.0, 'A8': 0.0, 'A9': 0.0, 'A10': 0.0, 'A11': 0.0, 'A12': 0.0, 'B1': 0.0, 'B2': 0.0, 'B3': 0.0, 'B4': 0.0, 'B5': 0.0, 'B6': 0.0, 'B7': 0.0, 'B8': 0.0, 'B9': 0.0, 'B10': 0.0, 'B11': 0.0, 'B12': 0.0, 'C1': 0.0, 'C2': 0.0, 'C3': 0.0, 'C4': 0.0, 'C5': 0.0, 'C6': 0.0, 'C7': 0.0, 'C8': 0.0, 'C9': 0.0, 'C10': 0.0, 'C11': 0.0, 'C12': 0.0, 'D1': 0.0, 'D2': 0.0, 'D3': 0.0, 'D4': 0.0, 'D5': 0.0, 'D6': 0.0, 'D7': 0.0, 'D8': 0.0, 'D9': 0.0, 'D10': 0.0, 'D11': 0.0, 'D12': 0.0, 'E1': 0.0, 'E2': 0.0, 'E3': 0.0, 'E4': 0.0, 'E5': 0.0, 'E6': 0.0, 'E7': 0.0, 'E8': 0.0, 'E9': 0.0, 'E10': 0.0, 'E11': 0.0, 'E12': 0.0, 'F1': 0.0, 'F2': 0.0, 'F3': 0.0, 'F4': 0.0, 'F5': 0.0, 'F6': 0.0, 'F7': 0.0, 'F8': 0.0, 'F9': 0.0, 'F10': 0.0, 'F11': 0.0, 'F12': 0.0, 'G1': 0.0, 'G2': 0.0, 'G3': 0.0, 'G4': 0.0, 'G5': 0.0, 'G6': 0.0, 'G7': 0.0, 'G8': 0.0, 'G9': 0.0, 'G10': 0.0, 'G11': 0.0, 'G12': 0.0, 'H1': 0.0, 'H2': 0.0, 'H3': 0.0, 'H4': 0.0, 'H5': 0.0, 'H6': 0.0, 'H7': 0.0, 'H8': 0.0, 'H9': 0.0, 'H10': 0.0, 'H11': 0.0, 'H12': 0.0}}
Polling exception
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/hardware_control/poller.py", line 98, in _poll_once
    await self._reader.read()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/modules/absorbance_reader.py", line 64, in read
    await self.get_plate_presence()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/modules/absorbance_reader.py", line 86, in get_plate_presence
    self.plate_presence = await self._driver.get_plate_presence()
  File "/opt/opentrons-robot-server/opentrons/drivers/absorbance_reader/driver.py", line 81, in get_plate_presence
    return await self._connection.get_plate_presence()
  File "/opt/opentrons-robot-server/opentrons/drivers/absorbance_reader/async_byonoy.py", line 250, in get_plate_presence
    err, presence = await self._loop.run_in_executor(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 821, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
Polling exception
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/hardware_control/poller.py", line 98, in _poll_once
    await self._reader.read()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/modules/heater_shaker.py", line 404, in read
    await self.read_labware_latch()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/modules/heater_shaker.py", line 417, in read_labware_latch
    self.labware_latch = await self._driver.get_labware_latch_status()
  File "/opt/opentrons-robot-server/opentrons/drivers/heater_shaker/driver.py", line 158, in get_labware_latch_status
    response = await self._connection.send_command(
  File "/opt/opentrons-robot-server/opentrons/drivers/asyncio/communication/serial_connection.py", line 435, in send_command
    return await self.send_data(
  File "/opt/opentrons-robot-server/opentrons/drivers/asyncio/communication/serial_connection.py", line 459, in send_data
    return await self._send_data(
  File "/opt/opentrons-robot-server/opentrons/drivers/asyncio/communication/serial_connection.py", line 480, in _send_data
    await self._serial.write(data=data_encode)
  File "/opt/opentrons-robot-server/opentrons/drivers/asyncio/communication/async_serial.py", line 106, in write
    await self._loop.run_in_executor(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 821, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Expected Nodes should have been specified, Setting expected nodes to [<NodeId.gantry_x: 48>, <NodeId.gripper: 32>, <NodeId.gantry_y: 64>, <NodeId.head: 80>, <NodeId.pipette_left: 96>, <NodeId.pipette_right: 112>]
Exception in CAN send
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 238, in _send
    await self._drive.send(
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/driver.py", line 98, in send
    await self._loop.run_in_executor(self._executor, self._bus.send, m)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 821, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Expected Nodes should have been specified, Setting expected nodes to [<NodeId.gantry_x: 48>, <NodeId.gripper: 32>, <NodeId.gantry_y: 64>, <NodeId.head: 80>, <NodeId.pipette_left: 96>, <NodeId.pipette_right: 112>]
Exception in CAN send
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 238, in _send
    await self._drive.send(
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/driver.py", line 98, in send
    await self._loop.run_in_executor(self._executor, self._bus.send, m)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 821, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Exception during post-run finish steps.
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 238, in _send
    await self._drive.send(
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/driver.py", line 98, in send
    await self._loop.run_in_executor(self._executor, self._bus.send, m)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 821, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/protocol_engine.py", line 504, in finish
    await exit_stack.aclose()
  File "/usr/lib/python3.10/contextlib.py", line 656, in aclose
    await self.__aexit__(None, None, None)
  File "/usr/lib/python3.10/contextlib.py", line 714, in __aexit__
    raise exc_details[1]
  File "/usr/lib/python3.10/contextlib.py", line 697, in __aexit__
    cb_suppress = await cb(*exc_details)
  File "/usr/lib/python3.10/contextlib.py", line 608, in _exit_wrapper
    await callback(*args, **kwds)
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/execution/hardware_stopper.py", line 139, in do_stop_and_recover
    await self._hardware_api.stop(home_after=False)
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 149, in wrapper
    return await call_coroutine_threadsafe(loop, attr, *args, **kwargs)
  File "/opt/opentrons-robot-server/opentrons/hardware_control/thread_manager.py", line 52, in call_coroutine_threadsafe
    return await wrapped
  File "/opt/opentrons-robot-server/opentrons/hardware_control/ot3api.py", line 878, in stop
    await self._stop_motors()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/ot3api.py", line 863, in _stop_motors
    await self._backend.halt()
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/ot3controller.py", line 1383, in halt
    error = await self._messenger.ensure_send(
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 271, in ensure_send
    return await self._ensure_send(node_id, message, timeout, expected_nodes)
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 300, in _ensure_send
    return await listener.send_and_verify_recieved()
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 149, in send_and_verify_recieved
    await self._can_messenger.send(self._node_id, self._message)
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 211, in send
    return await self._send(node_id, message)
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 245, in _send
    raise CanbusCommunicationError(
opentrons_shared_data.errors.exceptions.CanbusCommunicationError: Error 1001 CANBUS_COMMUNICATION_ERROR (CanbusCommunicationError): Exception in canbus.send
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-6' coro=<SubsystemManager._tool_detection_task_main() running at /opt/opentrons-robot-server/opentrons/hardware_control/backends/subsystem_manager.py:372> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-16' coro=<Poller._poll_forever() running at /opt/opentrons-robot-server/opentrons/hardware_control/poller.py:80> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Exception ignored in: <coroutine object _await_one_result at 0xffffa9329ee0>
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tools/detector.py", line 44, in _await_one_result
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 470, in __anext__
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 474, in read
  File "/usr/lib/python3.10/asyncio/queues.py", line 161, in get
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
Tool reader task failed
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/subsystem_manager.py", line 372, in _tool_detection_task_main
    await self._tool_detection_task_protected()
GeneratorExit
SocketcanBus was not properly shut down
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-35' coro=<TipDetector._main_task() running at /opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py:78> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-36' coro=<TipDetector._main_task() running at /opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py:78> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Exception ignored in: <coroutine object TipDetector._main_task at 0xffffa81b23b0>
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 78, in _main_task
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 84, in _main_task_protected
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 91, in _debounce_task
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 470, in __anext__
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 474, in read
  File "/usr/lib/python3.10/asyncio/queues.py", line 161, in get
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
Exception ignored in: <coroutine object TipDetector._main_task at 0xffffa81b22d0>
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 78, in _main_task
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 84, in _main_task_protected
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 91, in _debounce_task
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 470, in __anext__
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 474, in read
  File "/usr/lib/python3.10/asyncio/queues.py", line 161, in get
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-11' coro=<Poller._poll_forever() running at /opt/opentrons-robot-server/opentrons/hardware_control/poller.py:80> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
root@543f62:/data# ls
ODD                 logs                                 robot                     testing_data
config.json         opentrons_execute_absorbance_mwe.py  start_tailscale.sh        tip_lengths
feature_flags.json  performance_metrics_data             tailscale_1.82.0_arm      user_storage
labware             pipettes                             tailscale_1.82.0_arm.tgz
root@543f62:/data# vi absorbance_protocol_mwe.py
root@543f62:/data# cat absorbance_protocol_mwe.py 
from opentrons import protocol_api

metadata = {
    "description": "Written in 2025.05",
    "author": "Zeqing Bao and Yunhee Hwang",
}

requirements = {"robotType": "Flex", "apiLevel": "2.21"}


def run(protocol: protocol_api.ProtocolContext):

    pr_mod = protocol.load_module(module_name="absorbanceReaderV1", location="C3")

    pr_mod.close_lid()
    pr_mod.initialize(
        mode="single", wavelengths=[600]
    )  # can add (reference_wavelength=) for normalization (reference wavelenth data will be subtracted from wavelength indicated)
    pr_data = pr_mod.read()
    print(pr_data)
    pr_data[600]["A1"]
    pr_data = pr_mod.read(export_filename="raw_absorbance_in")  # CSV file

root@543f62:/data# opentrons_execute absorbance_protocol_mwe.py 
/data/robot_settings.json not found. Loading defaults
no pipette found
no pipette found
Execution of 1d5d0c0c-37e7-4ef0-a6be-db77ce4efbc5 failed
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/execution/command_executor.py", line 148, in execute
    result = await command_impl.execute(
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/commands/absorbance_reader/close_lid.py", line 112, in execute
    await self._labware_movement.move_labware_with_gripper(
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/execution/labware_movement.py", line 158, in move_labware_with_gripper
    raise GripperNotAttachedError(
opentrons.protocol_engine.errors.exceptions.GripperNotAttachedError: Error 3011 GRIPPER_NOT_PRESENT (GripperNotAttachedError): No gripper found for performing labware movements.
Exception raised by protocol
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/protocols/execution/execute_python.py", line 160, in exec_run
    exec("run(__context)", new_globs)
  File "<string>", line 1, in <module>
  File "absorbance_protocol_mwe.py", line 15, in run
    pr_mod.close_lid()
  File "/opt/opentrons-robot-server/opentrons/protocols/api_support/util.py", line 389, in _check_version_wrapper
    return decorated_obj(*args, **kwargs)
  File "/opt/opentrons-robot-server/opentrons/protocol_api/module_contexts.py", line 1022, in close_lid
    self._core.close_lid()
  File "/opt/opentrons-robot-server/opentrons/protocol_api/core/engine/module_core.py", line 676, in close_lid
    self._engine_client.execute_command(
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/clients/sync_client.py", line 60, in execute_command
    self._transport.execute_command_wait_for_recovery(create_request)
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/clients/transports.py", line 157, in execute_command_wait_for_recovery
    ).result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/clients/transports.py", line 137, in run_in_pe_thread
    raise ProtocolCommandFailedError(
opentrons.protocol_engine.errors.error_occurrence.ProtocolCommandFailedError: Error 4000 GENERAL_ERROR (ProtocolCommandFailedError): GripperNotAttachedError: No gripper found for performing labware movements.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/protocol_runner/task_queue.py", line 84, in _run
    await self._run_func()
  File "/opt/opentrons-robot-server/opentrons/protocol_runner/task_queue.py", line 61, in _do_run
    await func(*args, **kwargs)
  File "/opt/opentrons-robot-server/opentrons/protocol_runner/protocol_runner.py", line 254, in run_func
    await self._protocol_executor.execute(
  File "/opt/opentrons-robot-server/opentrons/protocol_runner/python_protocol_wrappers.py", line 159, in execute
    await to_thread.run_sync(
  File "/opt/opentrons-robot-server/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/opt/opentrons-robot-server/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/opt/opentrons-robot-server/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/opt/opentrons-robot-server/opentrons/protocols/execution/execute.py", line 39, in run_protocol
    exec_run(
  File "/opt/opentrons-robot-server/opentrons/protocols/execution/execute_python.py", line 169, in exec_run
    _raise_pretty_protocol_error(exception=e, filename=filename)
  File "/opt/opentrons-robot-server/opentrons/protocols/execution/execute_python.py", line 71, in _raise_pretty_protocol_error
    raise ExceptionInProtocolError(
opentrons.protocols.execution.errors.ExceptionInProtocolError: ProtocolCommandFailedError [line 15]: Error 4000 GENERAL_ERROR (ProtocolCommandFailedError): GripperNotAttachedError: No gripper found for performing labware movements.
Message did not receive ack for message index UInt32Field(value=1) Missing node(s) [<NodeId.pipette_left: 96>]
received error ErrorCode.timeout trying to disable NodeId.pipette_left 
Expected Nodes should have been specified, Setting expected nodes to [<NodeId.gantry_y: 64>, <NodeId.gripper: 32>, <NodeId.gantry_x: 48>, <NodeId.head: 80>, <NodeId.pipette_left: 96>, <NodeId.pipette_right: 112>]
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Expected Nodes should have been specified, Setting expected nodes to [<NodeId.gantry_y: 64>, <NodeId.gripper: 32>, <NodeId.gantry_x: 48>, <NodeId.head: 80>, <NodeId.pipette_left: 96>, <NodeId.pipette_right: 112>]
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
ProtocolCommandFailedError [line 15]: Error 4000 GENERAL_ERROR (ProtocolCommandFailedError): GripperNotAttachedError: No gripper found for performing labware movements.
root@543f62:/data# opentrons_execute absorbance_protocol_mwe.py
/data/robot_settings.json not found. Loading defaults
no pipette found
no pipette found
Execution of 306dae70-095e-4c9c-90fd-1036fdfbadc1 failed
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/execution/command_executor.py", line 148, in execute
    result = await command_impl.execute(
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/commands/absorbance_reader/close_lid.py", line 112, in execute
    await self._labware_movement.move_labware_with_gripper(
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/execution/labware_movement.py", line 158, in move_labware_with_gripper
    raise GripperNotAttachedError(
opentrons.protocol_engine.errors.exceptions.GripperNotAttachedError: Error 3011 GRIPPER_NOT_PRESENT (GripperNotAttachedError): No gripper found for performing labware movements.
Exception raised by protocol
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/protocols/execution/execute_python.py", line 160, in exec_run
    exec("run(__context)", new_globs)
  File "<string>", line 1, in <module>
  File "absorbance_protocol_mwe.py", line 15, in run
    pr_mod.close_lid()
  File "/opt/opentrons-robot-server/opentrons/protocols/api_support/util.py", line 389, in _check_version_wrapper
    return decorated_obj(*args, **kwargs)
  File "/opt/opentrons-robot-server/opentrons/protocol_api/module_contexts.py", line 1022, in close_lid
    self._core.close_lid()
  File "/opt/opentrons-robot-server/opentrons/protocol_api/core/engine/module_core.py", line 676, in close_lid
    self._engine_client.execute_command(
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/clients/sync_client.py", line 60, in execute_command
    self._transport.execute_command_wait_for_recovery(create_request)
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/clients/transports.py", line 157, in execute_command_wait_for_recovery
    ).result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/opentrons-robot-server/opentrons/protocol_engine/clients/transports.py", line 137, in run_in_pe_thread
    raise ProtocolCommandFailedError(
opentrons.protocol_engine.errors.error_occurrence.ProtocolCommandFailedError: Error 4000 GENERAL_ERROR (ProtocolCommandFailedError): GripperNotAttachedError: No gripper found for performing labware movements.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/protocol_runner/task_queue.py", line 84, in _run
    await self._run_func()
  File "/opt/opentrons-robot-server/opentrons/protocol_runner/task_queue.py", line 61, in _do_run
    await func(*args, **kwargs)
  File "/opt/opentrons-robot-server/opentrons/protocol_runner/protocol_runner.py", line 254, in run_func
    await self._protocol_executor.execute(
  File "/opt/opentrons-robot-server/opentrons/protocol_runner/python_protocol_wrappers.py", line 159, in execute
    await to_thread.run_sync(
  File "/opt/opentrons-robot-server/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/opt/opentrons-robot-server/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/opt/opentrons-robot-server/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/opt/opentrons-robot-server/opentrons/protocols/execution/execute.py", line 39, in run_protocol
    exec_run(
  File "/opt/opentrons-robot-server/opentrons/protocols/execution/execute_python.py", line 169, in exec_run
    _raise_pretty_protocol_error(exception=e, filename=filename)
  File "/opt/opentrons-robot-server/opentrons/protocols/execution/execute_python.py", line 71, in _raise_pretty_protocol_error
    raise ExceptionInProtocolError(
opentrons.protocols.execution.errors.ExceptionInProtocolError: ProtocolCommandFailedError [line 15]: Error 4000 GENERAL_ERROR (ProtocolCommandFailedError): GripperNotAttachedError: No gripper found for performing labware movements.
Message did not receive ack for message index UInt32Field(value=1) Missing node(s) [<NodeId.pipette_left: 96>]
received error ErrorCode.timeout trying to disable NodeId.pipette_left 
Expected Nodes should have been specified, Setting expected nodes to [<NodeId.gantry_x: 48>, <NodeId.gripper: 32>, <NodeId.gantry_y: 64>, <NodeId.head: 80>, <NodeId.pipette_left: 96>, <NodeId.pipette_right: 112>]
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Expected Nodes should have been specified, Setting expected nodes to [<NodeId.gantry_x: 48>, <NodeId.gripper: 32>, <NodeId.gantry_y: 64>, <NodeId.head: 80>, <NodeId.pipette_left: 96>, <NodeId.pipette_right: 112>]
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
ProtocolCommandFailedError [line 15]: Error 4000 GENERAL_ERROR (ProtocolCommandFailedError): GripperNotAttachedError: No gripper found for performing labware movements.

[sgbaird]

Further debugging where we're unable to even use opentrons_execute in a simple fashion, just loading some tip racks.

root@543f62:/data# cat absorbance_protocol_mwe.py
from opentrons import protocol_api

metadata = {
    "description": "Written in 2025.05",
    "author": "Zeqing Bao and Yunhee Hwang",
}

requirements = {"robotType": "Flex", "apiLevel": "2.21"}


def run(protocol: protocol_api.ProtocolContext):

    tip1000_1 = protocol.load_labware(load_name="opentrons_flex_96_filtertiprack_1000ul", location="B1")
    tip1000_2 = protocol.load_labware(load_name="opentrons_flex_96_filtertiprack_1000ul", location="A1")

    # pr_mod = protocol.load_module(module_name="absorbanceReaderV1", location="C3")

    # pr_mod.close_lid()
    #  pr_mod.initialize(
    #     mode="single", wavelengths=[600]
    # )  # can add (reference_wavelength=) for normalization (reference wavelenth data will be subtracted from wavelength indicated)
    # pr_data = pr_mod.read()
    # print(pr_data)
    # pr_data[600]["A1"]
    # pr_data = pr_mod.read(export_filename="raw_absorbance_in")  # CSV file

root@543f62:/data# opentrons_execute absorbance_protocol_mwe.py
/data/robot_settings.json not found. Loading defaults
no pipette found
no pipette found
Message did not receive ack for message index UInt32Field(value=1) Missing node(s) [<NodeId.pipette_left: 96>]
received error ErrorCode.timeout trying to disable NodeId.pipette_left 
Expected Nodes should have been specified, Setting expected nodes to [<NodeId.gantry_x: 48>, <NodeId.gripper: 32>, <NodeId.gantry_y: 64>, <NodeId.head: 80>, <NodeId.pipette_left: 96>, <NodeId.pipette_right: 112>]
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Expected Nodes should have been specified, Setting expected nodes to [<NodeId.gantry_x: 48>, <NodeId.gripper: 32>, <NodeId.gantry_y: 64>, <NodeId.head: 80>, <NodeId.pipette_left: 96>, <NodeId.pipette_right: 112>]
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
Received error message ErrorMessage(payload=ErrorMessagePayload(message_index=UInt32Field(value=0), severity=ErrorSeverityField(value=warning), error_code=ErrorCodeField(value=stop_requested)), payload_type=<class 'opentrons_hardware.firmware_bindings.messages.payloads.ErrorMessagePayload'>, message_id=<MessageId.error_message: 2>)
SocketcanBus was not properly shut down
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-6' coro=<SubsystemManager._tool_detection_task_main() running at /opt/opentrons-robot-server/opentrons/hardware_control/backends/subsystem_manager.py:372> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Exception ignored in: <coroutine object _await_one_result at 0xffff80a00190>
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tools/detector.py", line 44, in _await_one_result
    async for response, _ in callback:
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 470, in __anext__
    return await self.read()
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 474, in read
    return await self._queue.get()
  File "/usr/lib/python3.10/asyncio/queues.py", line 161, in get
    getter.cancel()  # Just in case getter is not done yet.
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Tool reader task failed
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons/hardware_control/backends/subsystem_manager.py", line 372, in _tool_detection_task_main
    await self._tool_detection_task_protected()
GeneratorExit
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-12' coro=<Poller._poll_forever() running at /opt/opentrons-robot-server/opentrons/hardware_control/poller.py:80> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-18' coro=<Poller._poll_forever() running at /opt/opentrons-robot-server/opentrons/hardware_control/poller.py:80> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-112' coro=<TipDetector._main_task() running at /opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py:78> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Exception ignored in: <coroutine object TipDetector._main_task_protected at 0xffff84bc3300>
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 84, in _main_task_protected
    await self._debounce_task(wc)
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 91, in _debounce_task
    async for response, _ in callback:
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 470, in __anext__
    return await self.read()
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 474, in read
    return await self._queue.get()
  File "/usr/lib/python3.10/asyncio/queues.py", line 161, in get
    getter.cancel()  # Just in case getter is not done yet.
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Caught exception: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-111' coro=<TipDetector._main_task() running at /opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py:78> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
Exception ignored in: <coroutine object TipDetector._main_task at 0xffff84bc2b90>
Traceback (most recent call last):
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 78, in _main_task
    await self._main_task_protected()
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 84, in _main_task_protected
    await self._debounce_task(wc)
  File "/opt/opentrons-robot-server/opentrons_hardware/hardware_control/tip_presence/detector.py", line 91, in _debounce_task
    async for response, _ in callback:
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 470, in __anext__
    return await self.read()
  File "/opt/opentrons-robot-server/opentrons_hardware/drivers/can_bus/can_messenger.py", line 474, in read
    return await self._queue.get()
  File "/usr/lib/python3.10/asyncio/queues.py", line 161, in get
    getter.cancel()  # Just in case getter is not done yet.
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

[sgbaird]

Clarified to try:

Please enter the command !systemctl stop opentrons-robot-server as shown in the picture below.

While we tried !systemctl stop opentrons-robot-server a variety of times, we can make sure to keep in the logs for reproducing and double check.

Additionally, it looks like you haven't defined the robot at the Flex in your protocol, therefore it will use OT-2 API information by default. Please add the following: requirements = { "robotType": "OT-3", "apiLevel": "2.23", } to your protocol.

Not sure we tried this. Seems very relevant.

Please also make sure that your deck configuration in the app is modified so that the plate reader is located in slot C3: https://docs.opentrons.com/flex/touchscreen/deck-config/.

Not sure if this was the case.

@shmaiii to debug