package request: Prefect

[Neil-YL]

package link

https://pypi.org/project/prefect/

minimum version

3.3.4

Extra notes

Hello,

We have been developing a series of autonomous experimental systems, with Opentrons products (OT-2 and Flex) serving as the major liquid handling robots. Recently, we have been working to integrate Prefect as the orchestrator of our workflow. However, we have encountered difficulties installing Prefect natively on OT-2.

Having Prefect on the robot would greatly enhance our ability and reduce cost to scale and replicate similar system across our organization. Otherwise, we may need to add extra standalone device and labour to handle orchestration for the Opentrons robots.

We’ve documented what we’ve tried here, and we would really appreciate any suggestions or support you could offer in making this possible.

Things to check while adding the request

• Installing the package on the OT-2 fails while installing this package
• Installing the package on the OT-2 fails at some other time, like while installing a dependency

[sgbaird]

The recommendation we received is to follow the README. Looks a bit complex, but worth a shot

[sgbaird]

Copying from email thread from @Neil-YL and myself to Opentrons support:

---

I finally got a chance to have a x86_64 Windows PC (running with WSL) to try the build tools in https://github.com/Opentrons/opentrons-python-packages/tree/main/tools. I successfully built the Docker container using the provided Dockerfile, and then attempted to build a new package, prefect, under the packages/ directory.

I created a build.py file with all necessary dependencies for the patched pendulum and the target Prefect version. However, when I run the build command, I consistently receive the following warning:

<frozen runpy>:128: RuntimeWarning: 'builder.package_build.orchestrate' found in sys.modules after import of package 'builder.package_build', but prior to execution of 'builder.package_build.orchestrate'; this may result in unpredictable behaviour.

And unfortunately, the build process does not proceed beyond this point.

I’ve tried several workarounds I found online, but none of them resolved the issue. I’m relatively new to Docker, though I’ve previously managed to cross-compile a Prefect wheel on my M2 Mac using a Docker environment that mimicked the OT-2 system. I assumed the build tools approach would be similar, but I seem to be stuck.

It seems like the issue might be related to how builder/package_build/orchestrate.py is being called, and I’m not confident enough with the internals to modify it safely. If you have any suggestions or a more detailed guidance on how to use the build tools, I’d really appreciate your help.

[sfoster1]

Let me give it a try. It might be another kind of failure - in particular, this is a pretty old repo targeting an even older toolchain and I don't have a ton of confidence it handles PEP-517 build processes (with pyproject.toml and controlled build dependencies) correctly.

[sfoster1]

Do you know which dependency has the conflict? It looks like prefect has a ton of dependencies, so we'd need to add a lot of stuff to the build - instead, anything that is pure-python won't get much benefit from being built here, so if we can identify a couple specific deps that have native dependencies with glibc conflicts then maybe we can just do those.

Overall, I'm not terribly hopeful, unfortunately - often, a glibc minimum requirement is because it really does need that version of glibc and can't really be worked around.

[Neil-YL]

Thanks.

At least one known conflicting package is pendulum==3.0.0. Additionaly, its latest release has a bug when installing on 32-bit systems, which was addressed in this commit: https://github.com/python-pendulum/pendulum.git@b84b97625cdea00f8ab150b8b35aa5ccaaf36948. Pendulum is the package requiring a newer version of glibc, which appears to be the root of the compatibility issue.

Personally, I’m not very confident that the build-tools will resolve the problem as well, since I believe it is not an issue with the building environment but the issue of the dependencies's dependencies.

[sgbaird]

Surprisingly and somewhat amazingly, we installed it and tested some of the basic functionality on an OT2 simulator (i.e., a separate Raspberry Pi 3B+ with the firmware installed). Copilot agent was persistent.

AccelerationConsortium/ac-dev-lab#109 (comment)

Unfortunately, we still have to go back and figure out a minimal set of commands to reproduce 😅