Global state to app.state to enable multiple APIs in a single process

[eimrek]

Fixes #2307

This PR moves the global CONFIG singleton and ENTRY_COLLECTIONS to the Fastapi app.state. This required quite a large refactor, but the default functionality should be retained (so it's fully backwards compatible).

Now, there is the create_app function that allows to create multiple apps with different configs. So one could use something like

from fastapi import FastAPI

from optimade.server.config import ServerConfig
from optimade.server.create_app import create_app

app = FastAPI()


@app.get("/")
def read_main():
   return {"message": "Hello World from main app"}


base_url = "http://127.0.0.1:8000"

conf1 = ServerConfig()
conf1.base_url = f"{base_url}/app1"
app1 = create_app(conf1)
app.mount("/app1", app1)

conf2 = ServerConfig()
conf2.base_url = f"{base_url}/app2"
app2 = create_app(conf2)
app.mount("/app2", app2)

conf3 = ServerConfig()
conf3.base_url = f"{base_url}/ind"
app3 = create_app(conf3, index=True)
app.mount("/ind", app3)

which starts two different optimade APIs and an index metadb API.

[codecov]

Codecov Report

❌ Patch coverage is 85.41667% with 63 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.66%. Comparing base (6335e46) to head (9770f6b).
⚠️ Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
optimade/server/routers/landing.py	13.33%	26 Missing ⚠️
optimade/server/create_app.py	90.29%	13 Missing ⚠️
optimade/server/entry_collections/mongo.py	75.60%	10 Missing ⚠️
optimade/server/logger.py	79.48%	8 Missing ⚠️
optimade/server/entry_collections/elasticsearch.py	91.30%	2 Missing ⚠️
optimade/server/mappers/entries.py	97.10%	2 Missing ⚠️
optimade/server/routers/utils.py	94.11%	1 Missing ⚠️
optimade/utils.py	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2313      +/-   ##
==========================================
- Coverage   90.66%   90.66%   -0.01%     
==========================================
  Files          75       75              
  Lines        4981     5022      +41     
==========================================
+ Hits         4516     4553      +37     
- Misses        465      469       +4     

Flag	Coverage Δ
project	90.66% <85.41%> (-0.01%)	⬇️
validator	90.66% <85.41%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[eimrek]

@ml-evs i think this is ready for review. The only thing failing is the build of optimade-validator-action, which should not really be related to this PR. Initially it failed due to missing pyyaml, now it's not building because of something else. I wonder how it build in the previous commits to this repo. any ideas?

[ml-evs]

You should be able to ignore the validator action errors, I'll make an effort to merge some thing on that end today and we'll see if this turns green.

[ml-evs]

(Just rebased with the changes from #2308)

[ml-evs]

@ml-evs i think this is ready for review. The only thing failing is the build of optimade-validator-action, which should not really be related to this PR. Initially it failed due to missing pyyaml, now it's not building because of something else. I wonder how it build in the previous commits to this repo. any ideas?

Somehow the validator now touches the server config when called via the validator action, whereas before it didn't (and thus didnt need yaml). Might have to be careful when unpicking that one...

[ml-evs]

Hi @eimrek, got about halfway through a review.

Some general comments:

• The mapper stuff is definitely a breaking change (at least for my usage in odbx etc.) and I'd want to double-check the effect on performance before merging (i.e., needing to create a mapper instance every request just to parse the query params). If its minimal (likely it is) then I'm happy to include this as it does simplify things.
• This PR removes the functionality for CONFIG.VALIDATE_API_RESPONSE, which was added to allow non-compliant databases to still host their best guess at OPTIMADE data, for example NOMAD including a "D" in chemical formulae. I'm not sure if its possible to override this in a way that works with the new config (e.g., if there's a FastAPI-native setting we can use to disable validation on a per API basis) but I think this is still required.
• Can you also write a little docs page about this under the "Deployment" section for running multiple APIs that use different dbs/collections?

[eimrek]

@ml-evs I think now it's in a better shape. Would be great to have another review.

The idea is that the main way of using is still the same - config is read either from the ENV variables or the JSON file. However, now, with the create_app function, one can, if they want to, override config for any specific app, allowing to set different base urls and optimade db that at crucial for serving multiple APIs.

I also re-did logging such that you can specify a tag that allows one to differentiate logs from different apps.

Finally, I also updates a bit the docs and added some description on how to run multiple apps.

[eimrek]

@ml-evs Ok, so now all of Materials Cloud OPTIMADE APIs are running on this PR (see https://github.com/materialscloud-org/optimade-server/tree/main/multi-optimade-server) and everything is much more resource-efficient :). Would be great to have this merged in the near future.

[ml-evs]

Hi @eimrek, thanks again for this! You running in production is a better review than I can give, so I'm happy to merge and call this a release candidate.

I'll try it out with my servers and note any config changes I have to make, if any.

[eimrek]

Awesome! Thanks a lot @ml-evs!

[ml-evs]

There's a GH release for v1.4.0-rc.1 but our PyPI release pipeline isn't set up to handle pre-releases, and I don't have time to faff around with it atm. Will test this out next week and hopefully release a proper 1.4.0 soon.