The TextGeneration ModuleBases raises exception when loading weights from a path

[VassilisVassiliadis]

Describe the bug

When using artifact_path to load weights from a path with one of the text_generation ModuleBases the caikit runtime emits the following error:

{"channel": "LLOAD", "exception": null, "level": "warning", "log_code": "<COR98539580W>", "message": "DEPRECATION: Loading f9181353-4ccf-4572-bd1e-f12bcda26792 failed with ModuleConfig. Using model_path. expected str, bytes or os.PathLike object, not ModuleConfig", "num_indent": 0, "thread_id": 140699291752000, "timestamp": "2023-10-27T11:21:44.284465"}

The source of the exception is os.path.join in

caikit-nlp/caikit_nlp/modules/text_generation/text_generation_local.py

Line 471 in 4f42230

base_model_path = os.path.join(model_path, base_model_path)

model_path is a ModuleConfig object instead of a str.

Platform

Please provide details about the environment you are using, including the following:

• Interpreter version: 3.11.5
• Library version: 0.22.0

Sample Code

Create the files /tmp/test/runtime_config.yml and /tmp/test/local_models_dir/text_generation/config.yml like so:

File /tmp/test/runtime_config.yml

runtime:
  local_models_dir: /tmp/test/local_models_dir/
  library: caikit_nlp

File /tmp/test/local_models_dir/text_generation/config.yml

module_id: f9181353-4ccf-4572-bd1e-f12bcda26792
artifact_path: /tmp/test/google-flan-t5-base

Download the weights of https://huggingface.co/google/flan-t5-base under /tmp/test/google-flan-t5-base

Then run:

export CONFIG_FILES=/tmp/test/runtime_config.yml
python3 -m caikit.runtime

Expected behavior

The TextGeneration ModuleBase in text_generation_local.py should successfully load the weights.

Observed behavior

The LocalModelManager is unable to trigger the loading of the weights using the TextGeneration ModuleBase. The terminal shows the following message:

{"channel": "LLOAD", "exception": null, "level": "warning", "log_code": "<COR98539580W>", "message": "DEPRECATION: Loading f9181353-4ccf-4572-bd1e-f12bcda26792 failed with ModuleConfig. Using model_path. expected str, bytes or os.PathLike object, not ModuleConfig", "num_indent": 0, "thread_id": 140699291752000, "timestamp": "2023-10-27T11:21:44.284465"}

Additional context

I think the fix is to expect a Union[str, ModuleConfig] for the model_path argument of the ModuleBase.load() methods.

[VassilisVassiliadis]

I put together a possible fix for this in VassilisVassiliadis#1. Would you like me to move the PR here ?

[gabe-l-hart]

Hi! The message you're seeing is a warning by design. This is a compatibility feature in the caikit.load functionality that allows it to fall back to loading from a str. Are you seeing the model fail to load after this message?

You're absolutely correct that the "right" way to do this will be to make load() in the individual module smart enough to take a ModuleConfig, but if the model is failing to load entirely, then there's likely a different bug that we need to track.

[VassilisVassiliadis]

Thank you Gabe, I was looking at the logs and started chasing down that warning message without first checking if the model ended up loading first!

I can confirm that caikit-nlp did load the model and I was able to send inference requests to it. Is there a way I can query the runtime to get back a list of loaded models? Or maybe look for something in the logs of the caikit runtime to figure out which model was loaded?

For example, I searched for text_generation (the name of the Caikit Model shim directory in ${local-models-dir}) in the logs of the caikit runtime and found these lines:

{"channel": "SERVR-GRPC", "exception": null, "level": "info", "log_code": "<RUN20247875I>", "message": "Enabling gRPC inference service", "num_indent": 0, "thread_id": 140473210508288, "timestamp": "2023-10-31T09:59:36.129317"}
{"channel": "MODEL-MANAGR", "exception": null, "level": "info", "log_code": "<RUN44739400I>", "message": "Loading local models into Caikit Runtime...", "num_indent": 0, "thread_id": 140473210508288, "timestamp": "2023-10-31T09:59:36.129468"}
{"channel": "MODEL-LOADER", "exception": null, "level": "info", "log_code": "<RUN89711114I>", "message": "Loading model 'text_generation'", "num_indent": 0, "thread_id": 140472049038912, "timestamp": "2023-10-31T09:59:36.130008"}
{"channel": "MODEL-SIZER", "exception": null, "level": "info", "log_code": "<RUN62161564I>", "message": "No configured model size multiplier found for model type 'standalone-model' for model 'text_generation'. Using default multiplier '10.000000'", "num_indent": 0, "thread_id": 140473210508288, "timestamp": "2023-10-31T09:59:36.130873"}
{"channel": "LLOAD", "exception": null, "level": "warning", "log_code": "<COR98539580W>", "message": "DEPRECATION: Loading f9181353-4ccf-4572-bd1e-f12bcda26792 failed with ModuleConfig. Using model_path. expected str, bytes or os.PathLike object, not ModuleConfig", "num_indent": 0, "thread_id": 140472049038912, "timestamp": "2023-10-31T09:59:36.131808"}

Is it safe to assume that if the Caikit Runtime attempts to load a model it will either:

1. print nothing if it loads the model successfully
2. print a message containing a a non-null exception field if the model loader runs into an unrecoverable error while loading the model

[gabe-l-hart]

Is there a way I can query the runtime to get back a list of loaded models?

Currently, no, there is not, but there is a feature request to add this and we're actively investigating what it would take

Or maybe look for something in the logs of the caikit runtime to figure out which model was loaded?

You can look for the log code RUN89711114I to indicate that a model has started loading