agent_examples.py using UI-TARS and OmniParser

[cy3021561]

Hi, I was trying to use this script to test different models, but only OpenAI CUA and Anthropic CUA worked successfully.

When I tested UI-TARS (deployed via TGI on HF, following this guide
) and Omniparser + [other thinking model], both failed.

For UI-TARS as a computer-use model, it failed right after the first turn:
raise APIError( litellm.exceptions.APIError: litellm.APIError: HuggingfaceException - Failed to deserialize the JSON body into the target type: messages[2]: data did not match any variant of untagged enum MessageBody at line 1 column 1985 LiteLLM Retried: 3 times

For the composed model, it seems like the message handling between components didn’t work as expected:

raise BadRequestError( litellm.exceptions.BadRequestError: litellm.BadRequestError: OpenAIException - Missing required parameter: 'tools[0].function'. LiteLLM Retried: 3 times

I’ve been trying to fix this by reading the docs for days, but I still can’t get it working.
Has anyone else run into similar issues? Or is there another test example I could follow?

Any suggestions would be helpful, thanks!

[d1hr2uv]

maybe its the warmup problem.

[jamesmurdza]

Hi @cy3021561, are you doing this via the web app or in code? If you can provide me with your code, I can try and reproduce the issue.

[cy3021561]

@d1hr2uv Thanks, I did try to keep the inference servers running for a while and keep sending requests to it, but still not working.

@jamesmurdza Thanks for the response. I’ve attached my test code along with the error I ran into below. Appreciate it!

CODE:

import asyncio
import os
from agent import ComputerAgent
from computer import Computer
from dotenv import load_dotenv

load_dotenv()
os.environ["HF_TOKEN"] = os.getenv("HF_TOKEN") or ""
os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY") or ""
os.environ["ANTHROPIC_API_KEY"] = os.getenv("ANTHROPIC_API_KEY") or ""
os.environ["TGI_BASE_URL"] = os.getenv("TGI_BASE_URL") or ""

IMAGE = "trycua/cua-ubuntu:latest"

async def main():
    computer = Computer(
        os_type="linux",
        provider_type="docker",
        image=IMAGE,
        name="my-cua-container",
        noVNC_port=6901,
        port=8000
    )

    agent = ComputerAgent(
        model="huggingface/ByteDance-Seed/UI-TARS-1.5-7B",
        api_base=os.getenv("TGI_BASE_URL"),
        tools=[computer],
        trajectory_dir="trajectories"
    )

    print("VNC Access: http://localhost:6901")
    print("API: http://localhost:8000")

    async for result in agent.run("Open the download folder"):
        print(type(result))

if __name__ == "__main__":
    asyncio.run(main())

ERROR:

  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 114, in _make_common_async_call
    response = await async_httpx_client.post(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/logging_utils.py", line 135, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 324, in post
    raise e
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 280, in post
    response.raise_for_status()
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '422 Unprocessable Entity' for url 'https://ttmgoplirv3o7rwh.us-east-1.aws.endpoints.huggingface.cloud/v1/chat/completions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/422

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/main.py", line 544, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 241, in async_completion
    response = await self._make_common_async_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 139, in _make_common_async_call
    raise self._handle_error(e=e, provider_config=provider_config)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 2409, in _handle_error
    raise provider_config.get_error_class(
litellm.llms.huggingface.common_utils.HuggingFaceError: Failed to deserialize the JSON body into the target type: messages[2]: data did not match any variant of untagged enum MessageBody at line 1 column 1621

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chun/Documents/Bridgent/cua/test_cua.py", line 41, in <module>
    asyncio.run(main())
  File "/Users/chun/.local/share/uv/python/cpython-3.12.8-macos-x86_64-none/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/chun/.local/share/uv/python/cpython-3.12.8-macos-x86_64-none/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chun/.local/share/uv/python/cpython-3.12.8-macos-x86_64-none/lib/python3.12/asyncio/base_events.py", line 686, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/chun/Documents/Bridgent/cua/test_cua.py", line 37, in main
    async for result in agent.run("Open the download folder"):
  File "/Users/chun/Documents/Bridgent/cua/libs/python/agent/agent/agent.py", line 609, in run
    result = await self.agent_loop.predict_step(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chun/Documents/Bridgent/cua/libs/python/agent/agent/loops/uitars.py", line 689, in predict_step
    response = await litellm.acompletion(**api_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/utils.py", line 1586, in wrapper_async
    raise e
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/utils.py", line 1437, in wrapper_async
    result = await original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/main.py", line 563, in acompletion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2301, in exception_type
    raise e
  File "/Users/chun/Documents/Bridgent/cua/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 1599, in exception_type
    raise APIError(
litellm.exceptions.APIError: litellm.APIError: HuggingfaceException - Failed to deserialize the JSON body into the target type: messages[2]: data did not match any variant of untagged enum MessageBody at line 1 column 1621 LiteLLM Retried: 3 times```

[jamesmurdza]

Thank you @cy3021561, I'll test this on Monday.

[tamoghnokandar]

Did this issue get resolved?