OpenAI Realtime API support

[bachittle]

https://platform.openai.com/docs/guides/realtime

it uses websockets to get and set audio files from gpt-4o-realtime.

[anhao]

+1

[WqyJh]

+1

[WqyJh]

I want to add support for the GPT-4o-realtime model, which relies on WebSocket technology. This necessitates the use of a WebSocket library, but introducing such a dependency conflicts with the zero-dependency philosophy of the existing library, as I previously discussed with @sashabaranov.

As a result, I've decided to create a new library dedicated exclusively to GPT-4o-realtime. This library will serve as a complement to go-openai, focusing solely on supporting GPT-4o-realtime functionality.

The new library is called go-openai-realtime. Feel free to check it out!

https://github.com/WqyJh/go-openai-realtime

[sashabaranov]

@WqyJh, thank you for your effort on this! I think websockets are a fair case to introduce a dependency to this library, and I would love to merge your changes if you'll decide to contribute 🙌🏻

[bachittle]

They just released support for audio input and output in the chat completions endpoint, using the gpt-4o-audio-preview model. This could be supported first in the meantime: https://platform.openai.com/docs/guides/audio/quickstart

[WqyJh]

They just released support for audio input and output in the chat completions endpoint, using the gpt-4o-audio-preview model. This could be supported first in the meantime: https://platform.openai.com/docs/guides/audio/quickstart

I just added support for gpt-4o-audio-preview. See #895

[WqyJh]

@WqyJh, thank you for your effort on this! I think websockets are a fair case to introduce a dependency to this library, and I would love to merge your changes if you'll decide to contribute 🙌🏻

I'd like to contribute to the project. Since it contains a lot of code and examples, it will take some time to complete. Mixing all the code together would create a mess, so I suggest organizing all the real-time code into a folder named realtime.

[sabuhigr]

No need to implement realtime api with websockets/webRTC I think.

Completion api supports Audio input now.
Ref: https://platform.openai.com/docs/guides/audio

The project is concentrated to api level functionalities with REST not with different protocol like WebRTC or Websocket.

[HaraldNordgren]

@sashabaranov I would be great if Realtime support could be added soon.

In the meantime, thanks for creating https://github.com/WqyJh/go-openai-realtime @WqyJh !