Which ContentBlocks actually can come in as chunks

[jwflicker]

• UserMessageChunk, AgentMessageChunk and AgentThoughtChunk all contain ContentBlock content.
• A ContentBlock can be TextBlock, ImageBlock, AudioBlock, ResourceLinkBlock, EmbeddedResourceBlock
• An EmbeddedResourceBlock's resource can be a TextResourceContents, BlobResourceContents

Which of these in practice can be received in chunk form?
Im not seeing how ResourceLinkBlock would be chunked for example...

[benbrandt]

A chunk represents a portion of a complete response. You are correct that these individual items may not be chunked, but they could be part of the same response (i.e. text, text, text, resource, image, text)
Some of these can be coalesced at the block level, others not. But it is assumed that all consecutive UserMessageChunks make up a complete response, and the first AgentMessageChunk starts a new agent message.

Hopefully that helps?

[jwflicker]

Yea i guess it is implied that markdown is the presentation language of these text chunks. In my case i am using a markdown->html converter and a code-styler for markdown ``` blocks. So when i get a chunk i see what the last element in the chat was and if it is a chunk of the same type (UserMessageChunks) I append the chunk to the array of chunks so far and regenerate the html from the markdown chunks so far. I'm guessing ImageBlock, AudioBlock, ResourceLinkBlock could be "embedded" in the overall markdown-ish content like this:

so... lets say i ask to explain a folder of code files... could the explanation contain somethink like

<text block> the file <text block>
<resource link> foo.bar<resource link>
<text block> references two other files.<text block> (chunk 1)
<text block>  It references <text block>. (chunk 2)
<resource link> alpha.bar<resource link>
<text block> for all things alpha and it references <text block>
<resource link> zed.bar<resource link>
<text block>for all things zed<text block>

In which case i might want to include these links inside the "transformed markdown" section of the dom rather than outside of it... so it can be presented as a single paragraph w/o line breaks

[jwflicker]

I guess for an EmbeddedResourceBlock, i check if the prior element has the same uri. if it has the same then i assume it is a chunk to a prior embedded content, else i assume its a new embedded content block...

not sure how i would figure out if consecutive image or audio chunks represent one image/audio or two...