how to add the "query similarity score" to the object in response tree

[ebarolo]

Is it possible to add the similarity score of the object attribute in the Tree response?

Thank you

[gulyabaniTR]

If your agent use "hybrid" or "vector" query to retrive data it can be possible. If no one attempts to do this within a few days, I can handle in both the backend and front end sides.

in elysia/tools/retrieval/util.py between line 606-630 there is part for query you can add return_metadata=MetadataQuery(distance=True) to return distances but you have to modify/refactor dtos/classes, return objects and all others parts in codebase.

            if tool_args["search_type"] == "hybrid":
                response = await collection.query.hybrid(
                    query=tool_args["search_query"],
                    limit=tool_args["limit"],
                    filters=combined_filter,
                    return_references=reference,
                    target_vector=(
                        named_vector_fields[collection.name]
                        if named_vector_fields
                        else None
                    ),
                )
            elif tool_args["search_type"] == "vector":
                response = await collection.query.near_text(
                    query=tool_args["search_query"],
                    limit=tool_args["limit"],
                    filters=combined_filter,
                    return_references=reference,
                    target_vector=(
                        named_vector_fields[collection.name]
                        if named_vector_fields
                        else None
                    ),
                )

In Weaviate, setting distance=True in the return_metadata parameter (return_metadata=MetadataQuery(distance=True)) of a query instructs the client to include the vector distance between the query and each returned object in the search results. This distance is a measure of similarity: a lower distance means higher similarity between the query and the object.

Docs:
https://docs.weaviate.io/academy/py/starter_text_data/text_searches/semantic
https://docs.weaviate.io/weaviate/search/similarity

[dannyjameswilliams]

Yep, @gulyabaniTR is spot on! We need to edit the weaviate query code itself to be able to achieve this, and then it'd be necessary to add a field to the returned objects themselves. It'd need to added alongside the actual object fields in the returned Result/Retrieval object, and then the frontend would need to be aware that this particular field for similarity score is not a field in the data, but a piece of metadata. We should probably have a standard string indicator e.g. "ELYSIA_*" gets auto-ignored by the frontend, but let's leave that for another time.

@gulyabaniTR I can assign this to you if you would like? I'm also happy to do it. I don't see any harm in adding this to the retrieved objects if possible. We'd just need to make sure it's not explicitly used anywhere else in the tree otherwise it will conflict with when there's no scores (e.g. fetch_objects)

[gulyabaniTR]

Hi Danny, thanks for your comment and guidance. This feature/improvement definitely affects a lot of areas in the codebase, so we need to proceed carefully, as you mentioned.

And yes, if it's not urgent, I'd love to look into this. I'll be waiting for you to provide the necessary guidance.

[dannyjameswilliams]

Amazing! So just like you said, the main change would be to add the return metadata to hybrid and near_text in:

elysia/elysia/tools/retrieval/util.py

Lines 606 to 630 in c5b6451

	if tool_args["search_type"] == "hybrid":
	response = await collection.query.hybrid(
	query=tool_args["search_query"],
	limit=tool_args["limit"],
	filters=combined_filter,
	return_references=reference,
	target_vector=(
	named_vector_fields[collection.name]
	if named_vector_fields
	else None
	),
	)
	elif tool_args["search_type"] == "vector":
	response = await collection.query.near_text(
	query=tool_args["search_query"],
	limit=tool_args["limit"],
	filters=combined_filter,
	return_references=reference,
	target_vector=(
	named_vector_fields[collection.name]
	if named_vector_fields
	else None
	),
	)

Then augment the objects with some kind of new property, similar to how uuid is added:

elysia/elysia/tools/retrieval/query.py

Lines 570 to 573 in c5b6451

	objects = []
	for obj in response.objects:
	objects.append({k: v for k, v in obj.properties.items()})
	objects[-1]["uuid"] = str(obj.uuid)

E.g. a ELYSIA_SIMILARITY_SCORE field, or similar.

Then I think the Retrieval object will need to be default initialised that the field ELYSIA_SIMILARITY_SCORE won't be overwritten when mapping the base object fields to those recognised by the frontend, so the unmapped_keys should include this field name:

elysia/elysia/objects.py

Lines 824 to 843 in c5b6451

	class Retrieval(Result):
	"""
	Store of returned objects from a query/aggregate/any displayed results.
	"""
	def __init__(
	self,
	objects: list[dict],
	metadata: dict = {},
	payload_type: str = "default",
	name: str | None = None,
	mapping: dict | None = None,
	unmapped_keys: list[str] = [
	"uuid",
	"ELYSIA_SUMMARY",
	"collection_name",
	"_REF_ID",
	],
	display: bool = True,
	) -> None:

This should be enough, I believe to give the LLM what the similarity score is. Then all we'd need is to give the LLM this knowledge when it summarises retrieved information, so adding an explanation of how to use this field (just explaining that this is a similarity score, higher=better) into the prompt, probably in the cited summariser:

elysia/elysia/tools/text/prompt_templates.py

Lines 6 to 50 in c5b6451

	class CitedSummarizingPrompt(dspy.Signature):
	"""
	Given a user_prompt, as well as a list of retrieved objects from the environment, summarize the information in the objects to answer the user's prompt.
	Your output will be a collection of TextWithCitation objects that form a complete summary when combined.
	Do not list any of the retrieved objects in your response. Do not give an itemised list of the objects, since they will be displayed to the user anyway.
	Your summary should be parsing the retrieved objects, and summarising the information in them that is relevant to the user's prompt.
	For example, picking the most relevant information from the retrieved objects, and discussing it, repeating ONLY relevant information if necessary.
	You should provide useful analysis, new information via analysing the existing objects, and synthesising the information.
	"""
	subtitle = dspy.OutputField(description="A subtitle for the summary")
	cited_text: List[TextWithCitation] = dspy.OutputField(
	description="""
	A list of TextWithCitation objects whose 'text' fields will be concatenated to form the complete summary.
	FORMATTING RULES:
	1. Use markdown for structure (headings, lists, etc.), this is encouraged and recommended to break down the information to be more readable.
	2. IMPORTANT: Always add newline characters (\\n) before and after markdown elements (lists, headers)
	3. When a formatting element spans multiple TextWithCitation objects, ensure each object has proper newlines
	4. Example: If a bullet point list spans multiple objects, each object must end with \\n and the next begin with proper indentation
	CITATION RULES:
	1. NEVER include reference IDs in the 'text' field itself
	2. NEVER include the literal string "_REF_ID" in your text
	3. NEVER create hyperlinks to references in the text field
	4. ALWAYS copy reference IDs exactly from the source's _REF_ID to the ref_ids list
	5. Keep the text clean and reference-free: all citations happen only through the ref_ids list
	6. Since you are referencing via _REF_ID, you do not need any other type of referencing such as URLs or hyperlinks.
	CORRECT EXAMPLE: text: "Here is some text", ref_ids: ["example_citation_1", "example_citation_2"]
	INCORRECT EXAMPLE: text: "Here is some text _REF_ID", ref_ids: ["example_citation_1", "example_citation_2"]
	INCORRECT EXAMPLE: text: "Here is some text [example_citation_1]", ref_ids: ["example_citation_1", "example_citation_2"]
	CONTENT GUIDELINES:
	1. Focus only on information relevant to the user's prompt
	2. Provide analysis and synthesis, not just extraction
	3. Only cite information that exists in the retrieved objects
	4. Only cite _relevant_ information from the objects as it relates to the text.
	5. Each text segment should map logically to its citations
	6. Use multiple TextWithCitation objects to align text with its specific sources
	7. Do not just repeat information from the environment. Create insights, offer suggestions, do not just list objects.
	Do not write multiple text segments which share the same ref_ids, combine the two text segments into one and use the ref_ids on both text segments.
	""".strip()
	)

Separately, the frontend should then be able to use the ELYSIA_SIMILARITY_SCORE field to display the scores when objects are retrieved.

I'd recommend opening a feature request at https://github.com/weaviate/elysia-frontend when your PR here is done, linking to it so that it can be implemented in due time. Or, making a separate PR over there manually, if you feel up to it.

I think this should be all the changes required, but of course it will be worth testing :) If you have some experimental setups in Python, make sure to use e.g. tree.complex_lm.inspect_history(5) or tree.base_lm.inspect_history(5) to see the raw LLM calls and check they include the new fields to ensure its been passed down to the tree correctly. Good luck and let me know if you need more advice!

[ebarolo]

Any news ? UP