Skip to content

Conversation

@meirk-brd
Copy link
Collaborator

@meirk-brd meirk-brd commented Nov 4, 2025

Summary

Optimizes token consumption for structured and markdown data:

  1. TOON Format Integration: Adds Token-Oriented Object Notation for web_data_* tools, reducing tokens by 10-30% vs JSON while maintaining readability

  2. Markdown Minification: Implements remark + strip-markdown plugin for scrape_as_markdown tool, achieving 60% token reduction by removing base64 images, HTML tags, and formatting while preserving meaningful content

  3. Google Search Response Sanitization: Cleanses search results by normalizing whitespace, removing Unicode formatting characters, deduplicating related keywords, and extracting only essential fields (link, title, description) from organic results (Thanks to Nikita for this valuable feedback | example response)

Changes

  • Integrated TOON serialization library for compact data representation
  • Added remark/strip-markdown processing pipeline for markdown responses
  • Implemented clean_google_search_payload() to sanitize and normalize search engine results

(Still being tested and evaluated for efficiency)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants