Custom Events and Outbound Links aren't being tracked (Self-hosted, Manual Docker, Nginx)

[timothyoesch]

Let me start off by saying: Fabulous app, finally some full featured FOSS analytics with a gorgeous UI.

My issue

Whatever method of rybbit integration I use (I have tried directly in the head, through the wordpress plugin, in nuxt), most functionality works (pageviews and even session replays for sure, I haven't tried error tracking but that's just cause my sites don't create any errors 😏) but custom events and outbound links aren't being tracked.

My investigation

Outbound link tracking is turned on in the site settings and the track event is being sent properly from the client side.

// POST payload to ANALYTICSDOMAIN/api/track
{
   "site_id":"3593434cf857",                // Site id is correct
   "hostname":"***************",
   "pathname":"/",
   "querystring":"",
   "screenWidth":1728,
   "screenHeight":1117,
   "language":"en-CH",
   "page_title":"Hello World :-)",
   "referrer":"",
   "type":"custom_event",                   // "outbound" for Outbound links
   "event_name":"Signup Button Clicked",    // Whatever event name doesn't work
   "properties":"{}"                        // Contains URL, text and target properties for outbound links
}

// With 200 Response
{
   "success":true                            // Somehow I don't believe you.
}

And yet: No custom events nor outbound link clicks are shown in the panel.

My setup

• Docker Docker Engine Version: 28.3.3
• Docker Compose Version 2.39.1
• Docker compose file:

services:
  ## Caddy Service completely removed cause I am using my own NGINX

  clickhouse:
    container_name: clickhouse
    image: clickhouse/clickhouse-server:25.4.2
    volumes:
      - clickhouse-data:/var/lib/clickhouse
    configs:
      - source: clickhouse_network
        target: /etc/clickhouse-server/config.d/network.xml
      - source: clickhouse_json
        target: /etc/clickhouse-server/config.d/enable_json.xml
      - source: clickhouse_logging
        target: /etc/clickhouse-server/config.d/logging_rules.xml
      - source: clickhouse_user_logging
        target: /etc/clickhouse-server/config.d/user_logging.xml
    environment:
      - CLICKHOUSE_DB=${CLICKHOUSE_DB:-analytics}
      - CLICKHOUSE_USER=${CLICKHOUSE_USER:-default}
      - CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD:-frog}
    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8123/ping"]
      interval: 3s
      timeout: 5s
      retries: 5
      start_period: 10s
    restart: unless-stopped

  postgres:
    image: postgres:17.4
    container_name: rb-postgres  ## Specificity, because another service is using postgres as well, might this be causing issues? 
    environment:
      - POSTGRES_USER=${POSTGRES_USER:-frog}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-frog}
      - POSTGRES_DB=${POSTGRES_DB:-analytics}
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 3s
      timeout: 5s
      retries: 5
      start_period: 10s
    restart: unless-stopped

  backend:
    image: ghcr.io/rybbit-io/rybbit-backend:${IMAGE_TAG:-latest}
    container_name: backend
    build:
      context: .
      dockerfile: server/Dockerfile
    ports:
      - "${HOST_BACKEND_PORT:-127.0.0.1:3001:3001}"
    environment:
      - NODE_ENV=production
      - CLICKHOUSE_HOST=http://clickhouse:8123
      - CLICKHOUSE_DB=${CLICKHOUSE_DB:-analytics}
      - CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD:-frog}
      - POSTGRES_HOST=rb-postgres     ## See above
      - POSTGRES_PORT=5432
      - POSTGRES_DB=${POSTGRES_DB:-analytics}
      - POSTGRES_USER=${POSTGRES_USER:-frog}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-frog}
      - BETTER_AUTH_SECRET=${BETTER_AUTH_SECRET}
      - BASE_URL=${BASE_URL}
      - DISABLE_SIGNUP=${DISABLE_SIGNUP}
      - DISABLE_TELEMETRY=${DISABLE_TELEMETRY}
    depends_on:
      clickhouse:
        condition: service_healthy
      postgres:
        condition: service_started
    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://127.0.0.1:3001/api/health"]
      interval: 3s
      timeout: 5s
      retries: 5
      start_period: 10s
    restart: unless-stopped

  client:
    image: ghcr.io/rybbit-io/rybbit-client:${IMAGE_TAG:-latest}
    container_name: client
    build:
      context: .
      dockerfile: client/Dockerfile
      args:
        NEXT_PUBLIC_BACKEND_URL: ${BASE_URL}
        NEXT_PUBLIC_DISABLE_SIGNUP: ${DISABLE_SIGNUP}
    ports:
      - "${HOST_CLIENT_PORT:-127.0.0.1:3002:3002}"
    environment:
      - NODE_ENV=production
      - NEXT_PUBLIC_BACKEND_URL=${BASE_URL}
      - NEXT_PUBLIC_DISABLE_SIGNUP=${DISABLE_SIGNUP}
    depends_on:
      - backend
    restart: unless-stopped

volumes:
  clickhouse-data:
  postgres-data:
  redis-data:

configs:
  clickhouse_network:
    content: |
      <clickhouse>
          <listen_host>0.0.0.0</listen_host>
      </clickhouse>

  clickhouse_json:
    content: |
      <clickhouse>
          <settings>
              <enable_json_type>1</enable_json_type>
          </settings>
      </clickhouse>

  clickhouse_logging:
    content: |
      <clickhouse>
        <logger>
            <level>warning</level>
            <console>true</console>
        </logger>
        <query_thread_log remove="remove"/>
        <query_log remove="remove"/>
        <text_log remove="remove"/>
        <trace_log remove="remove"/>
        <metric_log remove="remove"/>
        <asynchronous_metric_log remove="remove"/>
        <session_log remove="remove"/>
        <part_log remove="remove"/>
        <latency_log remove="remove"/>
        <processors_profile_log remove="remove"/>
      </clickhouse>

  clickhouse_user_logging:
    content: |
      <clickhouse>
        <profiles>
          <default>
            <log_queries>0</log_queries>
            <log_query_threads>0</log_query_threads>
            <log_processors_profiles>0</log_processors_profiles>
          </default>
        </profiles>
      </clickhouse>

• NGINX:

server {
    listen 80;
    server_name analytics.toes.ch;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name analytics.toes.ch;

    {{ssl_certificate_key}}
    {{ssl_certificate}}
    
    location ^~ /.well-known {
        auth_basic off;
        allow all;
        try_files $uri @reverse_proxy;
    }

    location /api/ {
        proxy_pass http://localhost:8801;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location / {
        proxy_pass http://localhost:8802;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

• .env File:

# Domain and URL Configuration
DOMAIN_NAME=analytics.MYDOMAIN.ch
BASE_URL="https://${DOMAIN_NAME}"

# Authentication and Security
BETTER_AUTH_SECRET=THESECRETOFALLSECRETESIAMOBVIOUSLYNOTSHARING
DISABLE_SIGNUP=true

# Webserver Configuration
# Set to false to disable the built-in Caddy webserver
USE_WEBSERVER=false
# Host port mappings - these control how services are exposed
# Format: "host_binding:container_port" or "port:container_port"
HOST_BACKEND_PORT="8801:3001"
HOST_CLIENT_PORT="8802:3002"

# ClickHouse Database Configuration
CLICKHOUSE_DB=analytics
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=*****************************

# PostgreSQL Database Configuration
POSTGRES_DB=analytics
POSTGRES_USER=frog
POSTGRES_PASSWORD=*****************************

00RESEND_API_KEY=rs_XXXXXXX

[goldflag]

I tried reproing this but wasn't able to. The payload you posted does get flagged as a bot payload so it won't track unless you unblock bots (setting enabled by default) but the 200 response you posted doesn't contain the bot message.

[timothyoesch]

Thanks for your reply.

The payload you posted does get flagged as a bot payload

When the request is from the browser, it doesn't flag it as a bot, does it? And the response doesn't indicate this either as you said. I am genuinely flummoxed.

As I am completely unaware how the services in the docker-compose interact, I wanna draw your attention to the fact that I renamed the postgres container to «rb-postgres» again. Did you use that container name in your repro as well? So is that not the cause?

[timothyoesch]

So it wasn't the name of the docker container that's at fault (I moved the other service that was using the name «postgres» and reset the docker-compose.yml).
I setup a page showcasing the behavior here with public analytics here. The tracking is IMHO executed correctly on the client side but it doesn't seem to register on the backend 🥲

[Enjer]

In your page head:

<script src="https://analytics.toes.ch/api/script.js" data-site-id="faa22b37cf7c" async=""></script>

in your page footer:

<script> function trackEvent() { window.rybbit.event("woof", { dog: "randomdog", price: 0.00, }); alert("Woof!"); } </script>

It is because your footer script is immediately executing, your script.js is async excuting, window.rybbit will be undefined before script.js full loaded.
You should set your footer <script defer>...</script>, the problem will be fixed.

[timothyoesch]

Sadly, that does not seem to fix the issue either, @Enjer... I am not surprised though, considering that outbound links should be tracked without any additional code and they aren't. I remain baffled by this, especially because I can see the request in the browser's network tab and the response is 200, but the events don't show up in the analytics dash.
I am really stumped, particularly because this is such a cool app. But without event tracking, my clients aren't convinced to switch :(

[Enjer]

First update your rybbit to lastest version and try

[timothyoesch]

Updated, version is now 2.1.2. Sadly, it's still not working.
Here's a wild idea (and I am a complete noob when it comes to networking, please forgive me if this is a dumb suggestion): There's a very restrictive firewall around my server, "unnecessary" ports are closed. Might that be causing issues?

[timothyoesch]

I just spun up rybbit on a different server quickly and got it working. The only difference is that I am using caddy for reverse proxying instead of NGINX. Works like a charm. That is sadly not a useable solution, since I need to run it on the other server («main server») for data privacy reasons (I am working in a field where it is easier when this sort of data is stored in country and that other server is just some temporary stuff I got from DO).

After restarting the installation process on my main server, I stumbled upon these docker logs when trying to track the «Meow like a cat» Event:

clickhouse  | 2025.11.18 14:52:21.371902 [ 1407 ] {d64367fb-73fb-426e-983c-cf7cc61bd36d} <Error> void DB::ParallelParsingInputFormat::onBackgroundException(): Code: 117. DB::Exception: Cannot parse JSON object here: {"cat":"randomcat","price":"0.00"}: (while reading the value of key props): (at row 1)
clickhouse  | . (INCORRECT_DATA), Stack trace (when copying this message, always include the lines below):
clickhouse  | 
clickhouse  | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f2bff3b
clickhouse  | 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009c74bec
clickhouse  | 2. DB::Exception::Exception<std::basic_string_view<char, std::char_traits<char>>&>(int, FormatStringHelperImpl<std::type_identity<std::basic_string_view<char, std::char_traits<char>>&>::type>, std::basic_string_view<char, std::char_traits<char>>&) @ 0x000000000bd39fab
clickhouse  | 3. DB::SerializationJSON<DB::SimdJSONParser>::deserializeObject(DB::IColumn&, std::basic_string_view<char, std::char_traits<char>>, DB::FormatSettings const&) const @ 0x000000000922719a
clickhouse  | 4. DB::SerializationJSON<DB::SimdJSONParser>::deserializeTextJSON(DB::IColumn&, DB::ReadBuffer&, DB::FormatSettings const&) const @ 0x0000000009226c66
clickhouse  | 5. DB::JSONUtils::readField(DB::ReadBuffer&, DB::IColumn&, std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::ISerialization const> const&, String const&, DB::FormatSettings const&, bool) @ 0x000000001495166a
clickhouse  | 6. DB::JSONEachRowRowInputFormat::readJSONObject(std::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&) @ 0x0000000014ad41cc
clickhouse  | 7. DB::JSONEachRowRowInputFormat::readRow(std::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&, DB::RowReadExtension&) @ 0x0000000014ad4a0e
clickhouse  | 8. DB::IRowInputFormat::read() @ 0x0000000014a1a02a
clickhouse  | 9. DB::IInputFormat::generate() @ 0x00000000149b85d6
clickhouse  | 10. DB::ISource::tryGenerate() @ 0x00000000149914de
clickhouse  | 11. DB::ISource::work() @ 0x00000000149910e7
clickhouse  | 12. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::ParallelParsingInputFormat::scheduleParserThreadForUnitWithNumber(unsigned long)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000014b30d51
clickhouse  | 13. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x000000000f3f941b
clickhouse  | 14. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000f3fff43


backend     | {"level":50,"time":1763477541373,"pid":1,"hostname":"92c0bfbd2c20","name":"rybbit","service":"pageview-queue","err":{"type":"ClickHouseError","message":"Cannot parse JSON object here: {\"cat\":\"randomcat\",\"price\":\"0.00\"}: (while reading the value of key props): (at row 1)\n: While executing ParallelParsingBlockInputFormat. ","stack":"Error: Cannot parse JSON object here: {\"cat\":\"randomcat\",\"price\":\"0.00\"}: (while reading the value of key props): (at row 1)\n: While executing ParallelParsingBlockInputFormat. \n    at parseError (/app/node_modules/@clickhouse/client-common/dist/error/parse_error.js:35:16)\n    at ClientRequest.onResponse (/app/node_modules/@clickhouse/client/dist/connection/node_base_connection.js:409:63)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","code":"117"},"msg":"Error processing pageview queue"}
clickhouse  | 15. ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000f3f674f
clickhouse  | 16. void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000f3fda1a
clickhouse  | 17. ? @ 0x00007f8a08e0cac3
clickhouse  | 18. ? @ 0x00007f8a08e9da04
clickhouse  | 
clickhouse  | Job's origin stack trace:
clickhouse  | 0. void boost::heap::priority_queue<JobWithPriority, boost::heap::stable<true>, boost::parameter::void_, boost::parameter::void_, boost::parameter::void_>::emplace<std::function<void ()>, Priority&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>&, DB::OpenTelemetry::TracingContextOnThread const, bool&, (anonymous namespace)::ScopedDecrement>(std::function<void ()>&&, Priority&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>&, DB::OpenTelemetry::TracingContextOnThread const&&, bool&, (anonymous namespace)::ScopedDecrement&&) @ 0x000000000f3fe657
clickhouse  | 1. void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x000000000f3fa7f8
clickhouse  | 2. DB::ParallelParsingInputFormat::segmentatorThreadFunction(std::shared_ptr<DB::ThreadGroup>) @ 0x0000000014b2ef8f
clickhouse  | 3. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<void (DB::ParallelParsingInputFormat::*)(std::shared_ptr<DB::ThreadGroup>), DB::ParallelParsingInputFormat*, std::shared_ptr<DB::ThreadGroup>>(void (DB::ParallelParsingInputFormat::*&&)(std::shared_ptr<DB::ThreadGroup>), DB::ParallelParsingInputFormat*&&, std::shared_ptr<DB::ThreadGroup>&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000014b32acd
clickhouse  | 4. ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000f3f674f
clickhouse  | 5. void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000f3fda1a
clickhouse  | 6. ? @ 0x00007f8a08e0cac3
clickhouse  | 7. ? @ 0x00007f8a08e9da04
clickhouse  | 
clickhouse  | Job's origin stack trace:
clickhouse  | 0. void boost::heap::priority_queue<JobWithPriority, boost::heap::stable<true>, boost::parameter::void_, boost::parameter::void_, boost::parameter::void_>::emplace<std::function<void ()>, Priority&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>&, DB::OpenTelemetry::TracingContextOnThread const, bool&, (anonymous namespace)::ScopedDecrement>(std::function<void ()>&&, Priority&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>&, DB::OpenTelemetry::TracingContextOnThread const&&, bool&, (anonymous namespace)::ScopedDecrement&&) @ 0x000000000f3fe657
clickhouse  | 1. void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x000000000f3f7a38
clickhouse  | 2. ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<void (DB::ParallelParsingInputFormat::*)(std::shared_ptr<DB::ThreadGroup>), DB::ParallelParsingInputFormat*, std::shared_ptr<DB::ThreadGroup>>(void (DB::ParallelParsingInputFormat::*&&)(std::shared_ptr<DB::ThreadGroup>), DB::ParallelParsingInputFormat*&&, std::shared_ptr<DB::ThreadGroup>&&) @ 0x0000000014b2fddd
clickhouse  | 3. DB::ParallelParsingInputFormat::read() @ 0x0000000014b2fba2
clickhouse  | 4. DB::IInputFormat::generate() @ 0x00000000149b85d6
clickhouse  | 5. DB::ISource::tryGenerate() @ 0x00000000149914de
clickhouse  | 6. DB::ISource::work() @ 0x00000000149910e7
clickhouse  | 7. DB::ExecutionThreadContext::executeTask() @ 0x00000000149ad496
clickhouse  | 8. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x00000000149a0ea3
clickhouse  | 9. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001499f9a5
clickhouse  | 10. DB::CompletedPipelineExecutor::execute() @ 0x000000001499e1ef
clickhouse  | 11. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&, String const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::FormatSettings> const&)>) @ 0x000000001359bf34
clickhouse  | 12. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000014861401
clickhouse  | 13. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000014865969
clickhouse  | 14. DB::HTTPServerConnection::run() @ 0x000000001491bb2a
clickhouse  | 15. Poco::Net::TCPServerConnection::start() @ 0x00000000180162e7
clickhouse  | 16. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018016739
clickhouse  | 17. Poco::PooledThread::run() @ 0x0000000017fe1a5b
clickhouse  | 18. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000017fdff3d
clickhouse  | 19. ? @ 0x00007f8a08e0cac3
clickhouse  | 20. ? @ 0x00007f8a08e9da04
clickhouse  |  (version 25.4.2.31 (official build))
clickhouse  | 2025.11.18 14:52:21.372351 [ 757 ] {d64367fb-73fb-426e-983c-cf7cc61bd36d} <Error> executeQuery: Code: 117. DB::Exception: Cannot parse JSON object here: {"cat":"randomcat","price":"0.00"}: (while reading the value of key props): (at row 1)
clickhouse  | : While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.4.2.31 (official build)) (from 192.168.96.4:34948) (in query: INSERT INTO events FORMAT JSONEachRow ), Stack trace (when copying this message, always include the lines below):
clickhouse  | 
clickhouse  | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f2bff3b
clickhouse  | 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009c74bec
clickhouse  | 2. DB::Exception::Exception<std::basic_string_view<char, std::char_traits<char>>&>(int, FormatStringHelperImpl<std::type_identity<std::basic_string_view<char, std::char_traits<char>>&>::type>, std::basic_string_view<char, std::char_traits<char>>&) @ 0x000000000bd39fab
clickhouse  | 3. DB::SerializationJSON<DB::SimdJSONParser>::deserializeObject(DB::IColumn&, std::basic_string_view<char, std::char_traits<char>>, DB::FormatSettings const&) const @ 0x000000000922719a
clickhouse  | 4. DB::SerializationJSON<DB::SimdJSONParser>::deserializeTextJSON(DB::IColumn&, DB::ReadBuffer&, DB::FormatSettings const&) const @ 0x0000000009226c66
clickhouse  | 5. DB::JSONUtils::readField(DB::ReadBuffer&, DB::IColumn&, std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::ISerialization const> const&, String const&, DB::FormatSettings const&, bool) @ 0x000000001495166a
clickhouse  | 6. DB::JSONEachRowRowInputFormat::readJSONObject(std::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&) @ 0x0000000014ad41cc
clickhouse  | 7. DB::JSONEachRowRowInputFormat::readRow(std::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&, DB::RowReadExtension&) @ 0x0000000014ad4a0e
clickhouse  | 8. DB::IRowInputFormat::read() @ 0x0000000014a1a02a
clickhouse  | 9. DB::IInputFormat::generate() @ 0x00000000149b85d6
clickhouse  | 10. DB::ISource::tryGenerate() @ 0x00000000149914de
clickhouse  | 11. DB::ISource::work() @ 0x00000000149910e7
clickhouse  | 12. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::ParallelParsingInputFormat::scheduleParserThreadForUnitWithNumber(unsigned long)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000014b30d51
clickhouse  | 13. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x000000000f3f941b
clickhouse  | 14. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000f3fff43
clickhouse  | 15. ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000f3f674f
clickhouse  | 16. void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000f3fda1a
clickhouse  | 17. ? @ 0x00007f8a08e0cac3
clickhouse  | 18. ? @ 0x00007f8a08e9da04
clickhouse  | 
clickhouse  | 2025.11.18 14:52:21.372789 [ 757 ] {d64367fb-73fb-426e-983c-cf7cc61bd36d} <Error> DynamicQueryHandler: Code: 117. DB::Exception: Cannot parse JSON object here: {"cat":"randomcat","price":"0.00"}: (while reading the value of key props): (at row 1)
clickhouse  | : While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA), Stack trace (when copying this message, always include the lines below):
clickhouse  | 
clickhouse  | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f2bff3b
clickhouse  | 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009c74bec
clickhouse  | 2. DB::Exception::Exception<std::basic_string_view<char, std::char_traits<char>>&>(int, FormatStringHelperImpl<std::type_identity<std::basic_string_view<char, std::char_traits<char>>&>::type>, std::basic_string_view<char, std::char_traits<char>>&) @ 0x000000000bd39fab
clickhouse  | 3. DB::SerializationJSON<DB::SimdJSONParser>::deserializeObject(DB::IColumn&, std::basic_string_view<char, std::char_traits<char>>, DB::FormatSettings const&) const @ 0x000000000922719a
clickhouse  | 4. DB::SerializationJSON<DB::SimdJSONParser>::deserializeTextJSON(DB::IColumn&, DB::ReadBuffer&, DB::FormatSettings const&) const @ 0x0000000009226c66
clickhouse  | 5. DB::JSONUtils::readField(DB::ReadBuffer&, DB::IColumn&, std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::ISerialization const> const&, String const&, DB::FormatSettings const&, bool) @ 0x000000001495166a
clickhouse  | 6. DB::JSONEachRowRowInputFormat::readJSONObject(std::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&) @ 0x0000000014ad41cc
clickhouse  | 7. DB::JSONEachRowRowInputFormat::readRow(std::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&, DB::RowReadExtension&) @ 0x0000000014ad4a0e
clickhouse  | 8. DB::IRowInputFormat::read() @ 0x0000000014a1a02a
clickhouse  | 9. DB::IInputFormat::generate() @ 0x00000000149b85d6
clickhouse  | 10. DB::ISource::tryGenerate() @ 0x00000000149914de
clickhouse  | 11. DB::ISource::work() @ 0x00000000149910e7
clickhouse  | 12. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::ParallelParsingInputFormat::scheduleParserThreadForUnitWithNumber(unsigned long)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000014b30d51
clickhouse  | 13. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x000000000f3f941b
clickhouse  | 14. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000f3fff43
clickhouse  | 15. ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000f3f674f
clickhouse  | 16. void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000f3fda1a
clickhouse  | 17. ? @ 0x00007f8a08e0cac3
clickhouse  | 18. ? @ 0x00007f8a08e9da04
clickhouse  | 
clickhouse  | Job's origin stack trace:
clickhouse  | 0. void boost::heap::priority_queue<JobWithPriority, boost::heap::stable<true>, boost::parameter::void_, boost::parameter::void_, boost::parameter::void_>::emplace<std::function<void ()>, Priority&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>&, DB::OpenTelemetry::TracingContextOnThread const, bool&, (anonymous namespace)::ScopedDecrement>(std::function<void ()>&&, Priority&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>&, DB::OpenTelemetry::TracingContextOnThread const&&, bool&, (anonymous namespace)::ScopedDecrement&&) @ 0x000000000f3fe657
clickhouse  | 1. void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x000000000f3fa7f8
clickhouse  | 2. DB::ParallelParsingInputFormat::segmentatorThreadFunction(std::shared_ptr<DB::ThreadGroup>) @ 0x0000000014b2ef8f
clickhouse  | 3. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<void (DB::ParallelParsingInputFormat::*)(std::shared_ptr<DB::ThreadGroup>), DB::ParallelParsingInputFormat*, std::shared_ptr<DB::ThreadGroup>>(void (DB::ParallelParsingInputFormat::*&&)(std::shared_ptr<DB::ThreadGroup>), DB::ParallelParsingInputFormat*&&, std::shared_ptr<DB::ThreadGroup>&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000014b32acd
clickhouse  | 4. ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000f3f674f
clickhouse  | 5. void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000f3fda1a
clickhouse  | 6. ? @ 0x00007f8a08e0cac3
clickhouse  | 7. ? @ 0x00007f8a08e9da04
clickhouse  | 
clickhouse  | Job's origin stack trace:
clickhouse  | 0. void boost::heap::priority_queue<JobWithPriority, boost::heap::stable<true>, boost::parameter::void_, boost::parameter::void_, boost::parameter::void_>::emplace<std::function<void ()>, Priority&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>&, DB::OpenTelemetry::TracingContextOnThread const, bool&, (anonymous namespace)::ScopedDecrement>(std::function<void ()>&&, Priority&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>&, DB::OpenTelemetry::TracingContextOnThread const&&, bool&, (anonymous namespace)::ScopedDecrement&&) @ 0x000000000f3fe657
clickhouse  | 1. void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x000000000f3f7a38
clickhouse  | 2. ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<void (DB::ParallelParsingInputFormat::*)(std::shared_ptr<DB::ThreadGroup>), DB::ParallelParsingInputFormat*, std::shared_ptr<DB::ThreadGroup>>(void (DB::ParallelParsingInputFormat::*&&)(std::shared_ptr<DB::ThreadGroup>), DB::ParallelParsingInputFormat*&&, std::shared_ptr<DB::ThreadGroup>&&) @ 0x0000000014b2fddd
clickhouse  | 3. DB::ParallelParsingInputFormat::read() @ 0x0000000014b2fba2
clickhouse  | 4. DB::IInputFormat::generate() @ 0x00000000149b85d6
clickhouse  | 5. DB::ISource::tryGenerate() @ 0x00000000149914de
clickhouse  | 6. DB::ISource::work() @ 0x00000000149910e7
clickhouse  | 7. DB::ExecutionThreadContext::executeTask() @ 0x00000000149ad496
clickhouse  | 8. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x00000000149a0ea3
clickhouse  | 9. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001499f9a5
clickhouse  | 10. DB::CompletedPipelineExecutor::execute() @ 0x000000001499e1ef
clickhouse  | 11. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&, String const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::FormatSettings> const&)>) @ 0x000000001359bf34
clickhouse  | 12. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000014861401
clickhouse  | 13. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000014865969
clickhouse  | 14. DB::HTTPServerConnection::run() @ 0x000000001491bb2a
clickhouse  | 15. Poco::Net::TCPServerConnection::start() @ 0x00000000180162e7
clickhouse  | 16. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018016739
clickhouse  | 17. Poco::PooledThread::run() @ 0x0000000017fe1a5b
clickhouse  | 18. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000017fdff3d
clickhouse  | 19. ? @ 0x00007f8a08e0cac3
clickhouse  | 20. ? @ 0x00007f8a08e9da04
clickhouse  |  (version 25.4.2.31 (official build))

Can you make sense of it? I sadly can't.