Segmentation Fault with spine v1.2.30

[earendilfr]

Describe the bug
I encounter some time the following error message in log from spine

2025-08-21 16:25:41 -  FATAL: Spine Encountered a Segmentation Fault
Generating backtrace...0 line(s)...
double free or corruption (fasttop)
malloc(): unaligned tcache chunk detected
2025-08-21 17:15:40 -  FATAL: Spine Encountered a Segmentation Fault
Generating backtrace...0 line(s)...

The processus continue but I have no idea if I have to be concerned...

Server (please complete the following information):

• OS: Rocky Linux 9.6
• Version: 1.2.30

Compiling (please complete the following information):

• compiler: gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5)
• autoconf: autoconf (GNU Autoconf) 2.69
• glibc: ldd (GNU libc) 2.34
• source: release

Note: you can find glibc version by running ldd --version

Note: if source is github, please include last commit reference

Additional context
If you need any other output, I will be happy to provide you the required information

[TheWitness]

Use the 1.2.x branch, the backtrace functions have been removed. Report back. Thanks!

[earendilfr]

With the git version (retrieve from git):

malloc(): unaligned tcache chunk detected
malloc(): unaligned tcache chunk detected
2025-08-22 08:07:19 -  FATAL: Spine Encountered a Segmentation Fault
double free or corruption (fasttop)
2025-08-22 08:15:50 -  FATAL: Spine Encountered a Segmentation Fault
2025-08-22 08:21:00 -  FATAL: Spine Encountered a Segmentation Fault
double free or corruption (fasttop)
malloc(): unaligned tcache chunk detected
2025-08-22 08:25:50 -  FATAL: Spine Encountered a Segmentation Fault
2025-08-22 08:31:20 -  FATAL: Spine Encountered a Segmentation Fault

And the version:

• spine:

# /var/www/cacti/bin/spine --version
SPINE 1.2.29  Copyright 2004-2024 by The Cacti Group

• git status

$ git log
commit af323e479d0a24ace2975d77d73f7f7f1e4ecbaa (HEAD, origin/1.2.x)
Author: TheWitness <thewitness@cacti.net>
Date:   Mon Jun 30 09:50:28 2025 -0400

    QA: Fix to warnings from user testing

[TheWitness]

What OS and version?

[TheWitness]

Can you run with gdb and do a backtrace? Do you know how to do that?

[earendilfr]

For OS and version:

• OS: Rocky Linux 9.6
• Cacti version: 1.2.30

For gdb, I think I need to use the command gdb -p $(pgrep spine) but how can I say to monitor each time a spine processes is launched because the spine doesn't do a segmentation fault each time... (and I launch a pool every 10 seconds...)

[TheWitness]

cd /use/local/spine/bin
gdb ./spine 
run
<<<segfault happens>>>
bt

[earendilfr]

I have this output:

GNU gdb (Rocky Linux) 14.2-4.1.el9_6
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 288250
Reading symbols from /var/www/cacti/bin/spine...
Reading symbols from /lib64/libnetsnmp.so.40...
Reading symbols from .gnu_debugdata for /lib64/libnetsnmp.so.40...
(No debugging symbols found in .gnu_debugdata for /lib64/libnetsnmp.so.40)
Reading symbols from /lib64/libmariadb.so.3...
Reading symbols from .gnu_debugdata for /lib64/libmariadb.so.3...
(No debugging symbols found in .gnu_debugdata for /lib64/libmariadb.so.3)
Reading symbols from /lib64/libcrypto.so.3...
Reading symbols from .gnu_debugdata for /lib64/libcrypto.so.3...
(No debugging symbols found in .gnu_debugdata for /lib64/libcrypto.so.3)
Reading symbols from /lib64/libz.so.1...
Reading symbols from .gnu_debugdata for /lib64/libz.so.1...
(No debugging symbols found in .gnu_debugdata for /lib64/libz.so.1)
Reading symbols from /lib64/libm.so.6...
Reading symbols from .gnu_debugdata for /lib64/libm.so.6...
(No debugging symbols found in .gnu_debugdata for /lib64/libm.so.6)
Reading symbols from /lib64/libssl.so.3...
Reading symbols from .gnu_debugdata for /lib64/libssl.so.3...
(No debugging symbols found in .gnu_debugdata for /lib64/libssl.so.3)
Reading symbols from /lib64/libc.so.6...
(No debugging symbols found in /lib64/libc.so.6)
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /lib64/libnss_sss.so.2...
Reading symbols from .gnu_debugdata for /lib64/libnss_sss.so.2...
(No debugging symbols found in .gnu_debugdata for /lib64/libnss_sss.so.2)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f0ef6904769 in select () from /lib64/libc.so.6
Continuing.
[New Thread 0x7f0ef67ff640 (LWP 288269)]
[New Thread 0x7f0ef5ffe640 (LWP 288271)]
[New Thread 0x7f0ef57fd640 (LWP 288272)]
[New Thread 0x7f0ef4ffc640 (LWP 288273)]
[Thread 0x7f0ef5ffe640 (LWP 288271) exited]
[New Thread 0x7f0ee7fff640 (LWP 288275)]
[Thread 0x7f0ef67ff640 (LWP 288269) exited]
[Thread 0x7f0ef4ffc640 (LWP 288273) exited]
[Thread 0x7f0ef57fd640 (LWP 288272) exited]
[New Thread 0x7f0ef57fd640 (LWP 288276)]
[New Thread 0x7f0ef4ffc640 (LWP 288277)]
[New Thread 0x7f0ef67ff640 (LWP 288278)]
[Thread 0x7f0ee7fff640 (LWP 288275) exited]
[New Thread 0x7f0ee7fff640 (LWP 288280)]
[Thread 0x7f0ef57fd640 (LWP 288276) exited]
[Thread 0x7f0ef4ffc640 (LWP 288277) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288281)]
[New Thread 0x7f0ef57fd640 (LWP 288282)]
[Thread 0x7f0ef67ff640 (LWP 288278) exited]
[New Thread 0x7f0ef67ff640 (LWP 288283)]
[Thread 0x7f0ee7fff640 (LWP 288280) exited]
[New Thread 0x7f0ee7fff640 (LWP 288284)]
[Thread 0x7f0ef4ffc640 (LWP 288281) exited]
[Thread 0x7f0ef57fd640 (LWP 288282) exited]
[New Thread 0x7f0ef57fd640 (LWP 288286)]
[New Thread 0x7f0ef4ffc640 (LWP 288287)]
[Thread 0x7f0ef67ff640 (LWP 288283) exited]
[New Thread 0x7f0ef67ff640 (LWP 288288)]
[Thread 0x7f0ee7fff640 (LWP 288284) exited]
[New Thread 0x7f0ee7fff640 (LWP 288289)]
[Thread 0x7f0ef57fd640 (LWP 288286) exited]
[New Thread 0x7f0ef57fd640 (LWP 288290)]
[Thread 0x7f0ef4ffc640 (LWP 288287) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288291)]
[Thread 0x7f0ef67ff640 (LWP 288288) exited]
[New Thread 0x7f0ef67ff640 (LWP 288293)]
[Thread 0x7f0ee7fff640 (LWP 288289) exited]
[New Thread 0x7f0ee7fff640 (LWP 288294)]
[Thread 0x7f0ef57fd640 (LWP 288290) exited]
[New Thread 0x7f0ef57fd640 (LWP 288295)]
[Detaching after vfork from child process 288296]
[Thread 0x7f0ee7fff640 (LWP 288294) exited]
[Thread 0x7f0ef4ffc640 (LWP 288291) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288297)]
[New Thread 0x7f0ee7fff640 (LWP 288298)]
[Thread 0x7f0ef4ffc640 (LWP 288297) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288299)]
[Thread 0x7f0ee7fff640 (LWP 288298) exited]
[New Thread 0x7f0ee7fff640 (LWP 288300)]
[Thread 0x7f0ef67ff640 (LWP 288293) exited]
[New Thread 0x7f0ef67ff640 (LWP 288301)]
[Detaching after vfork from child process 288303]
[Thread 0x7f0ef4ffc640 (LWP 288299) exited]
[Thread 0x7f0ee7fff640 (LWP 288300) exited]
[New Thread 0x7f0ee7fff640 (LWP 288304)]
[New Thread 0x7f0ef4ffc640 (LWP 288305)]
[Thread 0x7f0ef67ff640 (LWP 288301) exited]
[New Thread 0x7f0ef67ff640 (LWP 288307)]
[Detaching after vfork from child process 288308]
[Thread 0x7f0ee7fff640 (LWP 288304) exited]
[New Thread 0x7f0ee7fff640 (LWP 288309)]
[Thread 0x7f0ef4ffc640 (LWP 288305) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288310)]
[Thread 0x7f0ef67ff640 (LWP 288307) exited]
[New Thread 0x7f0ef67ff640 (LWP 288312)]
[Thread 0x7f0ef57fd640 (LWP 288295) exited]
[New Thread 0x7f0ef57fd640 (LWP 288313)]
[Thread 0x7f0ee7fff640 (LWP 288309) exited]
[New Thread 0x7f0ee7fff640 (LWP 288314)]
[Thread 0x7f0ef67ff640 (LWP 288312) exited]
[New Thread 0x7f0ef67ff640 (LWP 288315)]
[Thread 0x7f0ef4ffc640 (LWP 288310) exited]
[Thread 0x7f0ef57fd640 (LWP 288313) exited]
[New Thread 0x7f0ef57fd640 (LWP 288317)]
[New Thread 0x7f0ef4ffc640 (LWP 288318)]
[Thread 0x7f0ee7fff640 (LWP 288314) exited]
[New Thread 0x7f0ee7fff640 (LWP 288319)]
[Thread 0x7f0ef67ff640 (LWP 288315) exited]
[New Thread 0x7f0ef67ff640 (LWP 288320)]
[Thread 0x7f0ef4ffc640 (LWP 288318) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288322)]
[Thread 0x7f0ef57fd640 (LWP 288317) exited]
[New Thread 0x7f0ef57fd640 (LWP 288323)]
[Thread 0x7f0ee7fff640 (LWP 288319) exited]
[New Thread 0x7f0ee7fff640 (LWP 288324)]
[Thread 0x7f0ef67ff640 (LWP 288320) exited]
[New Thread 0x7f0ef67ff640 (LWP 288325)]
[Thread 0x7f0ef4ffc640 (LWP 288322) exited]
[Thread 0x7f0ef57fd640 (LWP 288323) exited]
[New Thread 0x7f0ef57fd640 (LWP 288327)]
[New Thread 0x7f0ef4ffc640 (LWP 288328)]
[Thread 0x7f0ee7fff640 (LWP 288324) exited]
[New Thread 0x7f0ee7fff640 (LWP 288329)]
[Thread 0x7f0ef67ff640 (LWP 288325) exited]
[New Thread 0x7f0ef67ff640 (LWP 288330)]
[Thread 0x7f0ef4ffc640 (LWP 288328) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288332)]
[Thread 0x7f0ef57fd640 (LWP 288327) exited]
[Thread 0x7f0ee7fff640 (LWP 288329) exited]
[New Thread 0x7f0ee7fff640 (LWP 288333)]
[New Thread 0x7f0ef57fd640 (LWP 288334)]
[Thread 0x7f0ef67ff640 (LWP 288330) exited]
[New Thread 0x7f0ef67ff640 (LWP 288336)]
[Thread 0x7f0ef4ffc640 (LWP 288332) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288337)]
[Thread 0x7f0ef57fd640 (LWP 288334) exited]
[New Thread 0x7f0ef57fd640 (LWP 288338)]
[Thread 0x7f0ee7fff640 (LWP 288333) exited]
[New Thread 0x7f0ee7fff640 (LWP 288339)]
[Thread 0x7f0ef67ff640 (LWP 288336) exited]
[New Thread 0x7f0ef67ff640 (LWP 288341)]
[Thread 0x7f0ef57fd640 (LWP 288338) exited]
[New Thread 0x7f0ef57fd640 (LWP 288343)]
[Thread 0x7f0ee7fff640 (LWP 288339) exited]
[Thread 0x7f0ef4ffc640 (LWP 288337) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288344)]
[Thread 0x7f0ef67ff640 (LWP 288341) exited]
[New Thread 0x7f0ee7fff640 (LWP 288345)]
[New Thread 0x7f0ef67ff640 (LWP 288346)]
[Thread 0x7f0ee7fff640 (LWP 288345) exited]
[New Thread 0x7f0ee7fff640 (LWP 288348)]
[Thread 0x7f0ef57fd640 (LWP 288343) exited]
[Thread 0x7f0ef67ff640 (LWP 288346) exited]
[Thread 0x7f0ef4ffc640 (LWP 288344) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288350)]
[New Thread 0x7f0ef67ff640 (LWP 288351)]
[New Thread 0x7f0ef57fd640 (LWP 288352)]
[Thread 0x7f0ee7fff640 (LWP 288348) exited]
[New Thread 0x7f0ee7fff640 (LWP 288354)]
[Thread 0x7f0ef4ffc640 (LWP 288350) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288355)]
[Thread 0x7f0ef67ff640 (LWP 288351) exited]
[Thread 0x7f0ef57fd640 (LWP 288352) exited]
[New Thread 0x7f0ef57fd640 (LWP 288356)]
[New Thread 0x7f0ef67ff640 (LWP 288357)]
[Thread 0x7f0ef4ffc640 (LWP 288355) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288359)]
[Thread 0x7f0ef67ff640 (LWP 288357) exited]
[New Thread 0x7f0ef67ff640 (LWP 288361)]
[Thread 0x7f0ee7fff640 (LWP 288354) exited]
[New Thread 0x7f0ee7fff640 (LWP 288362)]
[Thread 0x7f0ef57fd640 (LWP 288356) exited]
[New Thread 0x7f0ef57fd640 (LWP 288377)]
[Thread 0x7f0ef67ff640 (LWP 288361) exited]
[Thread 0x7f0ef4ffc640 (LWP 288359) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288378)]
[New Thread 0x7f0ef67ff640 (LWP 288379)]
[Thread 0x7f0ee7fff640 (LWP 288362) exited]
[New Thread 0x7f0ee7fff640 (LWP 288381)]
[Thread 0x7f0ef57fd640 (LWP 288377) exited]
[New Thread 0x7f0ef57fd640 (LWP 288383)]
[Thread 0x7f0ef67ff640 (LWP 288379) exited]
[New Thread 0x7f0ef67ff640 (LWP 288384)]
[Thread 0x7f0ef4ffc640 (LWP 288378) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288385)]
[Thread 0x7f0ee7fff640 (LWP 288381) exited]
[New Thread 0x7f0ee7fff640 (LWP 288386)]
[Thread 0x7f0ef57fd640 (LWP 288383) exited]
[New Thread 0x7f0ef57fd640 (LWP 288388)]
[Thread 0x7f0ef67ff640 (LWP 288384) exited]
[New Thread 0x7f0ef67ff640 (LWP 288389)]
[Thread 0x7f0ef4ffc640 (LWP 288385) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288390)]
[Thread 0x7f0ee7fff640 (LWP 288386) exited]
[New Thread 0x7f0ee7fff640 (LWP 288391)]
[Thread 0x7f0ef57fd640 (LWP 288388) exited]
[New Thread 0x7f0ef57fd640 (LWP 288393)]
[Thread 0x7f0ef57fd640 (LWP 288393) exited]
[Thread 0x7f0ee7fff640 (LWP 288391) exited]
[Thread 0x7f0ef67ff640 (LWP 288389) exited]
[New Thread 0x7f0ef67ff640 (LWP 288395)]
[New Thread 0x7f0ee7fff640 (LWP 288396)]
[New Thread 0x7f0ef57fd640 (LWP 288397)]
[Thread 0x7f0ef4ffc640 (LWP 288390) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288398)]
[Thread 0x7f0ef4ffc640 (LWP 288398) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288400)]
[Thread 0x7f0ee7fff640 (LWP 288396) exited]
[Thread 0x7f0ef57fd640 (LWP 288397) exited]
[New Thread 0x7f0ef57fd640 (LWP 288402)]
[New Thread 0x7f0ee7fff640 (LWP 288403)]
[Thread 0x7f0ef67ff640 (LWP 288395) exited]
[New Thread 0x7f0ef67ff640 (LWP 288405)]
[Thread 0x7f0ee7fff640 (LWP 288403) exited]
[New Thread 0x7f0ee7fff640 (LWP 288407)]
[Thread 0x7f0ef67ff640 (LWP 288405) exited]
[New Thread 0x7f0ef67ff640 (LWP 288408)]
[Thread 0x7f0ef57fd640 (LWP 288402) exited]
[New Thread 0x7f0ef57fd640 (LWP 288409)]
[Thread 0x7f0ef4ffc640 (LWP 288400) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288410)]
[Thread 0x7f0ef4ffc640 (LWP 288410) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288412)]
[Thread 0x7f0ef67ff640 (LWP 288408) exited]
[New Thread 0x7f0ef67ff640 (LWP 288414)]
[Thread 0x7f0ef57fd640 (LWP 288409) exited]
[New Thread 0x7f0ef57fd640 (LWP 288415)]
[Thread 0x7f0ee7fff640 (LWP 288407) exited]
[New Thread 0x7f0ee7fff640 (LWP 288417)]
[Thread 0x7f0ef67ff640 (LWP 288414) exited]
[Thread 0x7f0ee7fff640 (LWP 288417) exited]
[New Thread 0x7f0ee7fff640 (LWP 288419)]
[New Thread 0x7f0ef67ff640 (LWP 288420)]
[Thread 0x7f0ef57fd640 (LWP 288415) exited]
[New Thread 0x7f0ef57fd640 (LWP 288421)]
[Thread 0x7f0ef4ffc640 (LWP 288412) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288423)]
[Thread 0x7f0ef67ff640 (LWP 288420) exited]
[New Thread 0x7f0ef67ff640 (LWP 288425)]
[Thread 0x7f0ef57fd640 (LWP 288421) exited]
[New Thread 0x7f0ef57fd640 (LWP 288426)]
[Thread 0x7f0ef4ffc640 (LWP 288423) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288428)]
[Thread 0x7f0ee7fff640 (LWP 288419) exited]
[New Thread 0x7f0ee7fff640 (LWP 288429)]
[Thread 0x7f0ef67ff640 (LWP 288425) exited]
[New Thread 0x7f0ef67ff640 (LWP 288431)]
[Thread 0x7f0ef57fd640 (LWP 288426) exited]
[New Thread 0x7f0ef57fd640 (LWP 288432)]
[Thread 0x7f0ef4ffc640 (LWP 288428) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288433)]
[Thread 0x7f0ee7fff640 (LWP 288429) exited]
[New Thread 0x7f0ee7fff640 (LWP 288435)]
[Thread 0x7f0ef57fd640 (LWP 288432) exited]
[Thread 0x7f0ef67ff640 (LWP 288431) exited]
[New Thread 0x7f0ef67ff640 (LWP 288437)]
[New Thread 0x7f0ef57fd640 (LWP 288438)]
[Thread 0x7f0ef4ffc640 (LWP 288433) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288439)]
[Thread 0x7f0ee7fff640 (LWP 288435) exited]
[New Thread 0x7f0ee7fff640 (LWP 288440)]
[Thread 0x7f0ef57fd640 (LWP 288438) exited]
[New Thread 0x7f0ef57fd640 (LWP 288442)]
[Thread 0x7f0ef4ffc640 (LWP 288439) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288443)]
[Thread 0x7f0ef67ff640 (LWP 288437) exited]
[New Thread 0x7f0ef67ff640 (LWP 288444)]
[Thread 0x7f0ee7fff640 (LWP 288440) exited]
[New Thread 0x7f0ee7fff640 (LWP 288446)]
[Thread 0x7f0ef57fd640 (LWP 288442) exited]
[New Thread 0x7f0ef57fd640 (LWP 288447)]
[Thread 0x7f0ef4ffc640 (LWP 288443) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288449)]
[Thread 0x7f0ef67ff640 (LWP 288444) exited]
[New Thread 0x7f0ef67ff640 (LWP 288450)]
[Thread 0x7f0ee7fff640 (LWP 288446) exited]
[New Thread 0x7f0ee7fff640 (LWP 288452)]
[Thread 0x7f0ef4ffc640 (LWP 288449) exited]
[Thread 0x7f0ef57fd640 (LWP 288447) exited]
[Thread 0x7f0ef67ff640 (LWP 288450) exited]
[New Thread 0x7f0ef67ff640 (LWP 288565)]
[New Thread 0x7f0ef4ffc640 (LWP 288566)]
[New Thread 0x7f0ef57fd640 (LWP 288567)]
[Thread 0x7f0ee7fff640 (LWP 288452) exited]
[New Thread 0x7f0ee7fff640 (LWP 288568)]
[Thread 0x7f0ef4ffc640 (LWP 288566) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288570)]
[Thread 0x7f0ef67ff640 (LWP 288565) exited]
[Thread 0x7f0ef57fd640 (LWP 288567) exited]
[New Thread 0x7f0ef57fd640 (LWP 288572)]
[New Thread 0x7f0ef67ff640 (LWP 288573)]
[Thread 0x7f0ee7fff640 (LWP 288568) exited]
[New Thread 0x7f0ee7fff640 (LWP 288574)]
[Thread 0x7f0ef4ffc640 (LWP 288570) exited]
[Thread 0x7f0ee7fff640 (LWP 288574) exited]
[Thread 0x7f0ef57fd640 (LWP 288572) exited]
[New Thread 0x7f0ef57fd640 (LWP 288576)]
[New Thread 0x7f0ee7fff640 (LWP 288577)]
[New Thread 0x7f0ef4ffc640 (LWP 288578)]
[Thread 0x7f0ef67ff640 (LWP 288573) exited]
[New Thread 0x7f0ef67ff640 (LWP 288579)]
[Thread 0x7f0ee7fff640 (LWP 288577) exited]
[New Thread 0x7f0ee7fff640 (LWP 288581)]
[Thread 0x7f0ef67ff640 (LWP 288579) exited]
[Thread 0x7f0ef57fd640 (LWP 288576) exited]
[New Thread 0x7f0ef57fd640 (LWP 288582)]
[New Thread 0x7f0ef67ff640 (LWP 288583)]
[Thread 0x7f0ef4ffc640 (LWP 288578) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288585)]
[Thread 0x7f0ef67ff640 (LWP 288583) exited]
[Thread 0x7f0ef57fd640 (LWP 288582) exited]
[New Thread 0x7f0ef57fd640 (LWP 288586)]
[New Thread 0x7f0ef67ff640 (LWP 288587)]
[Thread 0x7f0ee7fff640 (LWP 288581) exited]
[Thread 0x7f0ef4ffc640 (LWP 288585) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288589)]
[New Thread 0x7f0ee7fff640 (LWP 288590)]
[Thread 0x7f0ef67ff640 (LWP 288587) exited]
[New Thread 0x7f0ef67ff640 (LWP 288592)]
[Thread 0x7f0ef4ffc640 (LWP 288589) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288594)]
[Thread 0x7f0ee7fff640 (LWP 288590) exited]
[New Thread 0x7f0ee7fff640 (LWP 288595)]
[Thread 0x7f0ef57fd640 (LWP 288586) exited]
[New Thread 0x7f0ef57fd640 (LWP 288596)]
[Thread 0x7f0ef67ff640 (LWP 288592) exited]
[New Thread 0x7f0ef67ff640 (LWP 288598)]
[Thread 0x7f0ee7fff640 (LWP 288595) exited]
[Thread 0x7f0ef4ffc640 (LWP 288594) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288599)]
[New Thread 0x7f0ee7fff640 (LWP 288601)]
[Thread 0x7f0ef57fd640 (LWP 288596) exited]
[New Thread 0x7f0ef57fd640 (LWP 288602)]
[Thread 0x7f0ef4ffc640 (LWP 288599) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288603)]
[Thread 0x7f0ef67ff640 (LWP 288598) exited]
[Thread 0x7f0ee7fff640 (LWP 288601) exited]
[New Thread 0x7f0ee7fff640 (LWP 288605)]
[New Thread 0x7f0ef67ff640 (LWP 288606)]
[Thread 0x7f0ef57fd640 (LWP 288602) exited]
[New Thread 0x7f0ef57fd640 (LWP 288607)]
[Thread 0x7f0ef4ffc640 (LWP 288603) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288609)]
[Thread 0x7f0ef67ff640 (LWP 288606) exited]
[Thread 0x7f0ee7fff640 (LWP 288605) exited]
[New Thread 0x7f0ee7fff640 (LWP 288610)]
[New Thread 0x7f0ef67ff640 (LWP 288611)]
[Thread 0x7f0ef57fd640 (LWP 288607) exited]
[New Thread 0x7f0ef57fd640 (LWP 288613)]
[Thread 0x7f0ef67ff640 (LWP 288611) exited]
[Thread 0x7f0ef4ffc640 (LWP 288609) exited]
[Thread 0x7f0ee7fff640 (LWP 288610) exited]
[New Thread 0x7f0ee7fff640 (LWP 288615)]
[New Thread 0x7f0ef4ffc640 (LWP 288616)]
[New Thread 0x7f0ef67ff640 (LWP 288617)]
[Thread 0x7f0ef57fd640 (LWP 288613) exited]
[New Thread 0x7f0ef57fd640 (LWP 288619)]
[Thread 0x7f0ef4ffc640 (LWP 288616) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288621)]
[Thread 0x7f0ef67ff640 (LWP 288617) exited]
[New Thread 0x7f0ef67ff640 (LWP 288622)]
[Thread 0x7f0ee7fff640 (LWP 288615) exited]
[New Thread 0x7f0ee7fff640 (LWP 288623)]
[Thread 0x7f0ef57fd640 (LWP 288619) exited]
[New Thread 0x7f0ef57fd640 (LWP 288624)]
[Thread 0x7f0ee7fff640 (LWP 288623) exited]
[New Thread 0x7f0ee7fff640 (LWP 288627)]
[Thread 0x7f0ef67ff640 (LWP 288622) exited]
[New Thread 0x7f0ef67ff640 (LWP 288628)]
[Thread 0x7f0ef57fd640 (LWP 288624) exited]
[New Thread 0x7f0ef57fd640 (LWP 288629)]
[Thread 0x7f0ef4ffc640 (LWP 288621) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288631)]
[Thread 0x7f0ef67ff640 (LWP 288628) exited]
[New Thread 0x7f0ef67ff640 (LWP 288633)]
[Thread 0x7f0ee7fff640 (LWP 288627) exited]
[Thread 0x7f0ef57fd640 (LWP 288629) exited]
[New Thread 0x7f0ef57fd640 (LWP 288634)]
[New Thread 0x7f0ee7fff640 (LWP 288635)]
[Thread 0x7f0ef4ffc640 (LWP 288631) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288637)]
[Thread 0x7f0ef67ff640 (LWP 288633) exited]
[New Thread 0x7f0ef67ff640 (LWP 288638)]
[Thread 0x7f0ef57fd640 (LWP 288634) exited]
[New Thread 0x7f0ef57fd640 (LWP 288639)]
[Thread 0x7f0ee7fff640 (LWP 288635) exited]
[New Thread 0x7f0ee7fff640 (LWP 288640)]
[Thread 0x7f0ef67ff640 (LWP 288638) exited]
[Thread 0x7f0ef4ffc640 (LWP 288637) exited]
[New Thread 0x7f0ef4ffc640 (LWP 288655)]
[New Thread 0x7f0ef67ff640 (LWP 288656)]
[Thread 0x7f0ef57fd640 (LWP 288639) exited]
[New Thread 0x7f0ef57fd640 (LWP 288657)]
[Thread 0x7f0ee7fff640 (LWP 288640) exited]
[New Thread 0x7f0ee7fff640 (LWP 288658)]

Thread 182 "spine" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f0ef67ff640 (LWP 288656)]
0x00007f0ef72cae03 in mthd_my_send_cmd () from /lib64/libmariadb.so.3
#0  0x00007f0ef72cae03 in mthd_my_send_cmd () from /lib64/libmariadb.so.3
#1  0x00007f0ef72c2fa3 in mysql_real_query () from /lib64/libmariadb.so.3
#2  0x0000000000403aea in db_insert (mysql=0x7f0ef67d97b0, type=0, 
    query=0x7f0ee001ef10 "INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (425, 'traffic_out', FROM_UNIXTIME(1756069490), '13826098751182') ON DUPLICATE KEY UPDATE output=VALUES(output)") at sql.c:63
#3  0x0000000000417167 in poll_host (device_counter=45, host_id=16, host_thread=2, host_threads=4, host_data_ids=1, 
    host_time=0x7f0ef67fed00 "1756069490", host_errors=0x7f0ef67fee0c, host_time_double=1756069490.7839191)
    at poller.c:1872
#4  0x000000000040ecf6 in child (arg=0x20342b0) at poller.c:108
#5  0x00007f0ef688a19a in start_thread () from /lib64/libc.so.6
#6  0x00007f0ef690f210 in clone3 () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.34-168.el9_6.20.x86_64 libgcc-11.5.0-5.el9_5.x86_64 mariadb-connector-c-3.2.6-1.el9_0.x86_64 net-snmp-libs-5.9.1-17.el9.x86_64 openssl-libs-3.2.2-6.el9_5.1.x86_64 sssd-client-2.9.6-4.el9_6.2.x86_64 zlib-1.2.11-40.el9.x86_64
�[?2004hA debugging session is active.

	Inferior 1 [process 288250] will be detached.

But I have also seen this segfault (that seems to come from another function...)

Thread 98 "spine" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f525e1fb640 (LWP 339443)]
0x00007f5260299e69 in malloc () from /lib64/libc.so.6
#0  0x00007f5260299e69 in malloc () from /lib64/libc.so.6
#1  0x00007f526029c6cf in strdup () from /lib64/libc.so.6
#2  0x00007f5260cf16a5 in snmp_build () from /lib64/libnetsnmp.so.40
#3  0x00007f5260cf503a in netsnmp_build_packet () from /lib64/libnetsnmp.so.40
#4  0x00007f5260cf52c4 in _build_initial_pdu_packet () from /lib64/libnetsnmp.so.40
#5  0x00007f5260cf6043 in snmp_sess_async_send () from /lib64/libnetsnmp.so.40
#6  0x00007f5260cd271e in snmp_sess_synch_response () from /lib64/libnetsnmp.so.40
#7  0x000000000040ccd9 in snmp_get_base (current_host=0x7f52580008d0, snmp_oid=0x7f5258006820 ".1.3.6.1.2.1.1.3.0",
    should_fail=true) at snmp.c:458
#8  0x000000000040d3d4 in snmp_get (current_host=0x7f52580008d0, snmp_oid=0x7f5258006820 ".1.3.6.1.2.1.1.3.0")
    at snmp.c:564
#9  0x000000000041a56a in ping_snmp (host=0x7f52580008d0, ping=0x7f52580021a0) at ping.c:188
#10 0x000000000041a283 in ping_host (host=0x7f52580008d0, ping=0x7f52580021a0) at ping.c:108
#11 0x000000000041082d in poll_host (device_counter=24, host_id=52, host_thread=1, host_threads=4, host_data_ids=2,
    host_time=0x7f525e1fad00 "1756070492", host_errors=0x7f525e1fae0c, host_time_double=1756070492.515074)
    at poller.c:738
#12 0x000000000040ecf6 in child (arg=0xe78230) at poller.c:108
#13 0x00007f526028a19a in start_thread () from /lib64/libc.so.6
#14 0x00007f526030f210 in clone3 () from /lib64/libc.so.6

[TheWitness]

Very interesting. The second issue appears to maybe be in the library. This is the 1.2.x branch correct? Also, how many database connections are you setup to use?

Console > Data Collection > Pollers > SUM(Processes + Threads)

Also attach a screen shot of:

Console > System Utilities > Technical Support. Should look as follows:

[earendilfr]

For the branch, yes, I'm on branch 1.2.x at commit af323e4

I have configured for a maximum of 200 connections for DB.

For the poller, I have this configured: 1 proc and 4 threads

Here the screenshot for the support page:

[TheWitness]

Go through the same steps here.

Try running with -R -V 3 -S with and without. See if there is a difference. Send the bt outcome.

e4c62fd

[TheWitness]

Also, run the collector with just one thread, and let's see if it's one data source that is causing the problem, and what data source it is:

gdb ./spine 
run -R -V 3 -S -t 1
<segfault>
bt

[TheWitness]

One more thing. Since some of the leak issues I found via the valgrind utility are in the MariaDB/MySQL library, let's do this:

rpm -qa | egrep -i '(mariadb|mysql)'
mysql --version

[TheWitness]

Here is what I get. Latest MariaDB-server LTS.

[root@vmhost5 spine]# rpm -qa | egrep -i '(mariadb|mysql)'
perl-DBD-MariaDB-1.21-16.el9_0.x86_64
rsyslog-mysql-8.2412.0-1.el9.x86_64
php-mysqlnd-8.0.30-3.el9_6.x86_64
MariaDB-shared-11.8.3-1.el9.x86_64
MariaDB-common-11.8.3-1.el9.x86_64
MariaDB-client-compat-11.8.3-1.el9.noarch
MariaDB-client-11.8.3-1.el9.x86_64
MariaDB-server-compat-11.8.3-1.el9.noarch
MariaDB-server-11.8.3-1.el9.x86_64
MariaDB-backup-11.8.3-1.el9.x86_64
MariaDB-devel-11.8.3-1.el9.x86_64
[root@vmhost5 spine]# mysql --version
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
mysql from 11.8.3-MariaDB, client 15.2 for Linux (x86_64) using  EditLine wrapper

[earendilfr]

run -R -V 3 -S -t 1

With just this command, I have this Segfault (seems not exactly the same...)

[New Thread 0x7f6cfcdff640 (LWP 369566)]
Total[9.9708] SNMP: Using privacy protocol(len): AES(64)
Total[9.9883] Device[27] INFO: SNMP Device Alive, Time:0.0053 ms
Total[9.9908] Device[27] HT[4] DQ[4] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 8 = output: 8)
Total[9.9911] Device[27] HT[4] NOTE: There are '1' Polling Items for this Device
Total[10.0028] SNMP: Using privacy protocol(len): AES(64)
Total[10.0095] ERROR: Device[43] HT[1] Spine Timed Out While Processing Devices External
Total[10.0095] ERROR: Polling timed out while waiting for 1 Threads to End
Total[10.0095] WARNING: There were 23 threads which did not run
Total[10.0236] Device[27] HT[4] DS[484] TT[32.44] SNMP: v3: pspe-pae-ert02.net.fhvi.ch, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.8, value: 3956942440
Total[10.0236] Device[27] HT[4] Total Time: 0.065 Seconds
                                                                                                                                                                                                                                            Thread 133 "spine" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f6cfcdff640 (LWP 369566)]
db_release_connection (type=0, id=29987504) at sql.c:516
516                     db_pool_local[id].free = TRUE;
Missing separate debuginfos, use: dnf debuginfo-install libgcc-11.5.0-5.el9_5.x86_64 sssd-client-2.9.6-4.el9_6.2.x86_64
(gdb) bt                                                                                                                                                                                                                                    #0  db_release_connection (type=0, id=29987504) at sql.c:516
#1  0x000000000041751b in poll_host (device_counter=32, host_id=27, host_thread=4, host_threads=4, host_data_ids=1, host_time=0x7f6cfcdfed00 "1756211970", host_errors=0x7f6cfcdfee0c, host_time_double=1756211970.75126) at poller.c:1934
#2  0x000000000040ecf6 in child (arg=0x1ce98c0) at poller.c:108
#3  0x00007f6cfce8a19a in start_thread () from /lib64/libc.so.6
#4  0x00007f6cfcf0f210 in clone3 () from /lib64/libc.so.6

Concerning the MySQL version,

[root@smgtcacti11 14:40:59 bin]# rpm -qa | egrep -i '(mariadb|mysql)'
mariadb-connector-c-config-3.2.6-1.el9_0.noarch
mariadb-connector-c-3.2.6-1.el9_0.x86_64
mariadb-common-10.5.27-1.el9_5.0.2.x86_64
perl-DBD-MariaDB-1.21-16.el9_0.x86_64
mariadb-errmsg-10.5.27-1.el9_5.0.2.x86_64
mysql-selinux-1.0.13-1.el9_5.noarch
mariadb-backup-10.5.27-1.el9_5.0.2.x86_64
mariadb-gssapi-server-10.5.27-1.el9_5.0.2.x86_64
mariadb-server-utils-10.5.27-1.el9_5.0.2.x86_64
mariadb-server-10.5.27-1.el9_5.0.2.x86_64
mariadb-10.5.27-1.el9_5.0.2.x86_64
php-mysqlnd-8.0.30-3.el9_6.x86_64
mariadb-connector-c-doc-3.2.6-1.el9_0.noarch
mariadb-connector-c-devel-3.2.6-1.el9_0.x86_64
mariadb-devel-10.5.27-1.el9_5.0.2.x86_64
mysqltuner-1.8.3-2.git.1333ea9.el9.noarch
[root@smgtcacti11 14:41:01 bin]# mysql --version
mysql  Ver 15.1 Distrib 10.5.27-MariaDB, for Linux (x86_64) using  EditLine wrapper

[TheWitness]

Okay, so you are using SNMPv3 devices. This is good. The next step is to install valgrind and run a test with it.

yum -y install valgrind
valgrind --leak-check=full --show-leak-kinds=all ./spine -R -V 3 -S -t 1

Make sure you are running the version that I provided in the hash above though. If you get a core file, you should zip it up and upload here if it's allowed.