How to delete a logger instance to free memory, and kill worker threads

[moda20]

Hi I am trying to create multiple pino instances and then delete them after finishing using them.
I am using workers to log messages as that drives well with my use case. I recently found that most pino instances, after creation, cannot be freed from memory even when deleted and the created threads don't seem to close.

Here is an example of my use case, it's not a real world use case but shows the issue best

import http from "http"
import pino from "pino"

const start = async () => {

    const server = http.createServer((_req, res) => {
        res.writeHead(200, { "Content-Type": "text/plain" })
        res.end("Server is running\n")
    })

    server.listen(3000, () => {
        console.log("HTTP server running on http://localhost:3000")
    })

    const loggers: pino.Logger[] = []

    for (let i = 0; i < 500; i++) {
        const transport = pino.transport({
            targets: [
                {
                    target: "pino-pretty",
                    level: "info",
                    options: {
                        destination: 1,
                        colorize: true,
                        ignore: "pid,hostname",
                        sync: true,
                    },
                }
            ]
        })
        const logger = pino(
            transport
        )
        logger.info(`Logger ${i} initialized`)
        loggers.push(logger)
    }

    const ll = loggers.length
    for (let i = 0; i < ll; i++) {
        loggers[i].flush(()=> { // the flush di not prove to be useful in reducing memory usage or thread count
            delete loggers[i]
            console.log("logger flushed")
        });
        await new Promise((res) => setTimeout(() => res(), 250)); // this is not useful in any way 

    }

};

start()

It's an HTTP server with 500 pino instance loggers being created that use the worker method, they also use pino-pretty as a large enough transport. Running this will take about 5Gb of RAM and will spawn around 1k+ threads. under no circumstances the memory will be freed.

I am using the latest pino@9.7 and pino-pretty@13.0 and on macOs (although the issue was seen on a linux server as well).

Note : I read about the child loggers, but in my current use-case my transports include different files and the child loggers don't allow for that kind of alteration

Note : I am running this via Bun

[mcollina]

You can call .end() on the transport once you are done with it, it should close the thread.

[moda20]

@mcollina thanks for the response, calling .end() on the transport seems to do the job, although it wasn't as clear in the docs.
a quick question, is there a way to call .end() on all transports from the pino instance instead of trying to keep a reference to the transports ?

sorry for the edit, I am trying to have a full back and forth starting with creating pino loggers to then completely deleting them from memory and effectively going back to the http server only memory usage (about 20Mb), After calling .end() on all transports (via the Threaded stream from the pino instance), i am still up about 350Mb of ram usage that theoretically shouldn't be there.

this is the latest code i am using :

import http from "http"
import pino from "pino"
import {writeHeapSnapshot} from "v8";

const start = async () => {

    const server = http.createServer((_req, res) => {
        res.writeHead(200, { "Content-Type": "text/plain" })
        res.end("Server is running\n")
    })

    server.listen(3000, () => {
        console.log("HTTP server running on http://localhost:3000")
    })

    const loggers: (pino.Logger|null)[] = []

    for (let i = 0; i < 100; i++) { // upped the instances to 100
        const transport = pino.transport({
            targets: [
                {
                    target: "pino-pretty",
                    level: "info",
                    options: {
                        destination: 1,
                        colorize: true,
                        ignore: "pid,hostname",
                        sync: true,
                    },
                }
            ]
        })
        const logger = pino(
            transport
        )
        loggers.push(logger)
    }

    for (let i = 0; i < 100; i++) {
        const symbols = Object.getOwnPropertySymbols(loggers[i]);
        const targetSymbol = symbols.find(e => e.toString() === "Symbol(pino.stream)"); // i use the symbol for access since the threadedStream is not directly available 
        loggers[i][targetSymbol]?.end()
        delete loggers[i];
    }
    loggers.length = 0;

    writeHeapSnapshot("my-app.heapsnapshot"); // <-- the heapshot used to get the memory content. 

};

start()

this is what the memory allocation looks like, There is clearly a lot of 4MB objects that are related to the logger instances, that are simply lingering on, they represent the [Symbol(kImpl)] from the ThreadStream object seen when logged, isn't calling the end() supposed to delete these ?