`append` performance with lots of arguments

[pauldraper]

The append action has O(n^2) performance, due to copying the list for every item.

argparse/argparse.js

Lines 1559 to 1560 in a645a9a

	items = _copy_items(items)
	items.push(values)

[puzrin]

It it a problem? What is the real use case to solve?

[pauldraper]

A command line with several thousand arguments. (Can be really long if @flagfile is used.)

./compiler.js --src=src1.js --src=src2.js --src=src3.js ... output 

For years, I've been curious why a tool I had was unusually slow to start, now I know.

[puzrin]

I understand it's technically possible to pass multiple thousands of params. Just interesting, where is this useful in the real world?

[rlidwka]

5000 arguments are parsed in 1 second for me... not ideal, but not horrible; python is 2x faster strangely enough, but complexity is the same.

python impl is literally the same with the same complexity:
https://github.com/python/cpython/blob/195d13c85e17ab5cf6ac2a6de098bbf514cae207/Lib/argparse.py#L1099-L1100

and that code has already turned 16 years old for sure, git blame goes all the way to svn import

[rlidwka]

Copying is required there because otherwise it's going to modify default args. Here's a case where it matters:

import { ArgumentParser } from 'npm:argparse';

let parser = new ArgumentParser({
  description: 'Count append-type CLI flags',
});

parser.add_argument('-a', '--append', {
  action: 'append',
  default: ["opt1", "opt2"]
});

let args = parser.parse_args();

let count = Array.isArray(args.append) ? args.append.length : 0;

console.log(`Number of flags passed: ${count}`);
console.log('Parsed values:', args);

args = parser.parse_args();

count = Array.isArray(args.append) ? args.append.length : 0;

console.log(`Number of flags passed: ${count}`);
console.log('Parsed values:', args);

[rlidwka]

copying the list for every item

I've measured it, and on 10k arguments (3 second runtime) copying the list of every item takes below 100ms (3% of the execution time)

so if there's a performance issue, I believe you need to search for it elsewhere

[pauldraper]

copying the list of every item takes below 100ms

The performance problem definitely appears.

const { ArgumentParser } = require("argparse");

const parser = new ArgumentParser({
  description: "Test",
  fromfile_prefix_chars: "@",
});

parser.add_argument("--file", { action: "append" });

const start = performance.now();
const args = parser.parse_args();
const end = performance.now();

console.log(`Arg parsing: ${(end - start).toFixed(1)}ms`);

seq 1 10000 | sed s/^/--file=/ > flags
node test.js @flags

Arg parsing: 2495.4ms

[pauldraper]

I understand it's technically possible to pass multiple thousands of params. Just interesting, where is this useful in the real world?

For example, let's say you wanted to list source files with the most lines.

git ls-files | xargs wc -l | grep -v ' total' | sort -h | tail -n10

If you have 5k files, that is 5k arguments to wc.

(Granted, this particular case uses positional args.....but the point is thousands of args is not unusual.)

[pauldraper]

Workaround, custom action.

const { Action } = require("argparse");

class AppendAction extends Action {
    constructor(options) {
        super(options);
        this.default = options.default;
    }
    call(_, namespace, values) {
        let items = namespace[this.dest];
        if (items === this.default) {
            items = this.default ? [...this.default] : [];
            namespace[this.dest] = items;
        }
        items.push(values);
    }
}