Poorly formed xml from MG5

[cspollard]

Madgraph sometimes writes out strange characters to the lhe file. For example, here's how vim displays a recently-produced file. I think it's safe to ignore these lines in the parsing. Currently they cause parsing to fail.

^[[1;31m#*                                                          *^[[1;0m
^[[1;31m#*          WARNING: UNKNOWN DEVELOPMENT VERSION.           *^[[1;0m
^[[1;31m#*            WARNING: DO NOT USE FOR PRODUCTION            *^[[1;0m
^[[1;31m#*                                                          *^[[1;0m

I could probably easily write a PR for this fix if others agree.

[APN-Pucky]

Hi @cspollard can you send the failing file here? We have a set of madgraph files that work fine (see https://github.com/scikit-hep/scikit-hep-testdata/tree/main/src/skhep_testdata/data pylhe-testfile-madgraph*.lhe).

I could probably easily write a PR for this fix if others agree.

You are very welcome to do so. Otherwise I can also do that once I have the file.

[APN-Pucky]

Ah, and ideally the file would be not too big so we can add it to our tests, to make sure everything works fine in the future too.

[APN-Pucky]

Also, if you want to look have a look at it it probably makes sense to not start before tomorrow, since I aim at merging #321 which streamlines and simplifies the parsing a bit.

[cspollard]

ok can you let me know when the relevant MR is merged, then I can give it a go. I can also make a small test file, but I suspect any non-release version of MG will create files like this.

[APN-Pucky]

>>> pylhe.LHEFile.fromfile("unweighted_events.lhe.gz")
WARNING. Parse Error: not well-formed (invalid token): line 48, column 0

I don't see an option to tell the parser to ignore that token, also I don't think patching pylhe to filter these characters is a reasonable way to fix it. Instead I think it makes more sense to change madgraphs output. Hence I created mg5amcnlo/mg5amcnlo#200

[eduardo-rodrigues]

Hi both. I have a bit of sympathy for the issue. In a way our package should not fall apart. But on the other hand invalid XML should not be in the files. That's a corrupted input.

I think the right thing is 2-fold:

• Discuss with the authors as @APN-Pucky did.
• Maybe add a try/except in reading and issue a warning message about invalid input XML upon reading. If you both agree, feel free @cspollard to suggest an enhancement.

(BTW, @cspollard, if you like/use the package, giving us a star - it not done already - does help us in the longer term ;-).)

[APN-Pucky]

ok can you let me know when the relevant MR is merged

I merged it yesterday so feel free to go ahead. I can think of either filter the input stream/buffer or moving the try-exception block closer to the loop such that one might be able to continue ignore the error as suggested by @eduardo-rodrigues.

[APN-Pucky]

@cspollard upstream madgraph sets some git infos now without the failing characters per mg5amcnlo/mg5amcnlo@6be5f19, however it would still be good if we could ignore the characters and continue parsing (just in case).

[APN-Pucky]

This kind of better error handling could be achieved with lxml instead of current pyhton etree.

https://lxml.de/api/lxml.etree.iterparse-class.html

Gives the option recover=True. Otherwise lxml is quite transparent wrt. the cpython ET (see #329).

EDIT: A quick test did not make it work. Instead the whole broken <init> block is skipped then instead.

[cspollard]

Hi. Thanks for all the quick acction! If MG now ensures to output valid xml, then maybe this is redundant?

[eduardo-rodrigues]

Hi. Shame the investigation with lxml did not work. On general grounds I would say it would still be useful to have a way to issue warnings on strange/spurious comment lines in the files and continue gracefully. I can see a use case. This being said, I went the opposite way with DecayLanguage, which complains about buggy files - a useful way in which I did find issues in several experiment files over the years ;-).

[cspollard]

I tend to agree: if the inputs are out of bounds, fail early and loudly. :)