JHove extracts nonsense CJK characters instead of title, other metadata from some PDFs

[dmolesUC]

Steps to reproduce:

1. download PDFs such as the attached Goldberg_gwu_0075M_10226.pdf (or any PDF you find by searching the Internet for the string きどっひはびはてぴ; you'll find a large number of PDF documents in institutional repositories that use FITS/Jhove for characterization, probably mostly via Hyrax/Samvera)

2. run jhove on them

Expected behavior:

• Jhove reports correct title, in this case Microsoft Word - Speech-Language Pathologists’ Self-Reported Definition of Cluttering and Confidence in Assessment of Cluttering

Actual behavior:

• Jhove reports title as a CJK nonsense string beginning with きどっひはびはてぴ〠しはひつ〠〭〠

(Note that exiftool reports the title correctly.)

Notes:

Similar behavior seems to happen with PDFs produced in different ways and with different issues. For instance, 4912.pdf.pdf clearly has some other things going on—macOS Preview gives the title as Microsoft Word - Dissertation_Master_6_12-21-11.doc, while exiftool (possibly reading RDF from embedded <x:xmpmeta>?) gives title, creator, and producer as (different) strings of escape codes. Jhove, however, gives the title as a (different) CJK nonsense string with the same initial きどっひはびはてぴ〠しはひつ〠〭〠, and also gives author and producer as CJK nonsense strings, while macOS Preview and exiftool both give the author as Sam.

Tested with jhove 1.32.1 on macOS, as well as 1.26.1 via the FITSservlet Docker image.

[ross-spencer]

Surprised this hasn't come up before @dmolesUC -- this type of mojibake text usually indicates something going wrong with the decoding of UTF-16.

Sure enough, the title does seem to be UTF-16BE -- you can see the bytes FEFF (UTF-16 byte-order-mark) at positions 9 and 10 here.

2f 54 69 74 6c 65 20 28 fe ff 5c 30 30 30 4d 5c  
30 30 30 69 5c 30 30 30 63 5c 30 30 30 72 5c 30  
30 30 6f 5c 30 30 30 73 5c 30 30 30 6f 5c 30 30  
30 66 5c 30 30 30 74 5c 30 30 30 20 5c 30 30 30  
57 5c 30 30 30 6f 5c 30 30 30 72 5c 30 30 30 64

And you can see double the bytes being used for the title, which begins as you say, "Microsoft Word".

/Title (..\000M\000i\000c\000r\000o\000s\000o\000f\000t\000 \000W\000o\000r\000d

According to some forums it has affected different tools/PDF handlers differently depending on what they're expecting.

I wouldn't dare to begin to guess why JHOVE isn't handling this here though.

[dmolesUC]

@ross-spencer good eye—I figured it was something like that but hadn't spent enough time looking at Unicode bytes to spot just what was going on.

[karenhanson]

Might this issue be related? #880

[carlwilson]

First to say thanks for reporting this @dmolesUC , it seems to have sparked enough input to merit investigation, and it might see off an old bug or two. I've added it to the next release stack.