grib2: NetcdfFiles.openInMemory does not work when there is no physical file

[michaeldiener]

Versions impacted by the bug

v5.7.0, maint-5.x

What went wrong?

The current grib parsing always expects a physical file to read/write index files. So the name supplied in NetcdfFiles.openInMemory has to be a physical file. Then in the code the file gets read from that physical location, although a byte[] was supplied.

1. So in case we have a physical file, this is inefficient (because the file is re-read).
2. And in the case the file exists only in memory and not physically, there will be an exception.

Here a small code example to provoke the exception:
byte[] data = ... // load grib2 file from somewhere
NetcdfFile ds = NetcdfFiles.openInMemory("xxx", data)

This should work without the need for a physical file. And in case the is one anyway, it should not be re-read.

Relevant stack trace

No records found in dataset xxx
java.lang.IllegalStateException: No records found in dataset xxx
	at ucar.nc2.grib.collection.Grib2CollectionBuilder.makeGroups(Grib2CollectionBuilder.java:149)
	at ucar.nc2.grib.collection.GribCollectionBuilder.createMultipleRuntimeCollections(GribCollectionBuilder.java:128)
	at ucar.nc2.grib.collection.GribCollectionBuilder.createIndex(GribCollectionBuilder.java:120)
	at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromDataFile(GribCdmIndex.java:825)
	at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromDataFile(GribCdmIndex.java:804)
	at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromRaf(GribCdmIndex.java:774)
	at ucar.nc2.grib.collection.GribIosp.build(GribIosp.java:150)
	at ucar.nc2.NetcdfFiles.build(NetcdfFiles.java:836)
	at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:771)
	at ucar.nc2.NetcdfFiles.openInMemory(NetcdfFiles.java:719)

Relevant log messages

No response

If you have an example file that you can share, please attach it to this issue.

If so, may we include it in our test datasets to help ensure the bug does not return once fixed?
Note: the test datasets are publicly accessible without restriction.

N/A

Code of Conduct

• I agree to follow the UCAR/Unidata Code of Conduct

[lesserwhirls]

See #638. The main issue is that the GRIB IOSP does not interact much with the original source of the GRIB records outside of reading from the data section, but rather relies on one or more index files. Some progress has been made on dealing with index files for non-filesystem based sources, such as GRIB files stored on S3, but it's still a work in progress and I am not up-to-speed on those additions yet.