std::bad_alloc core dump example

[jsta]

I'm trying to extract a raster from the gdb located at:

https://gisdata.mn.gov/dataset/water-lake-bathymetry

• ftp://ftp.gisdata.mn.gov/pub/gdrs/data/pub/us_mn_state_dnr/water_lake_bathymetry/fgdb_water_lake_bathymetry.zip

Running into an error (copied below) that I thought I'd share:

arc_raster.exe data/mn_bathy/water_lake_bathymetry.gdb/ 0 lake_bathymetric_elevation_model

arc_raster.exe data/mn_bathy/water_lake_bathymetry.gdb/ 0 lake_bathymetric_elevation_model
Arc Raster Rescue (url=github.com/r-barnes/ArcRasterRescue, hash=2f4140e9d209355e, compiled=2019-08-23 12:48:01 UTC)
Allocating 4x104576x117376 = 49098850304
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)

[Dave-Evans]

Hi @jsta did you end up figuring this out? I don't know anything about C++ but I was assuming that this meant that the file was too large to fit in RAM. Do you know if this is not the issue?

Thanks,
Dave

[r-barnes]

@jsta: That's a 49GB allocation, so it's not surprising it would fail. Are those dimensions (104576x117376) cells correct?

[jsta]

I'm getting slightly under those numbers but pretty close. It is quite large.

[r-barnes]

@jsta : Given that the file is only 1.1GB, I'm struggling to see how the dataset itself could be that large.

[jsta]

It's 1.7 GB when extracted as a tif. I don't know if it'll help but here is the gdalinfo readout:

gdalinfo lake_bathymetric_elevation_model.tif

Driver: GTiff/GeoTIFF
Files: lake_bathymetric_elevation_model.tif
Size is 104468, 117233
Coordinate System is:
PROJCRS["NAD_1983_UTM_Zone_15N",
    BASEGEOGCRS["NAD83",
        DATUM["North American Datum 1983",
            ELLIPSOID["GRS 1980",6378137,298.257222101004,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4269]],
    CONVERSION["UTM zone 15N",
        METHOD["Transverse Mercator",
            ID["EPSG",9807]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",-93,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["Scale factor at natural origin",0.9996,
            SCALEUNIT["unity",1],
            ID["EPSG",8805]],
        PARAMETER["False easting",500000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["easting",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    ID["EPSG",26915]]
Data axis to CRS axis mapping: 1,2
Origin = (199105.000000000000000,5403520.000000000000000)
Pixel Size = (5.000000000000000,-5.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  199105.000, 5403520.000) ( 97d 5'26.65"W, 48d42'43.25"N)
Lower Left  (  199105.000, 4817355.000) ( 96d43' 6.99"W, 43d26'55.03"N)
Upper Right (  721445.000, 5403520.000) ( 89d59'15.24"W, 48d44'43.05"N)
Lower Right (  721445.000, 4817355.000) ( 90d15'43.10"W, 43d28'34.80"N)
Center      (  460275.000, 5110437.500) ( 93d30'51.79"W, 46d 8'46.88"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
  NoData Value=-3.40282306073709653e+38
  Unit Type: metre
  Metadata:
    RepresentationType=ATHEMATIC

[r-barnes]

Thanks, @jsta. Does the raster contain large areas of no_data or with pixels that are all the same value? (Does it look like it would compress nicely?)

[andersmd7]

I'm encountering a similar issue trying to rescue a File Geodatabase raster from https://www.fs.usda.gov/rds/archive/products/RDS-2019-0030/RDS-2019-0030.zip (U.S. Forest Service Riparian Areas layer).

Allocating 1x462083x286682 = 132470878606
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted

This is a national (U.S.) layer, at 10m resolution (though w/ only 2 integer values, and lots of NoData, so the geodatabase is only around 2.15 GB, though an uncompressed version would be much larger. Does ArcRasterRescue have a method for handling rasters that are too large to fit in RAM?