Slower nc_open with version 4.9.3 #3183
Description
Slower nc_open with version 4.9.3
Hello, thanks for your work on this wonderful package, and for the heroic efforts getting 4.9.3 onto conda-forge 😊
The nc_open() function looks to be significantly slower in version 4.9.3 versus 4.9.2.
Recorded times on my machine
- Small dummy file (see CDL below):
400microseconds slower;2100versus1700 A1B_north_america.nc:5milliseconds slower;15versus10
dummy.nc CDL
Expand for CDL
netcdf simple_example {
dimensions:
x = 10;
y = 10;
variables:
float data(x, y);
data:
data = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
51, 52, 53, 54, 55, 56, 57, 58, 59, 60,
61, 62, 63, 64, 65, 66, 67, 68, 69, 70,
71, 72, 73, 74, 75, 76, 77, 78, 79, 80,
81, 82, 83, 84, 85, 86, 87, 88, 89, 90,
91, 92, 93, 94, 95, 96, 97, 98, 99, 100 ;
}
Why these small slowdowns still matter
SciTools/iris provides parallel computation+streaming of NetCDF files in distributed compute setups. This has seen successful operational use for several years (ping @bouweandela, @valeriupredoi). For this use-case, the small slowdowns seen here scale to multiple seconds for parallel computation of 1000 chunks or more.
Explanation
The distributed compute nodes must not simultaneously open a single NetCDF file for writing, since that causes concurrency errors at the HDF5 level. Such errors can be prevented with appropriate use of locking, and ensuring that each parallel task opens and closes the NetCDF file independently - i.e. the file must not be left open after a parallel task completes. Thus, 1000 chunks = 1000 calls to nc_open() = 1000x slowdown.
I have tried to mitigate this slowdown by having tasks 'share' an already open file handle (with careful locking), but I am not aware1 of a way to achieve this 'sharing' between distributed compute nodes.
Info for reproducing
NetCDF config
Expand for `nc-config --all` output
This netCDF 4.9.3 has been built with the following features:
--cc -> x86_64-conda-linux-gnu-cc
--cflags -> -I/home/users/martin.yeo/.conda/envs/tmp_libnetcdf_493/include
--libs -> -L/home/users/martin.yeo/.conda/envs/tmp_libnetcdf_493/lib -lnetcdf
--static -> -ljpeg -ldf -lmfhdf -lHDF5::HDF5 -lhdf5::hdf5_hl -lm -lz -lzip -lblosc -lzstd -lbz2 -lsz -lCURL::libcurl -lxml2
--has-dap -> yes
--has-dap2 -> yes
--has-dap4 -> yes
--has-nc2 -> yes
--has-nc4 -> yes
--has-hdf5 -> yes
--has-hdf4 -> yes
--has-logging -> no
--has-pnetcdf -> no
--has-szlib -> yes
--has-cdf5 -> yes
--has-parallel4 -> no
--has-parallel -> no
--has-nczarr -> yes
--has-zstd -> yes
--has-benchmarks -> yes
--has-multifilters -> yes
--has-stdfilters -> deflate szip blosc zstd bz2
--has-quantize -> yes
--prefix -> /home/users/martin.yeo/.conda/envs/tmp_libnetcdf_493
--includedir -> /home/users/martin.yeo/.conda/envs/tmp_libnetcdf_493/include
--libdir -> /home/users/martin.yeo/.conda/envs/tmp_libnetcdf_493/lib
--plugindir -> /home/users/martin.yeo/.conda/envs/tmp_libnetcdf_493/hdf5/lib/plugin
--plugin-searchpath -> /home/users/martin.yeo/.conda/envs/tmp_libnetcdf_493/hdf5/lib/plugin:/usr/local/hdf5/lib/plugin
--version -> netCDF 4.9.3
--build-system -> cmake
Environment
OS: RHEL9
gcc info: gcc (conda-forge gcc 15.2.0-7) 15.2.0
Expand for Conda environment spec
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
attr 2.5.2 h39aace5_0 conda-forge
binutils_impl_linux-64 2.44 hdf8817f_2 conda-forge
blosc 1.21.6 he440d0b_1 conda-forge
bzip2 1.0.8 hda65f42_8 conda-forge
c-ares 1.34.5 hb9d3cd8_0 conda-forge
ca-certificates 2025.10.5 hbd8a1cb_0 conda-forge
conda-gcc-specs 15.2.0 h56430cd_7 conda-forge
gcc 15.2.0 hc115cf6_7 conda-forge
gcc_impl_linux-64 15.2.0 hcacfade_7 conda-forge
hdf4 4.2.15 h2a13503_7 conda-forge
hdf5 1.14.6 nompi_h6e4c0c1_103 conda-forge
icu 75.1 he02047a_0 conda-forge
kernel-headers_linux-64 5.14.0 he073ed8_2 conda-forge
keyutils 1.6.3 hb9d3cd8_0 conda-forge
krb5 1.21.3 h659f571_0 conda-forge
ld_impl_linux-64 2.44 ha97dd6f_2 conda-forge
libaec 1.1.4 h3f801dc_0 conda-forge
libcurl 8.14.1 h332b0f4_0 conda-forge
libedit 3.1.20250104 pl5321h7949ede_0 conda-forge
libev 4.33 hd590300_2 conda-forge
libgcc 15.2.0 h767d61c_7 conda-forge
libgcc-devel_linux-64 15.2.0 h73f6952_107 conda-forge
libgcc-ng 15.2.0 h69a702a_7 conda-forge
libgfortran 15.2.0 h69a702a_7 conda-forge
libgfortran5 15.2.0 hcd61629_7 conda-forge
libgomp 15.2.0 h767d61c_7 conda-forge
libiconv 1.18 h3b78370_2 conda-forge
libjpeg-turbo 3.1.0 hb9d3cd8_0 conda-forge
liblzma 5.8.1 hb9d3cd8_2 conda-forge
libnetcdf 4.9.3 nompi_h11f7409_103 conda-forge
libnghttp2 1.67.0 had1ee68_0 conda-forge
libsanitizer 15.2.0 hb13aed2_7 conda-forge
libssh2 1.11.1 hcf80075_0 conda-forge
libstdcxx 15.2.0 h8f9b012_7 conda-forge
libstdcxx-ng 15.2.0 h4852527_7 conda-forge
libxml2 2.15.0 h26afc86_1 conda-forge
libxml2-16 2.15.0 ha9997c6_1 conda-forge
libzip 1.11.2 h6991a6a_0 conda-forge
libzlib 1.3.1 hb9d3cd8_2 conda-forge
lz4-c 1.10.0 h5888daf_1 conda-forge
ncurses 6.5 h2d0b736_3 conda-forge
openssl 3.5.4 h26f9b46_0 conda-forge
snappy 1.2.2 h03e3b7b_0 conda-forge
sysroot_linux-64 2.34 h087de78_2 conda-forge
tzdata 2025b h78e105d_0 conda-forge
zlib 1.3.1 hb9d3cd8_2 conda-forge
zstd 1.5.7 hb8e6e7a_2 conda-forge
Code
(With help from Copilot as my C skills are limited).
#include <netcdf.h>
#include <stdio.h>
#include <time.h>
#define N_RUNS 25
double time_nc_open(const char *filename) {
struct timespec start, end;
int ncid, retval;
clock_gettime(CLOCK_MONOTONIC, &start);
retval = nc_open(filename, NC_NOWRITE, &ncid);
if (retval != NC_NOERR) {
fprintf(stderr, "nc_open error: %s\n", nc_strerror(retval));
return -1;
}
nc_close(ncid);
clock_gettime(CLOCK_MONOTONIC, &end);
return (end.tv_sec - start.tv_sec) + (end.tv_nsec - start.tv_nsec) / 1e6;
}
int main(int argc, char **argv) {
if (argc != 2) {
printf("Usage: %s <netcdf_file>\n", argv[0]);
return 1;
}
double total = 0;
int errors = 0;
for (int i = 0; i < N_RUNS; ++i) {
double t = time_nc_open(argv[1]);
if (t < 0) {
errors++;
continue;
}
total += t;
}
printf("Average nc_open time: %.2f milliseconds over %d runs\n", total / (N_RUNS - errors), N_RUNS - errors);
if (errors > 0) printf("nc_open failed %d times\n", errors);
return 0;
}Compiling
gcc -o ncopen_bench ncopen_bench.c -lnetcdf
Footnotes
-
We are working at the Python level. ↩