#47056: nco/ncks and netcdf --------------------------------+----------------------- Reporter: barry.j.mcinnes@… | Owner: takeshi@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.3 Resolution: | Keywords: Port: nco netcdf | --------------------------------+----------------------- Changes (by ryandesign@…): * owner: macports-tickets@… => takeshi@… * keywords: nco netcdf => * cc: takeshi@… (removed) Old description:
The current nco and netcdf on macports, currently take more than 2 hours (killed), whereas the versions on linux for the same files and command take 13 seconds. A similar nccopy command takes 13 seconds on the same mac for the same files.
The linux setup is /usr/local/bin/ncks -r NCO netCDF Operators version "4.4.5" last modified 2014/07/15 built Aug 29 2014 on linux64.psd.esrl.noaa.gov by root ncks version 4.4.5 Linked to netCDF library version 4.3.3, compiled Feb 24 2015 07:21:57 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits
The macports is ncks -r NCO netCDF Operators version "4.4.7" last modified 2014/11/05 built Jan 6 2015 on mac73.psd.esrl.noaa.gov by root ncks version 4.4.7 Linked to netCDF library version 4.3.2, compiled Feb 3 2015 08:18:38 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits
The command is ncks -D 3 --fl_fmt=netcdf4_classic -L 5 --overwrite huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc /Projects/CMIP5A//Monthly/historical/ACCESS1-0//r1i1p1//huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc
The file size is 200MB. Maybe an update to netcdf 4.3.3 will speed things up ? Maybe its the chunking size the macports cannot handle ? One thing I note is that the chunck size for the ncks for the huss variable on Mac ends up being: 1872, 145, 192 but using nccopy, it's 1, 145, 192 which is more in line with what I would expect.
New description: The current nco and netcdf on macports, currently take more than 2 hours (killed), whereas the versions on linux for the same files and command take 13 seconds. A similar nccopy command takes 13 seconds on the same mac for the same files. The linux setup is {{{ /usr/local/bin/ncks -r NCO netCDF Operators version "4.4.5" last modified 2014/07/15 built Aug 29 2014 on linux64.psd.esrl.noaa.gov by root ncks version 4.4.5 Linked to netCDF library version 4.3.3, compiled Feb 24 2015 07:21:57 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits }}} The macports is {{{ ncks -r NCO netCDF Operators version "4.4.7" last modified 2014/11/05 built Jan 6 2015 on mac73.psd.esrl.noaa.gov by root ncks version 4.4.7 Linked to netCDF library version 4.3.2, compiled Feb 3 2015 08:18:38 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits }}} The command is {{{ ncks -D 3 --fl_fmt=netcdf4_classic -L 5 --overwrite huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc /Projects/CMIP5A//Monthly/historical/ACCESS1-0//r1i1p1//huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc }}} The file size is 200MB. Maybe an update to netcdf 4.3.3 will speed things up ? Maybe its the chunking size the macports cannot handle ? One thing I note is that the chunck size for the ncks for the huss variable on Mac ends up being: 1872, 145, 192 but using nccopy, it's 1, 145, 192 which is more in line with what I would expect. -- -- Ticket URL: <https://trac.macports.org/ticket/47056#comment:2> MacPorts <https://www.macports.org/> Ports system for OS X