#47056: nco/ncks and netcdf -------------------------------+-------------------------------- Reporter: barry.j.mcinnes@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.3 Keywords: nco netcdf | Port: nco netcdf -------------------------------+-------------------------------- The current nco and netcdf on macports, currently take more than 2 hours (killed), whereas the versions on linux for the same files and command take 13 seconds. A similar nccopy command takes 13 seconds on the same mac for the same files. The linux setup is /usr/local/bin/ncks -r NCO netCDF Operators version "4.4.5" last modified 2014/07/15 built Aug 29 2014 on linux64.psd.esrl.noaa.gov by root ncks version 4.4.5 Linked to netCDF library version 4.3.3, compiled Feb 24 2015 07:21:57 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits The macports is ncks -r NCO netCDF Operators version "4.4.7" last modified 2014/11/05 built Jan 6 2015 on mac73.psd.esrl.noaa.gov by root ncks version 4.4.7 Linked to netCDF library version 4.3.2, compiled Feb 3 2015 08:18:38 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits The command is ncks -D 3 --fl_fmt=netcdf4_classic -L 5 --overwrite huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc /Projects/CMIP5A//Monthly/historical/ACCESS1-0//r1i1p1//huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc The file size is 200MB. Maybe an update to netcdf 4.3.3 will speed things up ? Maybe its the chunking size the macports cannot handle ? One thing I note is that the chunck size for the ncks for the huss variable on Mac ends up being: 1872, 145, 192 but using nccopy, it's 1, 145, 192 which is more in line with what I would expect. -- Ticket URL: <https://trac.macports.org/ticket/47056> MacPorts <https://www.macports.org/> Ports system for OS X