[MacPorts] #47056: nco/ncks and netcdf
#47056: nco/ncks and netcdf -------------------------------+-------------------------------- Reporter: barry.j.mcinnes@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.3 Keywords: nco netcdf | Port: nco netcdf -------------------------------+-------------------------------- The current nco and netcdf on macports, currently take more than 2 hours (killed), whereas the versions on linux for the same files and command take 13 seconds. A similar nccopy command takes 13 seconds on the same mac for the same files. The linux setup is /usr/local/bin/ncks -r NCO netCDF Operators version "4.4.5" last modified 2014/07/15 built Aug 29 2014 on linux64.psd.esrl.noaa.gov by root ncks version 4.4.5 Linked to netCDF library version 4.3.3, compiled Feb 24 2015 07:21:57 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits The macports is ncks -r NCO netCDF Operators version "4.4.7" last modified 2014/11/05 built Jan 6 2015 on mac73.psd.esrl.noaa.gov by root ncks version 4.4.7 Linked to netCDF library version 4.3.2, compiled Feb 3 2015 08:18:38 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits The command is ncks -D 3 --fl_fmt=netcdf4_classic -L 5 --overwrite huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc /Projects/CMIP5A//Monthly/historical/ACCESS1-0//r1i1p1//huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc The file size is 200MB. Maybe an update to netcdf 4.3.3 will speed things up ? Maybe its the chunking size the macports cannot handle ? One thing I note is that the chunck size for the ncks for the huss variable on Mac ends up being: 1872, 145, 192 but using nccopy, it's 1, 145, 192 which is more in line with what I would expect. -- Ticket URL: <https://trac.macports.org/ticket/47056> MacPorts <https://www.macports.org/> Ports system for OS X
#47056: nco/ncks and netcdf --------------------------------+-------------------------------- Reporter: barry.j.mcinnes@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.3 Resolution: | Keywords: nco netcdf Port: nco netcdf | --------------------------------+-------------------------------- Comment (by takeshi@…): netcdf has been updated to 4.3.3.1 in r133622 (See #47040). nco has been updated to 4.4.8 in r133623. Do these updates help in anyway? -- Ticket URL: <https://trac.macports.org/ticket/47056#comment:1> MacPorts <https://www.macports.org/> Ports system for OS X
#47056: nco/ncks and netcdf --------------------------------+----------------------- Reporter: barry.j.mcinnes@… | Owner: takeshi@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.3 Resolution: | Keywords: Port: nco netcdf | --------------------------------+----------------------- Changes (by ryandesign@…): * owner: macports-tickets@… => takeshi@… * keywords: nco netcdf => * cc: takeshi@… (removed) Old description:
The current nco and netcdf on macports, currently take more than 2 hours (killed), whereas the versions on linux for the same files and command take 13 seconds. A similar nccopy command takes 13 seconds on the same mac for the same files.
The linux setup is /usr/local/bin/ncks -r NCO netCDF Operators version "4.4.5" last modified 2014/07/15 built Aug 29 2014 on linux64.psd.esrl.noaa.gov by root ncks version 4.4.5 Linked to netCDF library version 4.3.3, compiled Feb 24 2015 07:21:57 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits
The macports is ncks -r NCO netCDF Operators version "4.4.7" last modified 2014/11/05 built Jan 6 2015 on mac73.psd.esrl.noaa.gov by root ncks version 4.4.7 Linked to netCDF library version 4.3.2, compiled Feb 3 2015 08:18:38 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits
The command is ncks -D 3 --fl_fmt=netcdf4_classic -L 5 --overwrite huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc /Projects/CMIP5A//Monthly/historical/ACCESS1-0//r1i1p1//huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc
The file size is 200MB. Maybe an update to netcdf 4.3.3 will speed things up ? Maybe its the chunking size the macports cannot handle ? One thing I note is that the chunck size for the ncks for the huss variable on Mac ends up being: 1872, 145, 192 but using nccopy, it's 1, 145, 192 which is more in line with what I would expect.
New description: The current nco and netcdf on macports, currently take more than 2 hours (killed), whereas the versions on linux for the same files and command take 13 seconds. A similar nccopy command takes 13 seconds on the same mac for the same files. The linux setup is {{{ /usr/local/bin/ncks -r NCO netCDF Operators version "4.4.5" last modified 2014/07/15 built Aug 29 2014 on linux64.psd.esrl.noaa.gov by root ncks version 4.4.5 Linked to netCDF library version 4.3.3, compiled Feb 24 2015 07:21:57 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits }}} The macports is {{{ ncks -r NCO netCDF Operators version "4.4.7" last modified 2014/11/05 built Jan 6 2015 on mac73.psd.esrl.noaa.gov by root ncks version 4.4.7 Linked to netCDF library version 4.3.2, compiled Feb 3 2015 08:18:38 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: http://www.gnu.org/copyleft/gpl.html LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: http://nco.sf.net User Guide: http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes http://nco.sf.net/nco.html#mss_val Check missing_value No http://nco.sf.net/nco.html#mss_val DAP clients Yes http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl HDF4 support Unknown http://nco.sf.net/nco.html#hdf4 Internationalization No http://nco.sf.net/nco.html#i18n (pre- alpha) MPI parallelization No http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre- alpha) Regular Expressions Yes http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits }}} The command is {{{ ncks -D 3 --fl_fmt=netcdf4_classic -L 5 --overwrite huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc /Projects/CMIP5A//Monthly/historical/ACCESS1-0//r1i1p1//huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc }}} The file size is 200MB. Maybe an update to netcdf 4.3.3 will speed things up ? Maybe its the chunking size the macports cannot handle ? One thing I note is that the chunck size for the ncks for the huss variable on Mac ends up being: 1872, 145, 192 but using nccopy, it's 1, 145, 192 which is more in line with what I would expect. -- -- Ticket URL: <https://trac.macports.org/ticket/47056#comment:2> MacPorts <https://www.macports.org/> Ports system for OS X
#47056: nco/ncks and netcdf --------------------------------+----------------------- Reporter: barry.j.mcinnes@… | Owner: takeshi@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.3 Resolution: | Keywords: Port: nco netcdf | --------------------------------+----------------------- Comment (by barry.j.mcinnes@…): Thanks for the updates - after many tests on linux and macs it seems the chunking behavior since 4.4.5 is causing the problem. To get it to convert in 10 seconds in 4.4.7 and 4.4.8 we have to add --cnk_plc=r1d otherwise ncks runs for hours. In 4.4.5 it runs as expected without the extra parameter. We have logged it with no. Thanks for your help and the updates, which can be used now. -- Ticket URL: <https://trac.macports.org/ticket/47056#comment:3> MacPorts <https://www.macports.org/> Ports system for OS X
participants (1)
-
MacPorts