+ /bin/bash -x /tmp/tmp.Wu7rGugDif + SCRIPT_PID=3200274 + set +x + unset _mlshdbg + '[' 1 = 1 ']' + case "$-" in + set +x + unset _mlshdbg + set -x + cd /ccc/scratch/cont005/gen7298/couletpa/AMZ_Para/v20_2014_2023/2019_NOATM + module purge + local _mlredir=0 + '[' -n '' ']' + case " $@ " in + '[' 0 -eq 0 ']' + _module_raw purge + unset _mlshdbg + '[' 1 = 1 ']' + case "$-" in + set +x unload module nco/5.0.1 (NCO) unload module eccodes/2.23.0 (ecCodes GRIB encoding) unload module libaec/1.1.2 (libaec) unload module jasper/2.0.14 (jasper) unload module netcdf-fortran/4.5.3 (NetCDF Fortran HDF5 serial flavor) unload module netcdf-c/4.7.4 (NetCDF-CNetCDF-C HDF5 hdf5__serial flavor) unload module hdf5/1.12.0 (HDF5) unload module intel/20.0.0 (Intel Compiler Suite) unload module mkl/20.0.0 (Intel MKL LP64 Sequential without mpi interfaces) unload module feature/mkl/vector/amd (MKL vectorization feature) unload module feature/mkl/single_node (MKL without mpi interface) unload module feature/mkl/sequential (MKL feature) unload module feature/mkl/lp64 (MKL feature) unload module fortran/intel/20.0.0 (Intel Fortran compiler) unload module c/intel/20.0.0 (Intel C Compiler) unload module c++/intel/20.0.0 (Intel C++ Compiler) unload module licsrv/intel (License service) unload module flavor/buildcompiler/intel/20 (Compiler build flavor) unload module flavor/hdf5/serial (HDF5 flavor) module dfldatadir/gen7298 (Data Directory) cannot be unloaded Unloading datadir/gen7298 ERROR: Dependent dfldatadir/gen7298 is loaded Unloading ccc/1.0 ERROR: Dependent datadir/gen7298 and dfldatadir/gen7298 are loaded + unset _mlshdbg + return 1 + module load pnetcdf/1.12.1 + local _mlredir=0 + '[' -n '' ']' + case " $@ " in + '[' 0 -eq 0 ']' + _module_raw load pnetcdf/1.12.1 + unset _mlshdbg + '[' 1 = 1 ']' + case "$-" in + set +x load module flavor/buildmpi/openmpi/4 (MPI build flavor) load module feature/mkl/single_node (MKL without mpi interface) load module flavor/buildcompiler/intel/20 (Compiler build flavor) load module feature/openmpi/io/standard (MPI-IO feature) load module feature/openmpi/mpi_compiler/intel (MPI Compiler feature) load module feature/openmpi/net/auto (MPI Network backend feature) load module flavor/ucx/standard (ucx flavor) load module flavor/cuda/nvhpc-222 (Cuda flavor) load module cuda/11.6 (NVIDIA CUDA) load module flavor/libccc_user/hwloc2 (libccc_user flavor) load module hwloc/2.5.0 (Hwloc) load module pmix/4.2.2 (Process Management Interface (PMI) for eXascale) load module licsrv/intel (License service) load module c++/intel/20.0.0 (Intel C++ Compiler) load module c/intel/20.0.0 (Intel C Compiler) load module fortran/intel/20.0.0 (Intel Fortran compiler) load module feature/hcoll/multicast/enable (Hcoll features) load module sharp/2.4.2 (Mellanox backend) load module hcoll/4.7.3191 (Mellanox hcoll) load module ucx/1.12.1 (Mellanox backend) load module feature/mkl/lp64 (MKL feature) load module feature/mkl/sequential (MKL feature) load module feature/mkl/vector/amd (MKL vectorization feature) load module mkl/20.0.0 (Intel MKL LP64 Sequential without mpi interfaces) load module intel/20.0.0 (Intel Compiler Suite) load module mpi/openmpi/4.1.4 (Open MPI) load module pnetcdf/1.12.1 (PNetCDF) + unset _mlshdbg + return 0 + module load netcdf-fortran/4.5.3 + local _mlredir=0 + '[' -n '' ']' + case " $@ " in + '[' 0 -eq 0 ']' + _module_raw load netcdf-fortran/4.5.3 + unset _mlshdbg + '[' 1 = 1 ']' + case "$-" in + set +x load module flavor/hdf5/serial (HDF5 flavor) load module hdf5/1.12.0 (HDF5) load module netcdf-c/4.7.4 (NetCDF-CNetCDF-C HDF5 hdf5__serial flavor) load module netcdf-fortran/4.5.3 (NetCDF Fortran HDF5 serial flavor) + unset _mlshdbg + return 0 + cp /ccc/work/cont005/gen7298/couletpa/schism_5_10_v4/schism/build/bin/pschism_TVD-VL . + ccc_mprun ./pschism_TVD-VL 2 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 3983 in communicator MPI COMMUNICATOR 5 DUP FROM 0 srun: Job step aborted: Waiting up to 302 seconds for job step to finish. with errorcode 0. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- slurmstepd-irene1124: error: *** STEP 4915481.0 ON irene1124 CANCELLED AT 2024-05-13T18:28:32 *** slurmstepd-irene1777: error: mpi/pmix_v4: _errhandler: irene1777 [82]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3983] slurmstepd-irene1784: error: mpi/pmix_v4: _errhandler: irene1784 [89]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4272] slurmstepd-irene1202: error: mpi/pmix_v4: _errhandler: irene1202 [13]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:625] slurmstepd-irene1808: error: mpi/pmix_v4: _errhandler: irene1808 [95]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4568] slurmstepd-irene1780: error: mpi/pmix_v4: _errhandler: irene1780 [85]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4081] slurmstepd-irene1819: error: mpi/pmix_v4: _errhandler: irene1819 [99]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4753] slurmstepd-irene1762: error: mpi/pmix_v4: _errhandler: irene1762 [67]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3216] slurmstepd-irene1617: error: mpi/pmix_v4: _errhandler: irene1617 [40]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1920] slurmstepd-irene1782: error: mpi/pmix_v4: _errhandler: irene1782 [87]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4178] slurmstepd-irene1203: error: mpi/pmix_v4: _errhandler: irene1203 [14]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:672] slurmstepd-irene1786: error: mpi/pmix_v4: _errhandler: irene1786 [91]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4368] slurmstepd-irene1773: error: mpi/pmix_v4: _errhandler: irene1773 [78]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3745] slurmstepd-irene1816: error: mpi/pmix_v4: _errhandler: irene1816 [97]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4657] slurmstepd-irene1130: error: mpi/pmix_v4: _errhandler: irene1130 [1]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:48] slurmstepd-irene1214: error: mpi/pmix_v4: _errhandler: irene1214 [19]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:913] slurmstepd-irene1781: error: mpi/pmix_v4: _errhandler: irene1781 [86]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4155] slurmstepd-irene1656: error: mpi/pmix_v4: _errhandler: irene1656 [48]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2304] slurmstepd-irene1554: error: mpi/pmix_v4: _errhandler: irene1554 [28]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1345] slurmstepd-irene1754: error: mpi/pmix_v4: _errhandler: irene1754 [59]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2835] slurmstepd-irene1818: error: mpi/pmix_v4: _errhandler: irene1818 [98]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4704] slurmstepd-irene1788: error: mpi/pmix_v4: _errhandler: irene1788 [93]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4464] slurmstepd-irene1813: error: mpi/pmix_v4: _errhandler: irene1813 [96]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4609] slurmstepd-irene1783: error: mpi/pmix_v4: _errhandler: irene1783 [88]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4224] slurmstepd-irene1619: error: mpi/pmix_v4: _errhandler: irene1619 [42]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2016] slurmstepd-irene1787: error: mpi/pmix_v4: _errhandler: irene1787 [92]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4449] slurmstepd-irene1216: error: mpi/pmix_v4: _errhandler: irene1216 [21]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1009] slurmstepd-irene1752: error: mpi/pmix_v4: _errhandler: irene1752 [57]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2737] slurmstepd-irene1767: error: mpi/pmix_v4: _errhandler: irene1767 [72]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3456] slurmstepd-irene1196: error: mpi/pmix_v4: _errhandler: irene1196 [8]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:384] slurmstepd-irene1771: error: mpi/pmix_v4: _errhandler: irene1771 [76]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3648] slurmstepd-irene1785: error: mpi/pmix_v4: _errhandler: irene1785 [90]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4321] slurmstepd-irene1215: error: mpi/pmix_v4: _errhandler: irene1215 [20]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:960] slurmstepd-irene1789: error: mpi/pmix_v4: _errhandler: irene1789 [94]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4558] slurmstepd-irene1194: error: mpi/pmix_v4: _errhandler: irene1194 [6]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:288] slurmstepd-irene1654: error: mpi/pmix_v4: _errhandler: irene1654 [46]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2210] slurmstepd-irene1205: error: mpi/pmix_v4: _errhandler: irene1205 [16]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:768] slurmstepd-irene1556: error: mpi/pmix_v4: _errhandler: irene1556 [30]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1441] slurmstepd-irene1195: error: mpi/pmix_v4: _errhandler: irene1195 [7]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:336] slurmstepd-irene1750: error: mpi/pmix_v4: _errhandler: irene1750 [55]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2640] slurmstepd-irene1218: error: mpi/pmix_v4: _errhandler: irene1218 [23]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1104] slurmstepd-irene1768: error: mpi/pmix_v4: _errhandler: irene1768 [73]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3520] slurmstepd-irene1766: error: mpi/pmix_v4: _errhandler: irene1766 [71]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3453] slurmstepd-irene1778: error: mpi/pmix_v4: _errhandler: irene1778 [83]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3985] slurmstepd-irene1178: error: mpi/pmix_v4: _errhandler: irene1178 [4]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:192] slurmstepd-irene1757: error: mpi/pmix_v4: _errhandler: irene1757 [62]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2977] slurmstepd-irene1199: error: mpi/pmix_v4: _errhandler: irene1199 [10]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:481] slurmstepd-irene1613: error: mpi/pmix_v4: _errhandler: irene1613 [36]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1729] slurmstepd-irene1197: error: mpi/pmix_v4: _errhandler: irene1197 [9]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:474] slurmstepd-irene1748: error: mpi/pmix_v4: _errhandler: irene1748 [53]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2544] slurmstepd-irene1200: error: mpi/pmix_v4: _errhandler: irene1200 [11]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:529] slurmstepd-irene1772: error: mpi/pmix_v4: _errhandler: irene1772 [77]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3696] slurmstepd-irene1760: error: mpi/pmix_v4: _errhandler: irene1760 [65]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3120] slurmstepd-irene1552: error: mpi/pmix_v4: _errhandler: irene1552 [26]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1248] slurmstepd-irene1156: error: mpi/pmix_v4: _errhandler: irene1156 [3]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:144] slurmstepd-irene1775: error: mpi/pmix_v4: _errhandler: irene1775 [80]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3847] slurmstepd-irene1765: error: mpi/pmix_v4: _errhandler: irene1765 [70]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3361] slurmstepd-irene1764: error: mpi/pmix_v4: _errhandler: irene1764 [69]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3313] slurmstepd-irene1744: error: mpi/pmix_v4: _errhandler: irene1744 [50]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2415] slurmstepd-irene1770: error: mpi/pmix_v4: _errhandler: irene1770 [75]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3601] slurmstepd-irene1201: error: mpi/pmix_v4: _errhandler: irene1201 [12]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:577] slurmstepd-irene1207: error: mpi/pmix_v4: _errhandler: irene1207 [18]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:894] slurmstepd-irene1758: error: mpi/pmix_v4: _errhandler: irene1758 [63]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3024] slurmstepd-irene1563: error: mpi/pmix_v4: _errhandler: irene1563 [32]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1537] slurmstepd-irene1769: error: mpi/pmix_v4: _errhandler: irene1769 [74]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3552] slurmstepd-irene1747: error: mpi/pmix_v4: _errhandler: irene1747 [52]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2496] slurmstepd-irene1751: error: mpi/pmix_v4: _errhandler: irene1751 [56]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2689] slurmstepd-irene1616: error: mpi/pmix_v4: _errhandler: irene1616 [39]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1872] slurmstepd-irene1611: error: mpi/pmix_v4: _errhandler: irene1611 [34]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1633] slurmstepd-irene1776: error: mpi/pmix_v4: _errhandler: irene1776 [81]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3891] slurmstepd-irene1759: error: mpi/pmix_v4: _errhandler: irene1759 [64]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3073] slurmstepd-irene1555: error: mpi/pmix_v4: _errhandler: irene1555 [29]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1394] slurmstepd-irene1779: error: mpi/pmix_v4: _errhandler: irene1779 [84]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:4046] slurmstepd-irene1657: error: mpi/pmix_v4: _errhandler: irene1657 [49]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2354] slurmstepd-irene1553: error: mpi/pmix_v4: _errhandler: irene1553 [27]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1297] slurmstepd-irene1155: error: mpi/pmix_v4: _errhandler: irene1155 [2]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:96] slurmstepd-irene1614: error: mpi/pmix_v4: _errhandler: irene1614 [37]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1778] slurmstepd-irene1746: error: mpi/pmix_v4: _errhandler: irene1746 [51]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2449] slurmstepd-irene1749: error: mpi/pmix_v4: _errhandler: irene1749 [54]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2593] slurmstepd-irene1615: error: mpi/pmix_v4: _errhandler: irene1615 [38]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1826] slurmstepd-irene1204: error: mpi/pmix_v4: _errhandler: irene1204 [15]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:720] slurmstepd-irene1226: error: mpi/pmix_v4: _errhandler: irene1226 [24]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1152] slurmstepd-irene1761: error: mpi/pmix_v4: _errhandler: irene1761 [66]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3168] slurmstepd-irene1206: error: mpi/pmix_v4: _errhandler: irene1206 [17]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:836] slurmstepd-irene1124: error: mpi/pmix_v4: _errhandler: irene1124 [0]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1] slurmstepd-irene1227: error: mpi/pmix_v4: _errhandler: irene1227 [25]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1202] slurmstepd-irene1566: error: mpi/pmix_v4: _errhandler: irene1566 [33]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1585] slurmstepd-irene1755: error: mpi/pmix_v4: _errhandler: irene1755 [60]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2880] slurmstepd-irene1753: error: mpi/pmix_v4: _errhandler: irene1753 [58]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2784] slurmstepd-irene1763: error: mpi/pmix_v4: _errhandler: irene1763 [68]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3266] slurmstepd-irene1620: error: mpi/pmix_v4: _errhandler: irene1620 [43]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2064] slurmstepd-irene1756: error: mpi/pmix_v4: _errhandler: irene1756 [61]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2954] slurmstepd-irene1217: error: mpi/pmix_v4: _errhandler: irene1217 [22]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1056] slurmstepd-irene1618: error: mpi/pmix_v4: _errhandler: irene1618 [41]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1968] slurmstepd-irene1612: error: mpi/pmix_v4: _errhandler: irene1612 [35]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1681] slurmstepd-irene1653: error: mpi/pmix_v4: _errhandler: irene1653 [45]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2160] slurmstepd-irene1652: error: mpi/pmix_v4: _errhandler: irene1652 [44]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2112] slurmstepd-irene1655: error: mpi/pmix_v4: _errhandler: irene1655 [47]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:2269] slurmstepd-irene1774: error: mpi/pmix_v4: _errhandler: irene1774 [79]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:3793] slurmstepd-irene1557: error: mpi/pmix_v4: _errhandler: irene1557 [31]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:1489] slurmstepd-irene1193: error: mpi/pmix_v4: _errhandler: irene1193 [5]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4915481.0:240] + exit 0