+ SCRIPT_PID=390690 + /bin/bash -x /tmp/tmp.0GCXiSroV4 + set +x + unset _mlshdbg + '[' 1 = 1 ']' + case "$-" in + set +x + unset _mlshdbg + set -x + cd /ccc/scratch/cont005/gen7298/couletpa/AMZ_Para/v20_2014_2023/2019_REF + module purge + local _mlredir=0 + '[' -n '' ']' + case " $@ " in + '[' 0 -eq 0 ']' + _module_raw purge + unset _mlshdbg + '[' 1 = 1 ']' + case "$-" in + set +x unload module nco/5.0.1 (NCO) unload module eccodes/2.23.0 (ecCodes GRIB encoding) unload module libaec/1.1.2 (libaec) unload module jasper/2.0.14 (jasper) unload module netcdf-fortran/4.5.3 (NetCDF Fortran HDF5 serial flavor) unload module netcdf-c/4.7.4 (NetCDF-CNetCDF-C HDF5 hdf5__serial flavor) unload module hdf5/1.12.0 (HDF5) unload module intel/20.0.0 (Intel Compiler Suite) unload module mkl/20.0.0 (Intel MKL LP64 Sequential without mpi interfaces) unload module feature/mkl/vector/amd (MKL vectorization feature) unload module feature/mkl/single_node (MKL without mpi interface) unload module feature/mkl/sequential (MKL feature) unload module feature/mkl/lp64 (MKL feature) unload module fortran/intel/20.0.0 (Intel Fortran compiler) unload module c/intel/20.0.0 (Intel C Compiler) unload module c++/intel/20.0.0 (Intel C++ Compiler) unload module licsrv/intel (License service) unload module flavor/buildcompiler/intel/20 (Compiler build flavor) unload module flavor/hdf5/serial (HDF5 flavor) module dfldatadir/gen7298 (Data Directory) cannot be unloaded Unloading datadir/gen7298 ERROR: Dependent dfldatadir/gen7298 is loaded Unloading ccc/1.0 ERROR: Dependent datadir/gen7298 and dfldatadir/gen7298 are loaded + unset _mlshdbg + return 1 + module load pnetcdf/1.12.1 + local _mlredir=0 + '[' -n '' ']' + case " $@ " in + '[' 0 -eq 0 ']' + _module_raw load pnetcdf/1.12.1 + unset _mlshdbg + '[' 1 = 1 ']' + case "$-" in + set +x load module flavor/buildmpi/openmpi/4 (MPI build flavor) load module feature/mkl/single_node (MKL without mpi interface) load module flavor/buildcompiler/intel/20 (Compiler build flavor) load module feature/openmpi/io/standard (MPI-IO feature) load module feature/openmpi/mpi_compiler/intel (MPI Compiler feature) load module feature/openmpi/net/auto (MPI Network backend feature) load module flavor/ucx/standard (ucx flavor) load module flavor/cuda/nvhpc-222 (Cuda flavor) load module cuda/11.6 (NVIDIA CUDA) load module flavor/libccc_user/hwloc2 (libccc_user flavor) load module hwloc/2.5.0 (Hwloc) load module pmix/4.2.2 (Process Management Interface (PMI) for eXascale) load module licsrv/intel (License service) load module c++/intel/20.0.0 (Intel C++ Compiler) load module c/intel/20.0.0 (Intel C Compiler) load module fortran/intel/20.0.0 (Intel Fortran compiler) load module feature/hcoll/multicast/enable (Hcoll features) load module sharp/2.4.2 (Mellanox backend) load module hcoll/4.7.3191 (Mellanox hcoll) load module ucx/1.12.1 (Mellanox backend) load module feature/mkl/lp64 (MKL feature) load module feature/mkl/sequential (MKL feature) load module feature/mkl/vector/amd (MKL vectorization feature) load module mkl/20.0.0 (Intel MKL LP64 Sequential without mpi interfaces) load module intel/20.0.0 (Intel Compiler Suite) load module mpi/openmpi/4.1.4 (Open MPI) load module pnetcdf/1.12.1 (PNetCDF) + unset _mlshdbg + return 0 + module load netcdf-fortran/4.5.3 + local _mlredir=0 + '[' -n '' ']' + case " $@ " in + '[' 0 -eq 0 ']' + _module_raw load netcdf-fortran/4.5.3 + unset _mlshdbg + '[' 1 = 1 ']' + case "$-" in + set +x load module flavor/hdf5/serial (HDF5 flavor) load module hdf5/1.12.0 (HDF5) load module netcdf-c/4.7.4 (NetCDF-CNetCDF-C HDF5 hdf5__serial flavor) load module netcdf-fortran/4.5.3 (NetCDF Fortran HDF5 serial flavor) + unset _mlshdbg + return 0 + cp /ccc/work/cont005/gen7298/couletpa/schism_5_10_v4/schism/build/bin/pschism_TVD-VL . + ccc_mprun ./pschism_TVD-VL 2 -------------------------------------------------------------------------- srun: Job step aborted: Waiting up to 302 seconds for job step to finish. MPI_ABORT was invoked on rank 3983 in communicator MPI COMMUNICATOR 5 DUP FROM 0 with errorcode 0. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- slurmstepd-irene1415: error: *** STEP 4914725.0 ON irene1415 CANCELLED AT 2024-05-13T17:28:07 *** slurmstepd-irene2503: error: mpi/pmix_v4: _errhandler: irene2503 [86]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4128] slurmstepd-irene2511: error: mpi/pmix_v4: _errhandler: irene2511 [91]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4368] slurmstepd-irene2492: error: mpi/pmix_v4: _errhandler: irene2492 [82]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3983] slurmstepd-irene1536: error: mpi/pmix_v4: _errhandler: irene1536 [23]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1142] slurmstepd-irene2516: error: mpi/pmix_v4: _errhandler: irene2516 [94]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4514] slurmstepd-irene2507: error: mpi/pmix_v4: _errhandler: irene2507 [90]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4336] slurmstepd-irene2501: error: mpi/pmix_v4: _errhandler: irene2501 [84]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4032] slurmstepd-irene2528: error: mpi/pmix_v4: _errhandler: irene2528 [99]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4752] slurmstepd-irene1521: error: mpi/pmix_v4: _errhandler: irene1521 [11]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:575] slurmstepd-irene2432: error: mpi/pmix_v4: _errhandler: irene2432 [66]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3169] slurmstepd-irene2518: error: mpi/pmix_v4: _errhandler: irene2518 [95]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4563] slurmstepd-irene2411: error: mpi/pmix_v4: _errhandler: irene2411 [56]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2688] slurmstepd-irene1514: error: mpi/pmix_v4: _errhandler: irene1514 [10]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:481] slurmstepd-irene2474: error: mpi/pmix_v4: _errhandler: irene2474 [78]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3745] slurmstepd-irene2522: error: mpi/pmix_v4: _errhandler: irene2522 [96]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4610] slurmstepd-irene1532: error: mpi/pmix_v4: _errhandler: irene1532 [19]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:912] slurmstepd-irene2471: error: mpi/pmix_v4: _errhandler: irene2471 [76]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3648] slurmstepd-irene2515: error: mpi/pmix_v4: _errhandler: irene2515 [93]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4465] slurmstepd-irene2506: error: mpi/pmix_v4: _errhandler: irene2506 [89]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4273] slurmstepd-irene2430: error: mpi/pmix_v4: _errhandler: irene2430 [65]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3122] slurmstepd-irene1497: error: mpi/pmix_v4: _errhandler: irene1497 [6]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:288] slurmstepd-irene2456: error: mpi/pmix_v4: _errhandler: irene2456 [72]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3456] slurmstepd-irene1575: error: mpi/pmix_v4: _errhandler: irene1575 [42]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2017] slurmstepd-irene2525: error: mpi/pmix_v4: _errhandler: irene2525 [98]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4704] slurmstepd-irene1525: error: mpi/pmix_v4: _errhandler: irene1525 [13]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:624] slurmstepd-irene2524: error: mpi/pmix_v4: _errhandler: irene2524 [97]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4657] slurmstepd-irene1494: error: mpi/pmix_v4: _errhandler: irene1494 [5]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:256] slurmstepd-irene1547: error: mpi/pmix_v4: _errhandler: irene1547 [34]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1634] slurmstepd-irene1542: error: mpi/pmix_v4: _errhandler: irene1542 [29]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1394] slurmstepd-irene2514: error: mpi/pmix_v4: _errhandler: irene2514 [92]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4462] slurmstepd-irene1540: error: mpi/pmix_v4: _errhandler: irene1540 [27]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1321] slurmstepd-irene1498: error: mpi/pmix_v4: _errhandler: irene1498 [7]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:336] slurmstepd-irene1546: error: mpi/pmix_v4: _errhandler: irene1546 [33]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1584] slurmstepd-irene1539: error: mpi/pmix_v4: _errhandler: irene1539 [26]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1248] slurmstepd-irene1523: error: mpi/pmix_v4: _errhandler: irene1523 [12]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:620] slurmstepd-irene2420: error: mpi/pmix_v4: _errhandler: irene2420 [62]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2976] slurmstepd-irene1573: error: mpi/pmix_v4: _errhandler: irene1573 [40]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1920] slurmstepd-irene2384: error: mpi/pmix_v4: _errhandler: irene2384 [47]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2269] slurmstepd-irene2502: error: mpi/pmix_v4: _errhandler: irene2502 [85]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4117] slurmstepd-irene2423: error: mpi/pmix_v4: _errhandler: irene2423 [63]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3026] slurmstepd-irene2473: error: mpi/pmix_v4: _errhandler: irene2473 [77]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3696] slurmstepd-irene1447: error: mpi/pmix_v4: _errhandler: irene1447 [1]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:83] slurmstepd-irene1526: error: mpi/pmix_v4: _errhandler: irene1526 [14]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:675] slurmstepd-irene1646: error: mpi/pmix_v4: _errhandler: irene1646 [46]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2247] slurmstepd-irene2416: error: mpi/pmix_v4: _errhandler: irene2416 [59]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2835] slurmstepd-irene1541: error: mpi/pmix_v4: _errhandler: irene1541 [28]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1344] slurmstepd-irene1507: error: mpi/pmix_v4: _errhandler: irene1507 [9]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:477] slurmstepd-irene2408: error: mpi/pmix_v4: _errhandler: irene2408 [55]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2644] slurmstepd-irene2505: error: mpi/pmix_v4: _errhandler: irene2505 [88]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4265] slurmstepd-irene2479: error: mpi/pmix_v4: _errhandler: irene2479 [80]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3847] slurmstepd-irene2464: error: mpi/pmix_v4: _errhandler: irene2464 [74]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3552] slurmstepd-irene2504: error: mpi/pmix_v4: _errhandler: irene2504 [87]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:4176] slurmstepd-irene1548: error: mpi/pmix_v4: _errhandler: irene1548 [35]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1693] slurmstepd-irene2423: error: common_file_write_content: unable to open '/sys/fs/cgroup/freezer/slurm_irene2423/uid_40711/job_4914725/step_0/freezer.state' for writing: Permission denied slurmstepd-irene2423: error: common_file_write_content: unable to open '/sys/fs/cgroup/freezer/slurm_irene2423/uid_40711/job_4914725/step_0/freezer.state' for writing: Permission denied slurmstepd-irene2448: error: mpi/pmix_v4: _errhandler: irene2448 [70]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3360] slurmstepd-irene2402: error: mpi/pmix_v4: _errhandler: irene2402 [52]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2496] slurmstepd-irene2469: error: mpi/pmix_v4: _errhandler: irene2469 [75]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3608] slurmstepd-irene2444: error: mpi/pmix_v4: _errhandler: irene2444 [69]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3313] slurmstepd-irene1506: error: mpi/pmix_v4: _errhandler: irene1506 [8]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:389] slurmstepd-irene1538: error: mpi/pmix_v4: _errhandler: irene1538 [25]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1200] slurmstepd-irene1535: error: mpi/pmix_v4: _errhandler: irene1535 [22]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1056] slurmstepd-irene1572: error: mpi/pmix_v4: _errhandler: irene1572 [39]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1872] slurmstepd-irene1527: error: mpi/pmix_v4: _errhandler: irene1527 [15]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:720] slurmstepd-irene1534: error: mpi/pmix_v4: _errhandler: irene1534 [21]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1038] slurmstepd-irene2404: error: mpi/pmix_v4: _errhandler: irene2404 [54]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2612] slurmstepd-irene2392: error: mpi/pmix_v4: _errhandler: irene2392 [49]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2352] slurmstepd-irene1533: error: mpi/pmix_v4: _errhandler: irene1533 [20]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:960] slurmstepd-irene1415: error: mpi/pmix_v4: _errhandler: irene1415 [0]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1] slurmstepd-irene2494: error: mpi/pmix_v4: _errhandler: irene2494 [83]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3984] slurmstepd-irene2412: error: mpi/pmix_v4: _errhandler: irene2412 [57]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2737] slurmstepd-irene1574: error: mpi/pmix_v4: _errhandler: irene1574 [41]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1968] slurmstepd-irene1577: error: mpi/pmix_v4: _errhandler: irene1577 [44]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2112] slurmstepd-irene1544: error: mpi/pmix_v4: _errhandler: irene1544 [31]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1488] slurmstepd-irene1537: error: mpi/pmix_v4: _errhandler: irene1537 [24]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1152] slurmstepd-irene2390: error: mpi/pmix_v4: _errhandler: irene2390 [48]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2307] slurmstepd-irene1531: error: mpi/pmix_v4: _errhandler: irene1531 [18]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:871] slurmstepd-irene1530: error: mpi/pmix_v4: _errhandler: irene1530 [17]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:822] slurmstepd-irene1529: error: mpi/pmix_v4: _errhandler: irene1529 [16]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:777] slurmstepd-irene2424: error: mpi/pmix_v4: _errhandler: irene2424 [64]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3072] slurmstepd-irene1543: error: mpi/pmix_v4: _errhandler: irene1543 [30]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1441] slurmstepd-irene1545: error: mpi/pmix_v4: _errhandler: irene1545 [32]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1537] slurmstepd-irene2485: error: mpi/pmix_v4: _errhandler: irene2485 [81]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3909] slurmstepd-irene2440: error: mpi/pmix_v4: _errhandler: irene2440 [68]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3300] slurmstepd-irene1475: error: mpi/pmix_v4: _errhandler: irene1475 [3]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:145] slurmstepd-irene1471: error: mpi/pmix_v4: _errhandler: irene1471 [2]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:96] slurmstepd-irene2452: error: mpi/pmix_v4: _errhandler: irene2452 [71]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3413] slurmstepd-irene1485: error: mpi/pmix_v4: _errhandler: irene1485 [4]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:220] slurmstepd-irene1549: error: mpi/pmix_v4: _errhandler: irene1549 [36]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1728] slurmstepd-irene2415: error: mpi/pmix_v4: _errhandler: irene2415 [58]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2784] slurmstepd-irene2418: error: mpi/pmix_v4: _errhandler: irene2418 [60]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2880] slurmstepd-irene1576: error: mpi/pmix_v4: _errhandler: irene1576 [43]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2064] slurmstepd-irene1645: error: mpi/pmix_v4: _errhandler: irene1645 [45]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2161] slurmstepd-irene2396: error: mpi/pmix_v4: _errhandler: irene2396 [51]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2449] slurmstepd-irene2403: error: mpi/pmix_v4: _errhandler: irene2403 [53]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2544] slurmstepd-irene1550: error: mpi/pmix_v4: _errhandler: irene1550 [37]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1776] slurmstepd-irene2433: error: mpi/pmix_v4: _errhandler: irene2433 [67]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3243] slurmstepd-irene2457: error: mpi/pmix_v4: _errhandler: irene2457 [73]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3505] slurmstepd-irene2478: error: mpi/pmix_v4: _errhandler: irene2478 [79]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:3792] slurmstepd-irene2393: error: mpi/pmix_v4: _errhandler: irene2393 [50]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2401] slurmstepd-irene1551: error: mpi/pmix_v4: _errhandler: irene1551 [38]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:1824] slurmstepd-irene2419: error: mpi/pmix_v4: _errhandler: irene2419 [61]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.4914725.0:2928] + exit 0