Build: #5 failed

Job: Test Tasks MPI Many Linux 2.28 Rocky 8.10 Py3.12 failed

Job result summary

Completed
Duration
80 minutes
Agent
cbt-el7-11.cv.nrao.edu
Revision
873c61fbb37c8dc6af128afc90c1fdd0d3972b95
Total tests
171
Fixed in
#31 (Child of CASASPECS-MASTERBUILDCASAMPI-31)
Number of retries
1

Tests

  • 171 tests in total
  • 1 test failed
  • 1 failure is new
  • 10 tests were quarantined / skipped
  • 70 minutes taken in total.
New test failures 1
Status Test Duration
Collapse Failed test_0_MPIInterface test_PyParallelImagerHelper_interface
< 1 sec
TypeError: 'NoneType' object is not subscriptable
self = <casampi.tests.test_casampi.test_0_MPIInterface testMethod=test_PyParallelImagerHelper_interface>

    def test_PyParallelImagerHelper_interface(self):
    
        # Get cluster (getCluster should automatically initialize it)
        self.sc = MPIInterface.getCluster()
        self.CL = self.sc._cluster
(21 more lines...)

Error summary

The build generated some errors. See the full build log for more details.

From https://open-bitbucket.nrao.edu/scm/casa/casa-data
   faa54e9..ca094d9  master     -> origin/master
Downloading geodetic/IERSpredict/table.dat (3.3 KB)
Downloading geodetic/IERSpredict/table.f0 (230 KB)
Downloading geodetic/IERSpredict/table.lock (325 B)
fatal: Not a git repository (or any parent up to mount point /export/home/cbt-el7-11)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Error response from daemon: No such container: wheel-container-test
Error response from daemon: No such container: wheel-container-test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 6002k  100 6002k    0     0  21.7M      0 --:--:-- --:--:-- --:--:-- 22.1M
[16134] Failed to execute script 'atlutil' due to unhandled exception!
Traceback (most recent call last):
  File "atlutil.py", line 200, in <module>
  File "atlutil.py", line 165, in has_fix_version
  File "json/__init__.py", line 354, in loads
  File "json/decoder.py", line 339, in decode
  File "json/decoder.py", line 357, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 12 column 1 (char 11)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
Already on 'master'
Already on 'master'
2025-05-20 18:40:33        SEVERE        MPICommandClient::stop_services::MPICommandClient::stop_services::casa        Aborting command request with id# 70: {'command': 'pow(a,b)', 'parameters': {'a': 10, 'b': 100000000000000000}, 'mode': 'eval', 'id': 70, 'server': 1, 'status': 'request sent'}
2025-05-20 18:40:36        SEVERE        MPICommandClient::stop_services::MPICommandClient::stop_services::casa        MPIServers with rank [1] are in timeout condition, skipping MPI_Finalize()
2025-05-20 18:40:36        SEVERE        MPICommandClient::stop_services::MPICommandClient::stop_services::casa        Not possible to finalize gracefully... calling Aborting MPI environment
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
  Proc: [[53661,1],0]
  Errorcode: 0

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
prterun has exited due to process rank 0 with PID 5489 on node 9f37a20f6bd4 exiting
improperly. There are three reasons this could occur:

1. this process did not call "init" before exiting, but others in the
job did. This can cause a job to hang indefinitely while it waits for
all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

3. this process called "MPI_Abort" or "prte_abort" and the mca
parameter prte_create_session_dirs is set to false. In this case, the
run-time cannot detect that the abort call was an abnormal
termination. Hence, the only error message you will receive is this
one.

This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).

You can avoid this message by specifying -quiet on the prterun command
line.
--------------------------------------------------------------------------