Build #18

Plan branch:
duplicate of RELEASE674BUILDCASALITH dedicated to GPU-enabled builds

Build: #18 failed Child of CASASPECS-RELEASE674BUILDCASALITH-20

Build result summary

Details

Completed
Queue duration
6 hours
Duration
425 minutes
Labels
None
Revisions
Casa6
8a438e2bda026d614fdb20f6a2a0652610a6dce4
OPEN-CASA-PKG
e5285e68467f01199807a1f1a978aec8a4781be7
No failed test found. A possible compilation error occurred.

Responsible

  • Takahiro Tsutsumi Takahiro Tsutsumi Automatically assigned

Code commits

Casa6
Author Commit Message Commit date
Rui Xue Rui Xue 8a438e2bda026d614fdb20f6a2a0652610a6dce4 Pull request #1051: CAS-14799
Merge in CASA/casa6 from CAS-14799 to release/6.7.4

* commit '15507f974958f27189bd63c7f0aa56f675d47e82':
  CAS-14799: Fix out-of-bounds MPI segmentation faults in BriggsCubeWeightor
  CAS-14799: Fix ncores-dependent tclean segmentation fault in `BriggsCubeWeightor`
Rui Xue Rui Xue 15507f974958f27189bd63c7f0aa56f675d47e82 m CAS-14799: Fix out-of-bounds MPI segmentation faults in BriggsCubeWeightor
The SynthesisImager uses BriggsCubeWeightor to compute weights for the image cube. During distributed MPI execution (`ncores > 1`), the output cube's spectral axis is partitioned across multiple nodes into smaller local subcubes.

When evaluating MS data channels, `GridFT::channelMap` maps the visibilities' frequencies into the local subcube's channel indices. If an MS contains dataset frequencies (e.g., from large Doppler swings or wide ephemeris velocity changes) that land *above* the local MPI subcube's maximum channel, `channelMap` returns an index exceeding the local array bounds.

Previously, BriggsCubeWeightor only checked the lower bound (`chanMap(chn) > -1`) before using the calculated index to access `wgtDensity` and the `f2_p`/`d2_p` arrays. Accessing these arrays with indices belonging to higher MPI partitions resulted in unconstrained out-of-bounds memory accesses, causing silent heap corruption or explicit Segmentation Faults, heavily dependent on the MPI slicing and dataset footprint.

This commit resolves the issue by introducing strict upper-bound channel footprint constraints (`chanMap(chn) < shape()[3]`) in both `weightUniform` and `getWeightUniform`. Data channels that mathematically fall outside the upper bounds of the local MPI partition are now securely ignored.
Rui Xue Rui Xue e7b86d9c728c2758ad7d6580e46c7080360be957 m CAS-14799: Fix ncores-dependent tclean segmentation fault in `BriggsCubeWeightor`
The SynthesisImager uses `BriggsCubeWeightor` to compute weights for the image cube. During this calculation, `BriggsCubeWeightor::cube2Matrix` extracts raw storage pointers `pcube` and `pflags` from `casacore::Array::getStorage(deleteIt)`.

Previously, the code incremented these exact pointers (`*pflags = *pcube++`) during array traversal. At the end of the function, the incremented pointers were passed directly to `freeStorage()` and `putStorage()`.

When array memory isn't contiguous (which occurs when data is sliced across MPI ranks), `getStorage()` allocates a temporary buffer and sets `deleteIt = true`. Passing the shifted pointer into `freeStorage()` invokes `delete[]` on an invalid memory address, causing heap corruption and a Segmentation fault randomly depending on the data slicing.

This commit resolves the issue by securely caching the original pointers returned from `getStorage` into distinct variables. The traversal loops now use separate copies of these pointers, enabling the original allocation addresses to be correctly evaluated and freed by `freeStorage()` and `putStorage()`.

Jira issues

IssueDescriptionStatus
Unknown Issue TypeCAS-14799Could not obtain issue details from Jira

Error summary for Build and Package Many Linux 2.28 Python 3.10

The job generated some errors, drill down into the full build log for more details.

Cloning into 'casa-build-utils'...
Switched to a new branch 'release/6.7.4'
Cloning into 'casa6'...
Updating files:  46% (3043/6540)
Updating files:  47% (3074/6540)
Updating files:  48% (3140/6540)
Updating files:  49% (3205/6540)
Updating files:  50% (3270/6540)
Updating files:  51% (3336/6540)
Updating files:  52% (3401/6540)
Updating files:  53% (3467/6540)
Updating files:  54% (3532/6540)
Updating files:  55% (3597/6540)
Updating files:  56% (3663/6540)
Updating files:  57% (3728/6540)
Updating files:  58% (3794/6540)
Updating files:  59% (3859/6540)
Updating files:  60% (3924/6540)
Updating files:  61% (3990/6540)
Updating files:  62% (4055/6540)
Updating files:  63% (4121/6540)
Updating files:  64% (4186/6540)
Updating files:  65% (4251/6540)
Updating files:  66% (4317/6540)
Updating files:  67% (4382/6540)
Updating files:  68% (4448/6540)
Updating files:  69% (4513/6540)
Updating files:  70% (4578/6540)
Updating files:  71% (4644/6540)
Updating files:  72% (4709/6540)
Updating files:  73% (4775/6540)
Updating files:  74% (4840/6540)
Updating files:  75% (4905/6540)
Updating files:  76% (4971/6540)
Updating files:  77% (5036/6540)
Updating files:  78% (5102/6540)
Updating files:  79% (5167/6540)
Updating files:  80% (5232/6540)
Updating files:  81% (5298/6540)
Updating files:  82% (5363/6540)
Updating files:  83% (5429/6540)
Updating files:  84% (5494/6540)
Updating files:  85% (5559/6540)
Updating files:  86% (5625/6540)
Updating files:  87% (5690/6540)
Updating files:  88% (5756/6540)
Updating files:  88% (5816/6540)
Updating files:  89% (5821/6540)
Updating files:  90% (5886/6540)
Updating files:  91% (5952/6540)
Updating files:  92% (6017/6540)
Updating files:  93% (6083/6540)
Updating files:  94% (6148/6540)
Updating files:  95% (6213/6540)
Updating files:  96% (6279/6540)
Updating files:  97% (6344/6540)
Updating files:  98% (6410/6540)
Updating files:  99% (6475/6540)
Updating files: 100% (6540/6540)
Updating files: 100% (6540/6540), done.
Submodule 'casatools/casacore' (https://github.com/casacore/casacore.git) registered for path 'casatools/casacore'
Submodule 'casatools/grpc-source' (https://github.com/grpc/grpc) registered for path 'casatools/grpc-source'
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/casacore'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source'...
Submodule 'third_party/abseil-cpp' (https://github.com/abseil/abseil-cpp) registered for path 'casatools/grpc-source/third_party/abseil-cpp'
Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'casatools/grpc-source/third_party/benchmark'
Submodule 'third_party/bloaty' (https://github.com/google/bloaty.git) registered for path 'casatools/grpc-source/third_party/bloaty'
Submodule 'third_party/boringssl' (https://github.com/google/boringssl.git) registered for path 'casatools/grpc-source/third_party/boringssl'
Submodule 'third_party/boringssl-with-bazel' (https://github.com/google/boringssl.git) registered for path 'casatools/grpc-source/third_party/boringssl-with-bazel'
Submodule 'third_party/cares/cares' (https://github.com/c-ares/c-ares.git) registered for path 'casatools/grpc-source/third_party/cares/cares'
Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'casatools/grpc-source/third_party/gflags'
Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'casatools/grpc-source/third_party/googletest'
Submodule 'third_party/protobuf' (https://github.com/google/protobuf.git) registered for path 'casatools/grpc-source/third_party/protobuf'
Submodule 'third_party/zlib' (https://github.com/madler/zlib) registered for path 'casatools/grpc-source/third_party/zlib'
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/abseil-cpp'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/benchmark'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/bloaty'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/boringssl'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/boringssl-with-bazel'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/cares/cares'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/gflags'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/googletest'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/protobuf'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/zlib'...
Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'casatools/grpc-source/third_party/bloaty/third_party/googletest'
Submodule 'third_party/libFuzzer' (https://chromium.googlesource.com/chromium/llvm-project/llvm/lib/Fuzzer) registered for path 'casatools/grpc-source/third_party/bloaty/third_party/libFuzzer'
Submodule 'third_party/re2' (https://github.com/google/re2) registered for path 'casatools/grpc-source/third_party/bloaty/third_party/re2'
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/bloaty/third_party/googletest'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/bloaty/third_party/libFuzzer'...
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/bloaty/third_party/re2'...
Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'casatools/grpc-source/third_party/gflags/doc'
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/gflags/doc'...
Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'casatools/grpc-source/third_party/protobuf/third_party/benchmark'
Cloning into '/export/cvpost135-nvme/casa/bamboohome/xml-data/build-dir/CASASPECS-RELEASE674BUILDCASALITHGPU-BPML2P310/source/casa6/casatools/grpc-source/third_party/protobuf/third_party/benchmark'...
Switched to a new branch 'release/6.7.4'
Cloning into 'casashell'...
Switched to a new branch 'release/6.7.4'
Error response from daemon: No such container: test_container
Error response from daemon: No such container: test_container
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 6002k  100 6002k    0     0  67.3M      0 --:--:-- --:--:-- --:--:-- 67.3M
[2134158] Failed to execute script 'atlutil' due to unhandled exception!
Traceback (most recent call last):
  File "atlutil.py", line 200, in <module>
  File "atlutil.py", line 165, in has_fix_version
  File "json/__init__.py", line 354, in loads
  File "json/decoder.py", line 339, in decode
  File "json/decoder.py", line 357, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
ls: cannot access '*.whl': No such file or directory
Already on 'release/6.7.4'
Error response from daemon: No such container: build_container
Error response from daemon: No such container: casalith_container
Error response from daemon: No such container: build_container
Error response from daemon: No such container: casalith_container