Build #4

Plan branch:

Build: #4 failed Changes by Rui Xue <rx.astro@gmail.com>

Build result summary

Details

Completed
Queue duration
< 1 second
Duration
1 second
Labels
None
Agent
casaci-4.cv.nrao.edu test agent8
Revision
b04760eb8e5f1f793c53c497837b2d699aa760b2
Failing since
#3 (Changes by Rui Xue – )
No failed test found. A possible compilation error occurred.

Responsible

  • Rui Xue Rui Xue Automatically assigned

Code commits

Author Commit Message Commit date
Rui Xue <rx.astro@gmail.com> Rui Xue <rx.astro@gmail.com> b04760eb8e5f1f793c53c497837b2d699aa760b2 PIPE-3073: Improve Dask cluster robustness for CASA C++ workloads (borrowed from `pclean` experiments)
- Integrate Dask cluster robustness optimizations proven in `pclean`
  to accommodate ALMA pipeline's monolithic C++ bindings (casatools)
  which frequently hold the GIL and allocate memory outside Python.
- Introduce `_patch_dask_tcp` monkey-patch to reject implausibly
  large TCP frames (>1 GiB), gracefully recycling stale SLURM sockets
  rather than crashing workers with `MemoryError`.
- Override Dask's default memory management inside `start_daskcluster`
  (disabling pause/spill/terminate heuristics and clamping LocalCluster
  memory_limit to 0) to prevent the scheduler from incorrectly starving
  workers during intensive, unmanaged casatools executions.
- Massively increase TCP/comm/heartbeat timeouts (`worker-ttl` to 20m)
  to ensure workers aren't falsely terminated by the scheduler during
  prolonged blocking C++ tasks.
- Add an explanatory block in `pipeline/config.yaml` clarifying that these
  hardcoded stability overrides will ignore corresponding user memory
  adjustments for Dask.

Jira issues

IssueDescriptionStatus
Unknown Issue TypePIPE-3073Could not obtain issue details from Jira

Error summary for Pipeline PR Test 6.7.4

The job generated some errors, drill down into the full build log for more details.

Failed to prepare the build 'PIPESPECS - Pipeline Pull Request Test - PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement - Pipeline PR Test 6.7.4 #4 (PIPESPECS-PPRT559-PPRT674-4)'