Build: #32 was successful Code changes detected
Code commits
Pipeline
-
Rui Xue 3b33c60148be5e751a410e4be609824bc4f65497 m
PIPE-3073: Add pipeline CLI entry point (python -m pipeline run/submit)
Add pipeline/__main__.py providing two subcommands:
- `run CONFIG_YAML` — run the pipeline locally (blocking); dispatches via
pipeconfig.ppr (executeppr) or pipeconfig.vis (recipereducer)
- `submit CONFIG_YAML` — generate an sbatch script from pipeconfig.dask.jobqueue.slurm
and optionally call sbatch (non-blocking)
Both subcommands share --loglevel / --plotlevel / --importonly flags.
`run` also creates a rawdata symlink in the CWD (override via pipeconfig.rawdata).
Prefect API URL is applied before any pipeline import so Prefect picks up the
correct server URL from pipeconfig.prefect.api_url at settings-cache time.
Guard Dask autostart and the host environment banner in __init__.py behind a
sys.argv check (_cli_dask_allowed): only `pipeline run` permits autostart;
`--help`, `submit`, and bare invocations skip both.
Register the `pipeline` console script in pyproject.toml.
Add pipeconfig.rawdata to config.yaml defaults.
Co-authored-by: Copilot <copilot@github.com>
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 56745139e
# Last commands done (46 commands done):
# pick df35f17c1 PIPE-3073: Resolve host and dashboard address with fallback to local IP
# pick 0cb96bdec PIPE-3073: Add pipeline CLI entry point (python -m pipeline run/submit)
# Next commands to do (2 remaining commands):
# pick 92112577e PIPE-3073: Fix Prefect duplicate task run key and skip Dask for single-item lists
# pick d9e72b8cd PIPE-3073: update task_run_name lambda for new Prefect API
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pipeline/__init__.py
# new file: pipeline/__main__.py
# modified: pipeline/config.yaml
# modified: pyproject.toml
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
#- pipeline/__init__.py (version 3b33c60148be5e751a410e4be609824bc4f65497) (diffs)
- pipeline/__main__.py (version 3b33c60148be5e751a410e4be609824bc4f65497) (diffs)
- pipeline/config.yaml (version 3b33c60148be5e751a410e4be609824bc4f65497) (diffs)
- pyproject.toml (version 3b33c60148be5e751a410e4be609824bc4f65497) (diffs)
-
Rui Xue 030e05d500eac06c5544b067ee59089e01926806 m
PIPE-3073: Fix Prefect duplicate task run key and skip Dask for single-item lists
- prefect_adapter: append UUID suffix to task_run_name to avoid
unique constraint violation on uq_task_run__flow_run_id_task_key_dynamic_key
when Prefect server restarts and client retries a task submission
- daskhelpers: skip Dask cluster for single-item inputs in parallel_map
to avoid unnecessary overhead when parallelism provides no benefit
Co-authored-by: Copilot <copilot@github.com> -
Rui Xue ff5cba6381442222ee32f137db075ba25e0f9577 m
PIPE-3073: update task_run_name lambda for new Prefect API
Prefect's resolve_custom_task_run_name now uses inspect.signature to
detect whether a task_run_name callable accepts a 'parameters' kwarg.
If not, it calls the function with zero arguments, causing:
TypeError: <lambda>() missing 1 required positional argument: 'task_name'
Update the lambda signature to use 'parameters' so Prefect passes the
full task parameters dict, and index task_name from it. -
Rui Xue bc97478b98593947b26adad3e4118542a47f2ce1 m
PIPE-3073: Add `cloudpickle` dependency for extended pickling support
Co-authored-by: Copilot <copilot@github.com>
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# Author: Rui Xue <rxue@nrao.edu>
#
# interactive rebase in progress; onto 56745139e
# Last commands done (43 commands done):
# pick b2c829ee6 PIPE-3073: replace `pclean` with generic MPI dispatch via `mpirunner`
# pick e1b446e58 PIPE-3073: Add `cloudpickle` dependency for extended pickling support
# Next commands to do (5 remaining commands):
# pick 9d897bcc9 PIPE-3073: Update Dask cluster host configuration to use fully-qualified hostname if not set
# pick df35f17c1 PIPE-3073: Resolve host and dashboard address with fallback to local IP
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pyproject.toml
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
# -
Rui Xue ee0564742c1f2652345fc49b96edcc82d2dda576 m
PIPE-3073: Update Dask cluster host configuration to use fully-qualified hostname if not set
-
Rui Xue 47a5a425647485069ba792e17261a903eaa351ea m
PIPE-3073: Resolve host and dashboard address with fallback to local IP
Co-authored-by: Copilot <copilot@github.com> -
Rui Xue 6350ab3defc4fc98080d47d7569aa66e0d1656d4 m
PIPE-3073: replace `pclean` with generic MPI dispatch via `mpirunner`
Add pipeline/infrastructure/mpirunner.py providing call_with_mpi and
run_in_mpicasa: generic helpers that dispatch any callable to a fresh
MPI-enabled CASA subprocess, or execute in-process when an MPI client
session is already active.
Key changes:
- Remove infrastructure/utils/pclean.py and its tests; functionality
superseded by mpirunner
- casa_tasks._tclean wraps casatasks.tclean with call_with_mpi,
accepting parallel=False|True|{'nproc': N}; tclean() JobRequest
delegates to _tclean
- imageparams_base: extract _run_psf_imager / @call_with_mpi
_run_parallel_psf_imager; remove duplicate override in
imageparams_vlass_single_epoch_continuum
- Replace parse_mpi_input_parameter with parse_parallel_input_parameter
(Dask-aware) throughout findcont, makeimages, tclean, imageparams
- makeimages: enable parallel tclean when is_mpi_ready() OR
is_dask_ready(); nproc derived from active Dask worker count
- pipeline/__init__: skip Dask autostart when MPI is enabled to prevent
recursive process creation inside mpicasa workers
- xvfb-run -e /dev/null silences xkbcomp keysym warnings from Xvfb
- cloudpickle by-value serialization for Python callables; SWIG objects
(casatasks.tclean) fall back to module+qualname import path
- Add casa_tasks_test.py: 10 tests covering find_executable, serial and
parallel tclean error handling, and integration tests against real data
Co-authored-by: Copilot <copilot@github.com>
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 56745139e
# Last commands done (42 commands done):
# pick e2baa6f38 PIPE-3073: Update documentation: reorder `demo_prefect_tier0futures.md` inclusion in `index.rst` and fix table formatting
# pick b2c829ee6 PIPE-3073: replace `pclean` with generic MPI dispatch via `mpirunner`
# Next commands to do (6 remaining commands):
# pick e1b446e58 PIPE-3073: Add `cloudpickle` dependency for extended pickling support
# pick 9d897bcc9 PIPE-3073: Update Dask cluster host configuration to use fully-qualified hostname if not set
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pipeline/__init__.py
# modified: pipeline/hif/heuristics/imageparams_vlass_single_epoch_continuum.py
# modified: pipeline/hif/tasks/findcont/findcont.py
# modified: pipeline/hif/tasks/makeimages/makeimages.py
# modified: pipeline/hif/tasks/tclean/tclean.py
# modified: pipeline/infrastructure/casa_tasks.py
# new file: pipeline/infrastructure/casa_tasks_test.py
# new file: pipeline/infrastructure/mpirunner.py
# deleted: pipeline/infrastructure/utils/pclean.py
# deleted: pipeline/infrastructure/utils/pclean_test.py
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
#- pipeline/__init__.py (version 6350ab3defc4fc98080d47d7569aa66e0d1656d4) (diffs)
- pipeline/hif/heuristics/imageparams_vlass_single_epoch_continuum.py (version 6350ab3defc4fc98080d47d7569aa66e0d1656d4) (diffs)
- pipeline/hif/tasks/findcont/findcont.py (version 6350ab3defc4fc98080d47d7569aa66e0d1656d4) (diffs)
- pipeline/hif/tasks/makeimages/makeimages.py (version 6350ab3defc4fc98080d47d7569aa66e0d1656d4) (diffs)
- pipeline/hif/tasks/tclean/tclean.py (version 6350ab3defc4fc98080d47d7569aa66e0d1656d4) (diffs)
- pipeline/infrastructure/casa_tasks.py (version 6350ab3defc4fc98080d47d7569aa66e0d1656d4) (diffs)
- pipeline/infrastructure/casa_tasks_test.py (version 6350ab3defc4fc98080d47d7569aa66e0d1656d4) (diffs)
- pipeline/infrastructure/mpirunner.py (version 6350ab3defc4fc98080d47d7569aa66e0d1656d4) (diffs)
- pipeline/infrastructure/utils/pclean.py (version 6350ab3defc4fc98080d47d7569aa66e0d1656d4) (diffs)
- pipeline/infrastructure/utils/pclean_test.py (version 6350ab3defc4fc98080d47d7569aa66e0d1656d4) (diffs)
-
Rui Xue b7d76693f056530b1f8ed22e24f31a3c2b266826 m
PIPE-3073: Update documentation: reorder `demo_prefect_tier0futures.md` inclusion in `index.rst` and fix table formatting
Co-authored-by: Copilot <copilot@github.com>
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 56745139e
# Last commands done (41 commands done):
# pick 0ec2b2168 PIPE-3073: Handle FileNotFoundError for `Xvfb` binary in `start_xvfb` function
# pick e2baa6f38 PIPE-3073: Update documentation: reorder `demo_prefect_tier0futures.md` inclusion in `index.rst` and fix table formatting
# Next commands to do (7 remaining commands):
# pick b2c829ee6 PIPE-3073: replace `pclean` with generic MPI dispatch via `mpirunner`
# pick e1b446e58 PIPE-3073: Add `cloudpickle` dependency for extended pickling support
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: docs/source/devel/reference/demo_prefect_tier0futures.md
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
# -
Rui Xue 5c889d52737f4b6c196ae7dc37d359ad41d09cce m
PIPE-3073: Fix FileNotFoundError on worker crash/reschedule in Tier0PipelineTask
- mpihelpers: defer task_args file deletion to inside the returned
callable so the file survives a Dask worker crash and reschedule retry
- daskhelpers: fix pure=True→False so Dask does not cache/deduplicate
submissions of the same task class across different MSes
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# Author: Rui Xue <rxue@nrao.edu>
#
# interactive rebase in progress; onto 56745139e
# Last commands done (39 commands done):
# pick 47c7b2733 PIPE-3073: move prefect tier-0 demo doc to develdocmd, add to index
# pick 791c87bdf PIPE-3073: Fix FileNotFoundError on worker crash/reschedule in Tier0PipelineTask
# Next commands to do (9 remaining commands):
# pick 0ec2b2168 PIPE-3073: Handle FileNotFoundError for `Xvfb` binary in `start_xvfb` function
# pick e2baa6f38 PIPE-3073: Update documentation: reorder `demo_prefect_tier0futures.md` inclusion in `index.rst` and fix table formatting
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pipeline/infrastructure/daskhelpers.py
# modified: pipeline/infrastructure/mpihelpers.py
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
# -
Rui Xue 326a5e66e7f384941500d9481c15ac1255bc22b0 m
PIPE-3073: Handle FileNotFoundError for `Xvfb` binary in `start_xvfb` function
Co-authored-by: Copilot <copilot@github.com> -
Rui Xue 61b7792720b6a6dc7c220cb8d008b84bb2ab279e m
PIPE-3073: move prefect tier-0 demo doc to develdocmd, add to index
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 56745139e
# Last commands done (38 commands done):
# pick 2f5c0f61f PIPE-3073: add PostgreSQL setup instructions; use env vars for host/port
# pick 47c7b2733 PIPE-3073: move prefect tier-0 demo doc to develdocmd, add to index
# Next commands to do (10 remaining commands):
# pick 791c87bdf PIPE-3073: Fix FileNotFoundError on worker crash/reschedule in Tier0PipelineTask
# pick 0ec2b2168 PIPE-3073: Handle FileNotFoundError for `Xvfb` binary in `start_xvfb` function
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# renamed: scripts/demo_prefect_tier0futures.md -> docs/source/devel/reference/demo_prefect_tier0futures.md
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
# -
Rui Xue c9a2ecbae41a05f4adbda574be76c3d241b2a718 m
PIPE-3073: Add `demo_prefect_tier0futures_pltest1.py` and `demo_prefect_tier0futures_polcal.py`
Co-authored-by: Copilot <copilot@github.com>- scripts/demo_prefect_tier0futures.py (version c9a2ecbae41a05f4adbda574be76c3d241b2a718) (diffs)
- scripts/demo_prefect_tier0futures_pltest1.py (version c9a2ecbae41a05f4adbda574be76c3d241b2a718) (diffs)
- scripts/demo_prefect_tier0futures_polcal.py (version c9a2ecbae41a05f4adbda574be76c3d241b2a718) (diffs)
-
Rui Xue 12e2da244890704b63463ff584f829860d75646b m
PIPE-3073: parallelize importdata QA flagdata calls; fix xvfb DISPLAY propagation
- daskhelpers: add parallel_map() utility that runs fn(item) for each
item via dask.delayed/dask.compute when is_daskclient_allowed(), else
falls back to sequential map()
- importdata/qa: use parallel_map() to run flagdata(mode='summary')
across all EBs concurrently; replace serial for-loop with zip() over
pre-fetched results; no change to downstream logic
- __init__.start_xvfb(): switch Display(visible=0) to
Display(backend='xvfb', manage_global_env=True); explicitly set
os.environ['DISPLAY'] from new_display_var after .start() to
guarantee child processes (e.g. casaplotserver/Qt) inherit the
correct virtual display rather than the SSH-forwarded :0; include
effective DISPLAY value in debug log
Co-authored-by: Copilot <copilot@github.com> -
Rui Xue a55a033114745f8abde89c6b3ea9e7cc9965b7c9 m
PIPE-3073: Silence noisy Dask TCP probe and post-session log floods
* daskhelpers: add `_silence_stray_tcp_probe_logs()` to drop tornado/
distributed ERROR records produced when non-Dask clients (HTTP probes,
health checks, stale browser tabs) hit the scheduler port and trigger
bogus multi-EiB frame reads. Filter is targeted -- only matches the
stray-probe signature (`_handle_stream`, `BaseTCPListener`, our
"Implausible allocation request" marker, or MemoryError exc_info) so
gen -
Rui Xue 494fba83c5ce4c91f44f54bf7c01f73d03ef2549 m
PIPE-3073: prefect demo: consolidate scripts into single CLI and update docs
- Replace demo_prefect_tier0futures.py, _polcal.py, _pltest1.py with a
single argparse-based script (choices: if2016, polcal, pltest1)
- Add recommended server startup env vars to .md (pool size, overflow,
timeout, logging level) with explanation table and PostgreSQL option
Co-authored-by: Copilot <copilot@github.com>- scripts/demo_prefect_tier0futures.md (version 494fba83c5ce4c91f44f54bf7c01f73d03ef2549) (diffs)
- scripts/demo_prefect_tier0futures.py (version 494fba83c5ce4c91f44f54bf7c01f73d03ef2549) (diffs)
- scripts/demo_prefect_tier0futures_pltest1.py (version 494fba83c5ce4c91f44f54bf7c01f73d03ef2549) (diffs)
- scripts/demo_prefect_tier0futures_polcal.py (version 494fba83c5ce4c91f44f54bf7c01f73d03ef2549) (diffs)
-
Rui Xue a96cffbdc95c6b406c1c1255a9cf1c64bf10b681 m
PIPE-3073: add PostgreSQL setup instructions; use env vars for host/port
- pyproject.toml: add postgresql from conda-forge to exp feature dependencies
- demo_prefect_tier0futures.md:
- document optional PostgreSQL backend for Prefect server with one-time
initdb/pg_ctl/createdb setup via `pixi shell -e exp`
- abstract server host and port as SERVER_IP / SERVER_PORT env vars
throughout all server-start examples
- convert indented code blocks to fenced blocks (fix MD046)
- minor formatting: trailing whitespace, blank lines between list items
and code fences
Co-authored-by: Copilot <copilot@github.com> -
Rui Xue 63a5273d522132def3d448f03a1bee586cdf4960 m
PIPE-3073: add Prefect server tuning and remote access guidance
- Add recommended server startup env vars (DATABASE_POOL_SIZE,
MAX_OVERFLOW, TIMEOUT, LOGGING_LEVEL) with explanation table
- Add PostgreSQL alternative for heavy Dask workloads
- Document hostname -f trick and SSH tunnel for remote access
Co-authored-by: Copilot <copilot@github.com> -
Rui Xue 057af074f7ba7c951940bdaed620d7a8820f6bf7 m
PIPE-2926: Enhance error reporting in pclean: include remote traceback in RuntimeError
Co-authored-by: Copilot <copilot@github.com> -
Rui Xue baaf43f10dbe5f400c01d3cf5891fc72b9e0f4d0 m
PIPE-3073: Remove unused import of Optional from pclean.py
-
Rui Xue f91fa83246ead5cb6e231b53ed4e74a4746aed65 m
PIPE-3073: prefect: fix recipe flow run name and suppress 500 noise
- _recipe_flow_name: look up procedure under params['kwargs'] since
run_recipe_flow(**kwargs) causes Prefect to nest all args there;
previously always fell back to 'procedure_unknown'
- run_recipe_flow: add persist_result=False to prevent Prefect from
trying to serialize the large Context object, which caused 500 errors
on set_state at end of flow
- _apply_prefect_config: suppress httpx INFO logs (HTTP 500 noise) by
setting httpx logger to WARNING level
Co-authored-by: Copilot <copilot@github.com> -
Rui Xue 1c1cf5ab1aa267660de504302964e950bda7dbc3 m
PIPE-3037: pyproject.toml: add pytest filterwarnings and drop `casa671-py312` env
- Migrate filterwarnings from pytest_old.ini into [tool.pytest.ini_options]:
suppress deprecated Document/XmlObject .async and PyFITS warnings
- Remove obsolete casa671-py312 pixi feature and environment
Co-authored-by: Copilot <copilot@github.com>
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 56745139e
# Last commands done (27 commands done):
# pick 4b16dca15 PIPE-3073: Improve Dask worker logging and fix distributed environment issues
# pick ab256b723 PIPE-3037: pyproject.toml: add pytest filterwarnings and drop `casa671-py312` env
# Next commands to do (21 remaining commands):
# pick c58802313 PIPE-3073: Add R.L. to Contributors list
# pick de7285d11 PIPE-3073: Silence noisy Dask TCP probe and post-session log floods
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pyproject.toml
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
# -
Rui Xue a62eb030d71495ac0d00b8cc71c000ebbbb6e5b7 m
PIPE-3073: Improve Dask worker logging and fix distributed environment issues
Enhance logging tracing with `DASK_WORKER_ID`, isolate D-Bus to prevent
deadlocks, fix tar member handling, and update logging consistency across
Prefect adapter and infrastructure modules.- pipeline/config.yaml (version a62eb030d71495ac0d00b8cc71c000ebbbb6e5b7) (diffs)
- pipeline/h/tasks/exportdata/exportdata.py (version a62eb030d71495ac0d00b8cc71c000ebbbb6e5b7) (diffs)
- pipeline/infrastructure/daskhelpers.py (version a62eb030d71495ac0d00b8cc71c000ebbbb6e5b7) (diffs)
- pipeline/infrastructure/logging.py (version a62eb030d71495ac0d00b8cc71c000ebbbb6e5b7) (diffs)
- pipeline/infrastructure/prefect_adapter.py (version a62eb030d71495ac0d00b8cc71c000ebbbb6e5b7) (diffs)
-
Rui Xue 293f1eb8a188a68b3c0ec462c20d718f10a89038 m
PIPE-3072: Refactor type hints in `find_executable` and update error handling in `pclean`. Remove unnecessary MPI marker from test classes.
-
Rui Xue 0487be3057444f504b4c08c4d1b8a066af82f93f m
PIPE-3073: Add missing import for `unix_seconds_to_datetime` in `conversion_test.py` that was lost during the last conflict resolution.
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 56745139e
# Last commands done (24 commands done):
# pick 03edbbcec PIPE-2926: CI/CD workarounds
# pick fa9b78cfe PIPE-3073: Add missing import for `unix_seconds_to_datetime` in `conversion_test.py` that was lost during the last conflict resolution.
# Next commands to do (24 remaining commands):
# pick c99c1833c PIPE-3072: Refactor type hints in `find_executable` and update error handling in `pclean`. Remove unnecessary MPI marker from test classes.
# pick 4b16dca15 PIPE-3073: Improve Dask worker logging and fix distributed environment issues
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pipeline/infrastructure/utils/conversion_test.py
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
# -
Rui Xue 02cc7faded2ea12bc456d3f6b506001d67a9eb35 m
PIPE-2926: CI/CD workarounds
- Target macOS 15+ in pixi system-requirements
Set macos = "15.0" so pixi prefers the latest `casatools``
wheels on `osx-arm64``. `uv`` may still fall back to a 14.0
wheel when no 15.0 build is available.
- Update `Pixi` setup action to version `0.9.4` and adjust `Pixi`` version to `0.62.2`
- Disable `pytest-xdist` for parallel workers with '-n 0' to avoid
resource contention on the GitHub macOS runners.
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 56745139e
# Last commands done (23 commands done):
# pick a989bc2a2 PIPE-2926: Update `pclean.py` with improved docstrings
# pick 03edbbcec PIPE-2926: CI/CD workarounds
# Next commands to do (25 remaining commands):
# pick fa9b78cfe PIPE-3073: Add missing import for `unix_seconds_to_datetime` in `conversion_test.py` that was lost during the last conflict resolution.
# pick c99c1833c PIPE-3072: Refactor type hints in `find_executable` and update error handling in `pclean`. Remove unnecessary MPI marker from test classes.
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pipeline/infrastructure/utils/pclean_test.py
# modified: pyproject.toml
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
# -
Rui Xue 3ba0666776cb0f7d337c64dc8dca158e42d3b789 m
PIPE-1836: Update datetime handling to use timezone-aware UTC timestamps across multiple modules.
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# Author: Rui Xue <rxue@nrao.edu>
#
# interactive rebase in progress; onto 56745139e
# Last commands done (21 commands done):
# pick c2fc77f23 PIPE-2926: Apply suggestions from code review
# pick 75bb79f1f PIPE-1836: Update datetime handling to use timezone-aware UTC timestamps across multiple modules.
# Next commands to do (27 remaining commands):
# pick a989bc2a2 PIPE-2926: Update `pclean.py` with improved docstrings
# pick 03edbbcec PIPE-2926: CI/CD workarounds
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pipeline/hifv/tasks/syspower/display.py
# modified: pipeline/infrastructure/tablereader.py
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
# -
Rui Xue df17a85cee1fe423a75ace82f4fd4fa5d15289fb m
PIPE-2926: Update `pclean.py` with improved docstrings
-
Rui Xue 8b6b621a2f9f16c75500c37d380f96b91d843091 m
PIPE-1836: additional removal of naive datetime objects.
-
Rui Xue <r-xue@users.noreply.github.com> 601fbae6544e7ff3962726e8431f589852da729f m
PIPE-2926: Apply suggestions from code review
* Fix typos
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>- pipeline/infrastructure/renderer/logger.py (version 601fbae6544e7ff3962726e8431f589852da729f) (diffs)
- pipeline/infrastructure/utils/pclean.py (version 601fbae6544e7ff3962726e8431f589852da729f) (diffs)
- pipeline/infrastructure/utils/pclean_test.py (version 601fbae6544e7ff3962726e8431f589852da729f) (diffs)
-
Rui Xue b8f9ee9c72c6465049e696078d54dc66f7508d3e m
PIPE-1836: Swicth to use timezone-aware `datetime` objects; remove `unix_seconds_to_datetime` and the associated test.
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# Author: Rui Xue <rxue@nrao.edu>
#
# interactive rebase in progress; onto 56745139e
# Last commands done (17 commands done):
# pick d30c6812f PIPE-2924: Run unit tests for all branches
# pick 7fe8f805b PIPE-1836: Swicth to use timezone-aware `datetime` objects; remove `unix_seconds_to_datetime` and the associated test.
# Next commands to do (31 remaining commands):
# pick bef1b696b PIPE-2926: Apply suggestions from code review
# pick a77948d32 PIPE-1836: additional removal of naive datetime objects.
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pipeline/infrastructure/utils/conversion.py
# modified: pipeline/infrastructure/utils/conversion_test.py
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
# -
Rui Xue <r-xue@users.noreply.github.com> bed6b2e9f8a657e982c27fd2b67e740d9ff3b720 m
PIPE-2926: Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> -
Rui Xue 7328b1bb5418632a16bb457ba05c6ad1e0413174 m
PIPE-2926: Add the `utils.pclean` utility helper and corresponding tests for `CASA6` executable search and tclean/parallel execution in an independent subprocess.
-
Rui Xue 170b8e3be95e8b643438a9141fce3e799147164c m
PIPE-1836: Replace deprecated `datetime` and `spawn` APIs and clean up deprecation warnings.
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# Author: Rui Xue <rxue@nrao.edu>
#
# interactive rebase in progress; onto 56745139e
# Last commands done (14 commands done):
# pick 237377119 PIPE-3073: Exclude imaging tasks in the tier0-enabling script
# pick f50a992f3 PIPE-1836: Replace deprecated `datetime` and `spawn` APIs and clean up deprecation warnings.
# Next commands to do (34 remaining commands):
# pick bba8a9ec3 PIPE-2926: Add the `utils.pclean` utility helper and corresponding tests for `CASA6` executable search and tclean/parallel execution in an independent subprocess.
# pick d30c6812f PIPE-2924: Run unit tests for all branches
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pipeline/infrastructure/renderer/logger.py
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
# -
Rui Xue ffb39afe5d3ba68e418f0a74759dfccc692910df m
PIPE-3073: Add demo guide for Prefect Tier0Futures cluster execution
- Added `scripts/demo_prefect_tier0futures.md` to document the end-to-end
setup process for testing the experimental Prefect + Dask Tier0Futures
processing architecture on ALMA cluster nodes.
- Detailed step-by-step instructions for booting the isolated `pixi`
environment and connecting the local repository without requiring
global dependency installations.
- Provided examples for spinning up Prefect API servers locally and
configuring real-time telemetry endpoints (`PREFECT_API_URL`).
- Included job submission templates for testing execution either within
interactive nodes or pushed as batch workloads via SLURM (`sbatch`). -
Rui Xue 2d7cbae0903b44d5b7c4e0485917bdc79b819854 m
PIPE-3073: Exclude imaging tasks in the tier0-enabling script
-
Rui Xue 39a7ad009c568484d28d9761fc45374f29aa6dfb m
PIPE-3073: Improve Dask cluster robustness for CASA C++ workloads (borrowed from `pclean` experiments)
- Integrate Dask cluster robustness optimizations proven in `pclean`
to accommodate ALMA pipeline's monolithic C++ bindings (casatools)
which frequently hold the GIL and allocate memory outside Python.
- Introduce `_patch_dask_tcp` monkey-patch to reject implausibly
large TCP frames (>1 GiB), gracefully recycling stale SLURM sockets
rather than crashing workers with `MemoryError`.
- Override Dask's default memory management inside `start_daskcluster`
(disabling pause/spill/terminate heuristics and clamping LocalCluster
memory_limit to 0) to prevent the scheduler from incorrectly starving
workers during intensive, unmanaged casatools executions.
- Massively increase TCP/comm/heartbeat timeouts (`worker-ttl` to 20m)
to ensure workers aren't falsely terminated by the scheduler during
prolonged blocking C++ tasks.
- Add an explanatory block in `pipeline/config.yaml` clarifying that these
hardcoded stability overrides will ignore corresponding user memory
adjustments for Dask. -
Rui Xue 84d383406e3a6fa0e0106b3edf7fa2e392836840 m
PIPE-3073: demo_prefect_tier0futures: update config and robustness fixes
- Update dataset paths to 2016.1.00053.S on NAASC lustre and introduce
store_dir variable so ppr_file, raw_dir, and work_dir share a common
base prefix
- Add prefect_api_url config variable; set PREFECT_API_URL before pipeline
imports so flow runs are visible in the running Prefect server instead
of going to the ephemeral in-process API
- Fix rawdata symlink handling: always recreate the symlink (unlink if
it exists); if rawdata is a real directory, rename it with a timestamp
backup before symlinking
- Override SIGINT handler after imports to call os._exit(130), bypassing
the dask_jobqueue SIGINT→sys.exit→atexit→cluster.close() deadlock that
caused Ctrl+C to hang
pipeline/config.yaml:
- Increase SLURM job memory from 10gb to 20gb
pipeline/infrastructure/daskhelpers.py:
- Raise default local worker cap from 4 to 6 -
Rui Xue 2da0cfc76ef59faece27f04a5fbef7439c6547da m
PIPE-3073: Fix async deadlock in Prefect and document tier0futures
- Refactor `_log_dask_summary` in `prefect_adapter.py` to execute entirely
within a detached daemon thread. This prevents a critical deadlock where
Prefect's `anyio` thread-portal and Dask's `scheduler_info` IO loop
collide, effectively ensuring that sluggish telemetry or locked
schedulers can never hang the root scientific pipeline.
- Implement a 10s timeout on the telemetry retrieval as an absolute
fallback mechanism.
- Add comprehensive architecture diagrams and Tier0Futures use cases to
`notes/prefect_guide.md`, explaining how Prefect and Dask interact
during ALMA regression execution.
- Update `notes/implementation.md` to reflect completed feature stability
for the dynamic GUI flow routing and Dask artifacts plugin.
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 56745139e
# Last commands done (9 commands done):
# pick 0a1a7c8b7 PIPE-3073: Add demo script for Prefect and tier0futures
# pick e85de80ca PIPE-3073: Fix async deadlock in Prefect and document tier0futures
# Next commands to do (39 remaining commands):
# pick b035ba6bc PIPE-3073: demo_prefect_tier0futures: update config and robustness fixes
# pick b04760eb8 PIPE-3073: Improve Dask cluster robustness for CASA C++ workloads (borrowed from `pclean` experiments)
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pipeline/infrastructure/prefect_adapter.py
# modified: pyproject.toml
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
# -
Rui Xue 5dd2aa4f7e2378e6420812f13d3d689b49198b14 m
PIPE-3073: Add demo script for Prefect and tier0futures
- Introduce `scripts/demo_prefect_tier0futures.py` as a ready-to-run
experiment script for testing Prefect integration with tier0futures.
- Provide clear Usage documentation instructing developers to execute the
script from within the experimental `exp` Pixi environment.
- Correct naming convention in variables and script name to match
`tier0futures`.
- Configured with a static test dataset and pathing to demonstrate
the orchestration dashboard capabilities. -
Rui Xue f164967eba514477f004050f506c3e9b352fdb2f m
PIPE-3073: Enhance Prefect observability with Dask artifacts and dynamic flow names - Publish a persistent Prefect markdown artifact linking to the Dask dashboard upon initialization. - Log per-stage Dask worker metrics (CPU, Memory, Threads, Task count) as formatted Prefect dashboard artifacts. - Dynamically generate flow run names from runtime pipeline parameters (using `prefect.runtime.flow_run.parameters`) instead of static string placeholders, resolving the 'unknown' name bug.
-
Rui Xue 71b2eb021b29208a1d02ae0ca9356b1d1aebb6c7 m
PIPE-3073: Prefect server connection, enable xvfb, add shebang - prefect_adapter: call _apply_prefect_config() at module load time instead of inside @flow body, so PREFECT_API_URL is set before Prefect resolves its server connection (fixes flows not appearing in the dashboard) - config.yaml: enable xvfb by default (xvfb: true) - procedure_add_parallel.py: add shebang line so the script can be executed directly without explicitly invoking python
-
Rui Xue eef51556b007e62f94d78993b676e5fcab9b6f1d m
PIPE-3073: Consolidate environment variable configuration into `config.yaml`
Move hardcoded defaults and scattered os.environ/os.getenv lookups into
the central config.yaml, making it the single source of truth for
pipeline configuration values.
Changes:
- Add get_pipeline_config() helper to config.py that resolves config
values via dot-path navigation with env var override and bool coercion
- Add services, heuristics, and weblog sections to config.yaml with
snake_case keys and inline env var documentation
- Remove module-level URL constants (FLUX_SERVICE_URL,
FLUX_SERVICE_URL_BACKUP, ANTPOS_SERVICE_URL) from Python modules
- Migrate direct os.environ/os.getenv calls to get_pipeline_config()
in dbfluxes.py, almaantpos.py, qa.py, applycal.py, htmlrenderer.py
Config precedence: env var > config.yaml > get_pipeline_config default.
Consolidated settings:
flux_service_url (env: FLUX_SERVICE_URL)
flux_service_url_backup (env: FLUX_SERVICE_URL_BACKUP)
services.antpos_url (env: ANTPOS_SERVICE_URL)
heuristics.allow_getantposalma_failure (env: ALLOW_GETANTPOSALMA_FAILURE)
heuristics.pipe356_qa_mode (env: PIPE356_QA_MODE)
heuristics.disable_casa_callibrary (env: DISABLE_CASA_CALLIBRARY)
weblog.rerender_stages (env: WEBLOG_RERENDER_STAGES)
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 56745139e
# Last commands done (5 commands done):
# pick 7a06ae020 PIPE-3073: Consolidate Prefect configuration into `config.yaml`
# pick 057fdbdf7 PIPE-3073: Consolidate environment variable configuration into `config.yaml`
# Next commands to do (43 remaining commands):
# pick fd9c0217f PIPE-3073: Prefect server connection, enable xvfb, add shebang - prefect_adapter: call _apply_prefect_config() at module load time instead of inside @flow body, so PREFECT_API_URL is set before Prefect resolves its server connection (fixes flows not appearing in the dashboard) - config.yaml: enable xvfb by default (xvfb: true) - procedure_add_parallel.py: add shebang line so the script can be executed directly without explicitly invoking python
# pick e40f15c0e PIPE-3073: Enhance Prefect observability with Dask artifacts and dynamic flow names - Publish a persistent Prefect markdown artifact linking to the Dask dashboard upon initialization. - Log per-stage Dask worker metrics (CPU, Memory, Threads, Task count) as formatted Prefect dashboard artifacts. - Dynamically generate flow run names from runtime pipeline parameters (using `prefect.runtime.flow_run.parameters`) instead of static string placeholders, resolving the 'unknown' name bug.
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pipeline/config.py
# modified: pipeline/config.yaml
# modified: pipeline/h/tasks/applycal/applycal.py
# modified: pipeline/hifa/tasks/antpos/almaantpos.py
# modified: pipeline/hifa/tasks/applycal/qa.py
# modified: pipeline/hifa/tasks/importdata/dbfluxes.py
# modified: pipeline/infrastructure/renderer/htmlrenderer.py
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
#- pipeline/config.py (version eef51556b007e62f94d78993b676e5fcab9b6f1d) (diffs)
- pipeline/config.yaml (version eef51556b007e62f94d78993b676e5fcab9b6f1d) (diffs)
- pipeline/h/tasks/applycal/applycal.py (version eef51556b007e62f94d78993b676e5fcab9b6f1d) (diffs)
- pipeline/hifa/tasks/antpos/almaantpos.py (version eef51556b007e62f94d78993b676e5fcab9b6f1d) (diffs)
- pipeline/hifa/tasks/applycal/qa.py (version eef51556b007e62f94d78993b676e5fcab9b6f1d) (diffs)
- pipeline/hifa/tasks/importdata/dbfluxes.py (version eef51556b007e62f94d78993b676e5fcab9b6f1d) (diffs)
- pipeline/infrastructure/renderer/htmlrenderer.py (version eef51556b007e62f94d78993b676e5fcab9b6f1d) (diffs)
-
Rui Xue 53ee559c8e1f26257c6771a534df12e94b0a453b m
PIPE-3073: Consolidate Prefect configuration into `config.yaml`
Move URL and environment-variable-based Prefect settings into the
central pipeline config.yaml so users have a single configuration
source.
- Add `api_url` field under `pipeconfig.prefect` in config.yaml;
when set, it populates PREFECT_API_URL automatically (env var
still takes precedence if already set)
- Add `_apply_prefect_config()` to apply config.yaml settings to
the environment at the start of each Prefect flow
- Read `flow_name` and `log_prints` from config.yaml to drive
@flow decorator parameters instead of hardcoding them
- Add `_get_prefect_config()` helper for reading the prefect
config section with safe defaults -
Rui Xue ac1263ee079111b196cef2a075635fd160d1d825 m
PIPE-3073: Add Prefect workflow adapter and fix Dask spawn safety
Add non-invasive Prefect 3 orchestration layer for pipeline workflows:
- Add prefect_adapter.py with @flow/@task wrappers for `executeppr`,
`executevlappr`, `recipereducer.reduce`, and run_named_tasks
- Add Dask observability bridge (SchedulerPlugin) that logs tier0
task transitions and cluster memory utilization to Prefect UI
- Add Prefect delegation guards in entry points (executeppr.py,
executevlappr.py, recipereducer.py) with graceful fallback
- Add is_prefect_enabled() gated by PIPELINE_USE_PREFECT env var
and `config.yaml` pipeconfig.prefect.enabled
- Add unit tests for is_prefect_enabled and TaskResultSummary
Fix Dask LocalCluster spawn safety in daskhelpers.py:
- Add multiprocessing.current_process().daemon check to
is_daskclient_allowed() to prevent recursive cluster creation
when workers spawned via multiprocessing.spawn re-import pipeline- pipeline/infrastructure/executeppr.py (version ac1263ee079111b196cef2a075635fd160d1d825) (diffs)
- pipeline/infrastructure/executevlappr.py (version ac1263ee079111b196cef2a075635fd160d1d825) (diffs)
- pipeline/infrastructure/prefect_adapter.py (version ac1263ee079111b196cef2a075635fd160d1d825) (diffs)
- pipeline/infrastructure/prefect_adapter_test.py (version ac1263ee079111b196cef2a075635fd160d1d825) (diffs)
- pipeline/recipereducer.py (version ac1263ee079111b196cef2a075635fd160d1d825) (diffs)
-
Rui Xue ea651782ee461a4735a01b95d95e147dff80279c m
PIPE-3073: add optional Prefect orchestration layer for pipeline task execution and workflow monitoring
-
Rui Xue 1fdec841e7e3905f36db2ee824cad525d7349ff2 m
PIPE-3073: Disallow starting dask client if this is a daemon process
-
Hajime Ezawa 56745139edf63b1a6dd4dc351ecde473f08e2156 m
Pull request #1968: PIPE-3246: add overwrite=True to image_args
Merge in PIPE/pipeline from PIPE-3246-hsd_imaging-sometimes-hsd_imaging-crashes-with-runtimeerror-output-file-exists to main
* commit '1113f6eeab147278cd16547f836b6d5012ed49df':
PIPE-3246: add overwrite=True to image_args -
Rui Xue e334846327e00b2cda5bb860498fa94a3f63db6f m
Pull request #2006: PIPE-3252/PIPE-3264
Merge in PIPE/pipeline from PIPE-3252-docs-update-to-main to main
* commit '721235e8ba3766673fef78026556b927f802fdcb':
PIPE-3252: add AGENTS.md and align ruff pre-commit
PIPE-3252: add JM to the contributor list -
Rui Xue 721235e8ba3766673fef78026556b927f802fdcb m
PIPE-3252: add AGENTS.md and align ruff pre-commit
- Add AGENTS.md defining operational directives, architectural invariants, modern Python standards, and developer guidelines.
- Point .github/copilot-instructions.md to AGENTS.md as the canonical authority for AI assistant instructions.
- Switch .pre-commit-config.yaml to local hooks invoking pixi-managed ruff to guarantee parity across IDE, pre-commit, and agent workflows. -
Rui Xue 86ba85641552eef9b6f903264182026fbe19f74e m
PIPE-3252: add JM to the contributor list
-
Rui Xue 09e7ce9075867e9c005c6a45926f0500ec33c5d9 m
Pull request #2003: PIPE-3252: add link to CASA Docs citation guide in acknowledgements
Merge in PIPE/pipeline from PIPE-3252-add-casa-citation-page to main
* commit '8c6a6a273861cfa5da6e3f57ed6248f35169d7bb':
PIPE-3252: add link to CASA Docs citation guide in acknowledgements -
Rui Xue 8c6a6a273861cfa5da6e3f57ed6248f35169d7bb m
PIPE-3252: add link to CASA Docs citation guide in acknowledgements
-
Remy Indebetouw a0586fcdb966530af2d64068f1c84376fdbd5144 m
Pull request #2000: reordered chronologically, added people who contributed significantly albeit not directly in the code repo
Merge in PIPE/pipeline from PIPE-3263-doc-updates to main
* commit 'a0038a8a2d5338d32976b6ba80057eff54bc66f7':
fixed "present" and double Andy
reordered chronologically, added people who contributed significantly albeit not directly in the code repo -
Remy Indebetouw <remy@virginia.edu> a0038a8a2d5338d32976b6ba80057eff54bc66f7 m
fixed "present" and double Andy
-
Remy Indebetouw <remy@virginia.edu> 0669c460be9ade6b26ce65288d1024f5d0d667bf m
reordered chronologically, added people who contributed significantly albeit not directly in the code repo
-
Rui Xue 6ebe41c7f7a634c6c0bd19ec6bc9a3dede8771da m
Pull request #1998: PIPE-3252: update release notes with ALMA task references and improve formatting
Merge in PIPE/pipeline from PIPE-3252-append-alma-task-ref-to-plug-main to main
* commit '8ac857aa5055fe97677559d13504a077fa590f08':
PIPE-3252: update release notes with ALMA task references and improve formatting -
Rui Xue 8ac857aa5055fe97677559d13504a077fa590f08 m
PIPE-3252: update release notes with ALMA task references and improve formatting
-
Rui Xue 17dd28330aea2d87240b0db9ff1e5883da443528 m
Pull request #1996: PIPE-3252: Add AL to Contributors list
Merge in PIPE/pipeline from PIPE-3252-append-alma-task-ref-to-plug-main to main
* commit '79b00e3aef7de55ad5b9c185aa71ccd39f6d86bd':
PIPE-3252: Add AL to Contributors list -
Rui Xue 79b00e3aef7de55ad5b9c185aa71ccd39f6d86bd m
PIPE-3252: Add AL to Contributors list
-
Rui Xue d4c1cf365fb7cf6405ee6f69b5aa57c67cc1c050 m
Pull request #1988: PIPE-3260: Fix false QA failures in `VLASetjy`
Merge in PIPE/pipeline from PIPE-3260-false-qa-warning-about-lack-of-flux-calibrator-in-hifv_vlasetjy-task to main
* commit '34228abc590663d2714f3d8cc7e37e8d124b2293':
PIPE-3260: Fix false QA failures in `VLASetjy` -
Rui Xue b0920bd33c2728cf8292bc773bf49ed5cb9e5ca4 m
Pull request #1983: PIPE-3258: fix reference antenna fallback and caltable cleanup in `hifv_semiFinalBPdcals`
Merge in PIPE/pipeline from PIPE-3258-antenna-fully-flagged-during-bandpass-scan-still-used-as-refant to main
* commit '8dabe1b5b997da832ea10696bb0a42d11d6c2337':
PIPE-3258: preserve list type in refant fallback and align testBPdcals
PIPE-3258: fix reference antenna fallback and caltable cleanup in `hifv_semiFinalBPdcals` -
Rui Xue 0e65d3296eb61d1facf16f2708c2a9cd6711f316 m
Pull request #1992: PIPE-3252 append alma task ref to plug main
Merge in PIPE/pipeline from PIPE-3252-append-alma-task-ref-to-plug-main to main
* commit 'c32c5ee6f49ee372db9c51fcd18132440540ac23':
PIPE-3252: partition dev docs and add contributors page
PIPE-3252: include ALMA tasks in User's Guide PDF and clean up doc links -
Rui Xue 9b876959674a29dd5f275d64375dd01d13d472fe m
PIPE-3252: include ALMA tasks in User's Guide PDF and clean up doc links
- Add `users_guide/alma_tasks.rst` and Sphinx hooks in conf.py to resolve task hyperlinks in `users_guide.pdf` while excluding them from HTML builds.
- Drop "known_issues" from `users_guide/index.md` to eliminate duplicate toctree warnings and align navigation hierarchy with VLA.
- Fix broken cross-references, link syntax, code fences, and typos across the PLUG documentation files.- docs/source/conf.py (version 9b876959674a29dd5f275d64375dd01d13d472fe) (diffs)
- docs/source/users_guide/alma_tasks.rst (version 9b876959674a29dd5f275d64375dd01d13d472fe) (diffs)
- docs/source/users_guide/data-processing-files.md (version 9b876959674a29dd5f275d64375dd01d13d472fe) (diffs)
- docs/source/users_guide/helper-text-files.md (version 9b876959674a29dd5f275d64375dd01d13d472fe) (diffs)
- docs/source/users_guide/index.md (version 9b876959674a29dd5f275d64375dd01d13d472fe) (diffs)
- docs/source/users_guide/known_issues.md (version 9b876959674a29dd5f275d64375dd01d13d472fe) (diffs)
- docs/source/users_guide/lowSNR-workflow.md (version 9b876959674a29dd5f275d64375dd01d13d472fe) (diffs)
- docs/source/users_guide/pipeline-weblog.md (version 9b876959674a29dd5f275d64375dd01d13d472fe) (diffs)
-
Rui Xue c32c5ee6f49ee372db9c51fcd18132440540ac23 m
PIPE-3252: partition dev docs and add contributors page
- partition dev toctrees and `conf.py` exclusions between public and internal builds
- move `dependencies.md` under `devel/setup/` and update cross-references
- add Contributors to Basics navigation with courtesy note
- tweak citation and transition notices in `docs/README.rst`
- clean up "CASA pipeline tasks" terminology across documentation- Contributors.rst (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
- docs/README.rst (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
- docs/source/conf.py (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
- docs/source/contributors.rst (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
- docs/source/devel/reference/callibrary.md (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
- docs/source/devel/setup/dependencies.md (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
- docs/source/devel/setup/index.md (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
- docs/source/devel/testing/diffing_execution_logs.md (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
- docs/source/devel/usage/running_pipeline.md (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
- docs/source/index.rst (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
- docs/source/qa.md (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
- docs/source/users_guide/pipeline-versions-and-documentation.md (version c32c5ee6f49ee372db9c51fcd18132440540ac23) (diffs)
-
Rui Xue c394948a650b5fbd04e48e7d39f16e711270d163 m
Pull request #1990: PIPE-3262
Merge in PIPE/pipeline from PIPE-3262 to main
* commit '8d3c214a77b5638e53eed418fafc0f126617e138':
PIPE-3262: prevent MPI import deadlock and drop `casaplotserver` in pixi
PIPE-3262: avoid unconditional mpi4py import in `daskhelpers.py` -
Rui Xue 8d3c214a77b5638e53eed418fafc0f126617e138 m
PIPE-3262: prevent MPI import deadlock and drop `casaplotserver` in pixi
- Add dynamic `scripts/stubs/casalith.py` to defer MPI initialization to casashell script scope and avoid casatasks import-lock deadlocks (CAS-14037).
- Propagate PYTHONPATH with scripts/stubs in pixi `casa` and `casampi` tasks.
- Remove unused `casaplotserver` dependency from `pyproject.toml` and lockfile. -
Rui Xue 8f568c69ee1cef7d1b872cb1261b5ee41ca6a9bb m
PIPE-3262: avoid unconditional mpi4py import in `daskhelpers.py`
Use `MPIEnvironment.mpi_processor_rank` instead of `mpi4py.MPI.COMM_WORLD.Get_rank()` to prevent unmanaged `MPI_Init` calls and potentially crashes in macOS serial sessions environments with OpenMPI build/version mismatches. -
Rui Xue 34228abc590663d2714f3d8cc7e37e8d124b2293 m
PIPE-3260: Fix false QA failures in `VLASetjy`
- `pipeline/hifv/tasks/setmodel/qa.py`: filter standard calibrators to fields with AMPLITUDE intent, safeguard statistics lookups, and disambiguate MS field names in messages.
- `pipeline/hifv/tasks/setmodel/qa_test.py`: add unit tests covering single/multiple calibrators, pointing aliases, and edge cases. -
Takeshi Nakazato 0660fcc12294974a6420446a231dcac6217fd3ab m
Pull request #1961: PIPE-3206 hsd flagdata offset pointing plots for ephemeris data is not updated after deterministic
Merge in PIPE/pipeline from PIPE-3206-hsd_flagdata-offset-pointing-plots-for-ephemeris-data-is-not-updated-after-deterministic to main
* commit '4f052fb9e0c1e9af3f020512d7b8fb4a0ac52c03':
PIPE-3206 better condition to check if the target is ephemeris -
Takeshi Nakazato 5b6b584ff3fa9d99e0aab0be3f1cbd46b23919e5 m
Pull request #1957: PIPE-3228 clone hsd imaging fail of atm line identification in contamination plots
Merge in PIPE/pipeline from PIPE-3228-clone-hsd_imaging-fail-of-atm-line-identification-in-contamination-plots to main
* commit 'b9106cae3b557a37333cf2492ade928904ba0732':
updated schematic diagram
fix wrong index in docstring
format docstring
cache interpolator object for each baseband
avoid unbound local warning
ensure ascending order
PIPE-3226 fix a bug in interpolating tau
add unit test for SDcalatmcorr -
Rui Xue 8dabe1b5b997da832ea10696bb0a42d11d6c2337 m
PIPE-3258: preserve list type in refant fallback and align testBPdcals
Use list slicing instead of `np.delete` for refant fallback, clean up delay caltables on retry in testBPdcals, and streamline docstrings. -
Rui Xue 15dcc142b4d05281e356875261f733e0c026ef78 m
PIPE-3258: fix reference antenna fallback and caltable cleanup in `hifv_semiFinalBPdcals`
Propagate the updated reference antenna list in the delaycal retry loop, clean up preexisting delay caltables, and guard against empty antenna lists. -
Rui Xue fb50ffeb80b378fda7ec498eff8147372a29d87b m
Pull request #1982: PIPE-3211: update GitHub Actions workflows, add git-resilient helper, and fix doc badge clicks
Merge in PIPE/pipeline from PIPE-3211-update-gh-workflows to main
* commit '2271c07d0be45594b7856f9734d4ee9d92b7ae73':
PIPE-3211: modernize GitHub Actions workflows, add git-resilient helper, and fix doc badge clicks -
Rui Xue 2271c07d0be45594b7856f9734d4ee9d92b7ae73 m
PIPE-3211: modernize GitHub Actions workflows, add git-resilient helper, and fix doc badge clicks
- Add .github/scripts/git-resilient.sh with in-memory tuning, retries, and native HTTP timeouts
- Modernize workflows to Pixi, align artifact actions to @v7, and upgrade to checkout@v5 / cache@v5
- Make Bitbucket scheduled sync resilient to on-prem server downtime
- Remove legacy conda workflows (build-gh-pages.yml, env-setup-run-template.yml, test-unit.yml)
- Unify Pages deployment under official actions and remove third-party peaceiris/actions-gh-pages
- Fix Viewer.js lightbox hijacking on Sphinx doc badges and document LaTeX prerequisites- .github/scripts/git-resilient.sh (version 2271c07d0be45594b7856f9734d4ee9d92b7ae73) (diffs)
- .github/workflows/build-gh-pages-pixi.yml (version 2271c07d0be45594b7856f9734d4ee9d92b7ae73) (diffs)
- .github/workflows/build-gh-pages.yml (version 2271c07d0be45594b7856f9734d4ee9d92b7ae73) (diffs)
- .github/workflows/env-setup-run-template-pixi.yml (version 2271c07d0be45594b7856f9734d4ee9d92b7ae73) (diffs)
- .github/workflows/env-setup-run-template.yml (version 2271c07d0be45594b7856f9734d4ee9d92b7ae73) (diffs)
- .github/workflows/sync-from-open-bitbucket.yml (version 2271c07d0be45594b7856f9734d4ee9d92b7ae73) (diffs)
- .github/workflows/test-unit-pixi.yml (version 2271c07d0be45594b7856f9734d4ee9d92b7ae73) (diffs)
- .github/workflows/test-unit.yml (version 2271c07d0be45594b7856f9734d4ee9d92b7ae73) (diffs)
- docs/source/_static/custom_theme.css (version 2271c07d0be45594b7856f9734d4ee9d92b7ae73) (diffs)
- docs/source/_static/viewer_init.js (version 2271c07d0be45594b7856f9734d4ee9d92b7ae73) (diffs)
- docs/source/devel/setup/readthedocs.md (version 2271c07d0be45594b7856f9734d4ee9d92b7ae73) (diffs)
-
Rui Xue 363f65f480c16c4b39c6e96e23983877248f687f m
Pull request #1981: PIPE-3211: clean up pyproject.toml by removing unnecessary line breaks and comments
Merge in PIPE/pipeline from PIPE-3211-followup to main
* commit 'bfd57924b3a31bacb83ed99ab683e92b53822bfc':
PIPE-3211: clean up pyproject.toml by removing unnecessary line breaks and comments -
Rui Xue bfd57924b3a31bacb83ed99ab683e92b53822bfc m
PIPE-3211: clean up pyproject.toml by removing unnecessary line breaks and comments
-
Rui Xue 62d0284b0eb2bfe226c950f2020117c260320c51 m
Pull request #1973: PIPE-3211
Merge in PIPE/pipeline from PIPE-3211 to main
* commit '050c7ff9597ba670223e7d6e55ce1419bd0f2fc6':
PIPE-3251/PIPE-3252: clarify parallel imaging docstrings
PIPE-3211: update `astropy` dependency to version 8.0
PIPE-3211: remove stale test procedures for `hifa_cal_tier0` and `hifa_calimage_tier0`; should use `scripts/procedure_add_parallel.py` instead.
PIPE-3211: single-source dynamic version from `pipeline._version.py`
PIPE-3211: define CASA aliases upfront and add export option for dependencies
PIPE-3211: update CI/CD workflow trigger path filters
PIPE-3211: prune redundant six dependency and obsolete egg package data
PIPE-3211: prune obsolete and unused documentation dependencies
PIPE-3211: prune redundant and obsolete development dependencies
PIPE-3211: bump runtime dependencies in `pyproject.toml` and update `pixi.lock`
PIPE-3211: fold requirements into `pyproject.toml` and add CASA dependency installer
PIPE-3211: migrate dynamic versioning to in-tree build backend and remove `setup.py`
PIPE-3211: add documentation for weblog resources and developer scripts
PIPE-3211: decouple weblog asset minification from `setup.py`
PIPE-3211/PIPE-2930: Remove duplicated test tasks from `pyproject.toml` and update Numpy version constraint in `requirements.txt` -
Takeshi Nakazato 4f052fb9e0c1e9af3f020512d7b8fb4a0ac52c03 m
Merge branch 'main' into PIPE-3206-hsd_flagdata-offset-pointing-plots-for-ephemeris-data-is-not-updated-after-deterministic
-
Rui Xue 050c7ff9597ba670223e7d6e55ce1419bd0f2fc6 m
PIPE-3251/PIPE-3252: clarify parallel imaging docstrings
- Clarify `parallel` docstrings in makeimlist, checkproductsize, and imageprecheck: document automatic fallback to serial imager (PySynthesisImager) when parallel=True in serial sessions, and remove references to tclean(parallel=True).
- Add note in makeimages docstring citing CASAdocs regarding CASA built-in tclean parallelization behavior for spectral cubes vs. continuum.- pipeline/hif/heuristics/imageparams_vlass_single_epoch_continuum.py (version 050c7ff9597ba670223e7d6e55ce1419bd0f2fc6) (diffs)
- pipeline/hif/tasks/checkproductsize/checkproductsize.py (version 050c7ff9597ba670223e7d6e55ce1419bd0f2fc6) (diffs)
- pipeline/hif/tasks/makeimages/makeimages.py (version 050c7ff9597ba670223e7d6e55ce1419bd0f2fc6) (diffs)
- pipeline/hif/tasks/makeimlist/makeimlist.py (version 050c7ff9597ba670223e7d6e55ce1419bd0f2fc6) (diffs)
- pipeline/hifa/tasks/imageprecheck/imageprecheck.py (version 050c7ff9597ba670223e7d6e55ce1419bd0f2fc6) (diffs)
- pixi.lock (version 050c7ff9597ba670223e7d6e55ce1419bd0f2fc6) (diffs)
-
Rui Xue 7242f0943dc9ff5d969f976c8c4050cdcf2e8ab3 m
PIPE-3211: prune obsolete and unused documentation dependencies
- Remove legacy Python 2/3 compatibility shim future and redundant transitive dependency urllib3 from docs optional-dependencies.
- Remove dead PyPI dummy package nodejs (Node.js is provided by Conda).
- Remove unused Sphinx extensions (nbsphinx, sphinx-autoapi), unused theme (sphinx_rtd_theme), and ipympl widget not active in conf.py.
- Update documentation list in `docs/source/dependencies.md`.
- Synchronize and prune `pixi.lock` across docs environments. -
Rui Xue 05f7e92c5c32b44b1c005b054229de35d298e55b m
PIPE-3211: update `astropy` dependency to version 8.0
-
Rui Xue 74823d13b4eddac41d072131f0e80b908a971467 m
PIPE-3211: update CI/CD workflow trigger path filters
- Replace deleted setup.py and environment*.yml paths with active
project components (scripts/**, pixi.lock, MANIFEST.in).
- Ensure modifications to in-tree build backend, asset bundler,
and dependency installer trigger test and docs workflows.- .github/workflows/build-gh-pages-pixi.yml (version 74823d13b4eddac41d072131f0e80b908a971467) (diffs)
- .github/workflows/build-gh-pages.yml (version 74823d13b4eddac41d072131f0e80b908a971467) (diffs)
- .github/workflows/test-unit-pixi.yml (version 74823d13b4eddac41d072131f0e80b908a971467) (diffs)
- .github/workflows/test-unit.yml (version 74823d13b4eddac41d072131f0e80b908a971467) (diffs)
-
Rui Xue c51df6219e629aa1991514534969533117f189b2 m
PIPE-3211: remove stale test procedures for `hifa_cal_tier0` and `hifa_calimage_tier0`; should use `scripts/procedure_add_parallel.py` instead.
-
Rui Xue c5814b3059e280ff7102a2e49dfea61ca1740b97 m
PIPE-3211: prune redundant and obsolete development dependencies
- Remove legacy linters and formatters (black, isort, flake8, pylint, pydocstyle, pycodestyle) superseded by ruff.
- Remove deprecated pytest-forked (covered by pytest-xdist) and unused CLI inspector johnnydep from dev dependencies.
- Update developer documentation in `docs/source/dependencies.md`.
- Synchronize and prune `pixi.lock` across all environments. -
Rui Xue 79d49872125a4170b64ea819c286bb465459a632 m
PIPE-3211: prune redundant six dependency and obsolete egg package data
- Remove redundant top-level six requirement from pixi pypi-dependencies
(not imported by Pipeline; resolved transitively where needed).
- Remove obsolete '*.egg' pattern from setuptools package-data. -
Rui Xue 99c9b87a741c4f3802da8bb954e72287ba074e1a m
PIPE-3211: single-source dynamic version from `pipeline._version.py`
- Configure setuptools dynamic version via attr = "pipeline._version.version"
in pyproject.toml instead of file = "version".
- Update `scripts/build_backend.py` to eliminate root version file generation,
leaving `pipeline/_version.py` as the single source of truth for both build
and runtime.
- Remove redundant /version ignore from .gitignore and package-data.
- Add docstrings to `build_backend.py` PEP 517/660 hooks and format with Ruff.- .gitignore (version 99c9b87a741c4f3802da8bb954e72287ba074e1a) (diffs)
- pyproject.toml (version 99c9b87a741c4f3802da8bb954e72287ba074e1a) (diffs)
- scripts/README.md (version 99c9b87a741c4f3802da8bb954e72287ba074e1a) (diffs)
- scripts/build_backend.py (version 99c9b87a741c4f3802da8bb954e72287ba074e1a) (diffs)
-
Rui Xue 7d3b841391659d55a82dd233871517f4b0d2dd5a m
PIPE-3211: define CASA aliases upfront and add export option for dependencies
- Move CASA environment shortcuts and aliases (`casa_pip`, `casa_python`)
upfront under Prerequisites in monolithic CASA documentation.
- Add `--export [FILE]` (`-o`) option to `scripts/install_dependencies.py`
to extract dependencies in requirements.txt format.
- Add TOML parser fallback (tomllib -> tomli -> pip._vendor.tomli) in
`scripts/install_dependencies.py` for Python < 3.11 (monolithic CASA) support.
- Document standalone Python 3.11+ tomllib one-liners with standard line wrapping.
- Clarify test execution commands in `daskhelpers_test.py` docstring.
- Cross-reference monolithic CASA environment setup from pytest reference guide.- docs/source/devel/setup/monolithic_casa.md (version 7d3b841391659d55a82dd233871517f4b0d2dd5a) (diffs)
- docs/source/devel/testing/pytest_reference.md (version 7d3b841391659d55a82dd233871517f4b0d2dd5a) (diffs)
- docs/source/qa.md (version 7d3b841391659d55a82dd233871517f4b0d2dd5a) (diffs)
- pipeline/infrastructure/daskhelpers_test.py (version 7d3b841391659d55a82dd233871517f4b0d2dd5a) (diffs)
- scripts/README.md (version 7d3b841391659d55a82dd233871517f4b0d2dd5a) (diffs)
- scripts/install_dependencies.py (version 7d3b841391659d55a82dd233871517f4b0d2dd5a) (diffs)
-
Rui Xue 08dc2a857608a09d87d0209a3a8065e1c22e25a5 m
PIPE-3211: bump runtime dependencies in `pyproject.toml` and update `pixi.lock`
- Bump bdsf to ~=1.14 and drop macOS platform/version constraint now that native wheels support Python 3.13+ on Apple Silicon.
- Bump low-risk runtime dependencies: intervaltree to ~=3.2, mako to ~=1.4, and pypubsub to ==4.0.7.
- Synchronize pixi.lock across all environments. -
Rui Xue 9d3fc84477d803a0c67f325e7375d13af31a6caa m
PIPE-3211/PIPE-2930: Remove duplicated test tasks from `pyproject.toml` and update Numpy version constraint in `requirements.txt`
-
Rui Xue 6919fb4f257da7fffc1aaa1ccb8390336b5e583f m
PIPE-3211: add documentation for weblog resources and developer scripts
- Add `templates/resources/README.md` documenting vendor assets, bundling, and licenses
- Add `scripts/README.md` documenting maintenance and utility scripts
- Fix minor docstring formatting in `scripts/update_references.py` -
Rui Xue 3e14baf569ff0feea0bb288ba1c74d6dab451268 m
PIPE-3211: fold requirements into `pyproject.toml` and add CASA dependency installer
- Declare dependencies and optional extras (dev, docs, exp) statically in `pyproject.toml` per PEP 621, removing dynamic dependency file tables.
- Remove `requirements*.txt` and legacy `Dockerfile`.
- Add `scripts/install_dependencies.py` to allow installing dependencies into monolithic CASA environments without installing Pipeline.
- Update `MANIFEST.in` to include `scripts/install_dependencies.py` in sdists.
- Update developer and setup documentation to reflect `pyproject.toml` as SSoT.- Dockerfile (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- MANIFEST.in (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- docs/source/dependencies.md (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- docs/source/devel/setup/conda_setup.rst (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- docs/source/devel/setup/monolithic_casa.md (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- docs/source/qa.md (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- pixi.lock (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- pyproject.toml (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- requirements.txt (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- requirements_dev.txt (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- requirements_docs.txt (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- requirements_exp.txt (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- scripts/README.md (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- scripts/bundle_assets.py (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
- scripts/install_dependencies.py (version 3e14baf569ff0feea0bb288ba1c74d6dab451268) (diffs)
-
Rui Xue 17c5d5815dfaf728dd7209b4e630cde9ae0a282e m
PIPE-3211: migrate dynamic versioning to in-tree build backend and remove `setup.py`
- Add in-tree PEP 517/660 build backend wrapper (`scripts/build_backend.py`)
delegating to `setuptools.build_meta` to generate NRAO Git versioning
on-demand during build and installation.
- Configure `build-backend = "build_backend"` and `version = { file = "version" }`
in `pyproject.toml`.
- Add `MANIFEST.in` to include `scripts/build_backend.py` in source distributions.
- Remove legacy `setup.py` and ignore generated `/version` in `.gitignore`.
- Document `build_backend.py` in `scripts/README.md`.- .gitignore (version 17c5d5815dfaf728dd7209b4e630cde9ae0a282e) (diffs)
- MANIFEST.in (version 17c5d5815dfaf728dd7209b4e630cde9ae0a282e) (diffs)
- pyproject.toml (version 17c5d5815dfaf728dd7209b4e630cde9ae0a282e) (diffs)
- scripts/README.md (version 17c5d5815dfaf728dd7209b4e630cde9ae0a282e) (diffs)
- scripts/build_backend.py (version 17c5d5815dfaf728dd7209b4e630cde9ae0a282e) (diffs)
- setup.py (version 17c5d5815dfaf728dd7209b4e630cde9ae0a282e) (diffs)
-
Rui Xue 9912dcb94977a770df4c6263bc0ff59a38a03ee9 m
PIPE-3211: decouple weblog asset minification from `setup.py`
- Add `scripts/bundle_assets.py` and track pre-generated minified CSS/JS bundles
- Remove minification commands and jsmin/csscompressor from `setup.py` and build-system requires
- Add bundle-assets pixi task and move minification dependencies to `requirements_dev.txt`- pipeline/infrastructure/renderer/templates/resources/css/all.min.css (version 9912dcb94977a770df4c6263bc0ff59a38a03ee9) (diffs)
- pipeline/infrastructure/renderer/templates/resources/js/pipeline_common.min.js (version 9912dcb94977a770df4c6263bc0ff59a38a03ee9) (diffs)
- pipeline/infrastructure/renderer/templates/resources/js/pipeline_plots.min.js (version 9912dcb94977a770df4c6263bc0ff59a38a03ee9) (diffs)
- pyproject.toml (version 9912dcb94977a770df4c6263bc0ff59a38a03ee9) (diffs)
- requirements_dev.txt (version 9912dcb94977a770df4c6263bc0ff59a38a03ee9) (diffs)
- scripts/bundle_assets.py (version 9912dcb94977a770df4c6263bc0ff59a38a03ee9) (diffs)
- setup.py (version 9912dcb94977a770df4c6263bc0ff59a38a03ee9) (diffs)
-
Rui Xue 29cff76c4fd3245bc64db0053f61774cff205fe0 m
Pull request #1974: PIPE-3251: Fallback to `PySynthesisImager` in serial CASA session when `parallel=True`
Merge in PIPE/pipeline from PIPE-3251-fallback-to-pysynthesisimager-in-a-serial-casa-session-when-parallel-true-for to main
* commit 'b50c8ea978c605e96d1961f213ebad974c87615f':
PIPE-3251: Fallback to `PySynthesisImager` in serial CASA session when `parallel=True` -
Andres Guzman 1f1b1cd216b977a61b80fb30fd4ceee9fd383e70 m
Pull request #1970: PIPE-3240.
Merge in PIPE/pipeline from PIPE-3240-fixes-to-extern-code-tsys_contamination to main
* commit 'c7fb9fc782c4403fad4e2dea6c8ec0b94f4bf205':
PIPE-3240: Fix mp.min typo and handle empty peak prominences in model_break
PIPE-3240. Apply the three changes in the ticket description. -
Rui Xue 44b7d0d26085365533abe8f12f47ddf4778c9ebe m
Pull request #1978: PIPE-3252: update citation example to Pipeline 2026.2.0 and CASA 6.7.4
Merge in PIPE/pipeline from PIPE-3252-update-acknowledgement-to-main to main
* commit '44db00801842e89d4d13c3f5ab24ae640fcfddcc':
PIPE-3252: update citation example to Pipeline 2026.2.0 and CASA 6.7.4 -
Rui Xue 44db00801842e89d4d13c3f5ab24ae640fcfddcc m
PIPE-3252: update citation example to Pipeline 2026.2.0 and CASA 6.7.4
- Bump example citation versions in acknowledgement.md to Pipeline 2026.2.0.27
and CASA 6.7.4-8.
- Clarify cross-reference link to software design overview. -
Rui Xue c7fb9fc782c4403fad4e2dea6c8ec0b94f4bf205 m
PIPE-3240: Fix mp.min typo and handle empty peak prominences in model_break
-
Rui Xue b50c8ea978c605e96d1961f213ebad974c87615f m
PIPE-3251: Fallback to `PySynthesisImager` in serial CASA session when `parallel=True`
- Add fallback in imageparams heuristics to revert to serial imaging mode when parallel=True is requested but no CASA MPI session is active.
- Clarify parallel synthesis imaging docstrings in makeimlist, imageprecheck, and checkproductsize.- pipeline/hif/heuristics/imageparams_base.py (version b50c8ea978c605e96d1961f213ebad974c87615f) (diffs)
- pipeline/hif/heuristics/imageparams_vlass_single_epoch_continuum.py (version b50c8ea978c605e96d1961f213ebad974c87615f) (diffs)
- pipeline/hif/tasks/checkproductsize/checkproductsize.py (version b50c8ea978c605e96d1961f213ebad974c87615f) (diffs)
- pipeline/hif/tasks/makeimlist/makeimlist.py (version b50c8ea978c605e96d1961f213ebad974c87615f) (diffs)
- pipeline/hifa/tasks/imageprecheck/imageprecheck.py (version b50c8ea978c605e96d1961f213ebad974c87615f) (diffs)
-
Rui Xue 0c9ad4a7f5ad701bd618ee484bf476351051b988 m
Pull request #1971: PIPE-2738: update task reference manual title formatting
Merge in PIPE/pipeline from PIPE-2738-main-update-refman-title to main
* commit '8e292b95f92e36ed0cbf0c8f02a3d2acab947b6e':
PIPE-2738: update task reference manual title formatting