PIPE-3073: Add pipeline CLI entry point (python -m pipeline run/submit)
Add pipeline/__main__.py providing two subcommands:
- `run CONFIG_YAML` — run the pipeline locally (blocking); dispatches via
pipeconfig.ppr (executeppr) or pipeconfig.vis (recipereducer)
- `submit CONFIG_YAML` — generate an sbatch script from pipeconfig.dask.jobqueue.slurm
and optionally call sbatch (non-blocking)
Both subcommands share --loglevel / --plotlevel / --importonly flags.
`run` also creates a rawdata symlink in the CWD (override via pipeconfig.rawdata).
Prefect API URL is applied before any pipeline import so Prefect picks up the
correct server URL from pipeconfig.prefect.api_url at settings-cache time.
Guard Dask autostart and the host environment banner in __init__.py behind a
sys.argv check (_cli_dask_allowed): only `pipeline run` permits autostart;
`--help`, `submit`, and bare invocations skip both.
Register the `pipeline` console script in pyproject.toml.
Add pipeconfig.rawdata to config.yaml defaults.
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 56745139e
# Last commands done (46 commands done):
# pick df35f17c1 PIPE-3073: Resolve host and dashboard address with fallback to local IP
# pick 0cb96bdec PIPE-3073: Add pipeline CLI entry point (python -m pipeline run/submit)
# Next commands to do (2 remaining commands):
# pick 92112577e PIPE-3073: Fix Prefect duplicate task run key and skip Dask for single-item lists
# pick d9e72b8cd PIPE-3073: update task_run_name lambda for new Prefect API
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pipeline/__init__.py
# new file: pipeline/__main__.py
# modified: pipeline/config.yaml
# modified: pyproject.toml
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
#
PIPE-3073: Fix Prefect duplicate task run key and skip Dask for single-item lists
- prefect_adapter: append UUID suffix to task_run_name to avoid
unique constraint violation on uq_task_run__flow_run_id_task_key_dynamic_key
when Prefect server restarts and client retries a task submission
- daskhelpers: skip Dask cluster for single-item inputs in parallel_map
to avoid unnecessary overhead when parallelism provides no benefit
PIPE-3073: update task_run_name lambda for new Prefect API
Prefect's resolve_custom_task_run_name now uses inspect.signature to
detect whether a task_run_name callable accepts a 'parameters' kwarg.
If not, it calls the function with zero arguments, causing:
PIPE-3073: Add `cloudpickle` dependency for extended pickling support
Co-authored-by: Copilot <copilot@github.com>
# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# Author: Rui Xue <rxue@nrao.edu>
#
# interactive rebase in progress; onto 56745139e
# Last commands done (43 commands done):
# pick b2c829ee6 PIPE-3073: replace `pclean` with generic MPI dispatch via `mpirunner`
# pick e1b446e58 PIPE-3073: Add `cloudpickle` dependency for extended pickling support
# Next commands to do (5 remaining commands):
# pick 9d897bcc9 PIPE-3073: Update Dask cluster host configuration to use fully-qualified hostname if not set
# pick df35f17c1 PIPE-3073: Resolve host and dashboard address with fallback to local IP
# You are currently rebasing branch 'PIPE-3073-prefect-workflow-task-adapter-and-tier0future-backend-improvement' on '56745139e'.
#
# Changes to be committed:
# modified: pyproject.toml
#
# Untracked files:
# applycalQA_outliers.txt
# disk_usage_report.csv
# docs/source/develdocmd/
# flux.csv
# notes/
# pipe2903_diff.patch
# scratch_prefect_name.py
# scripts/cli.md
# scripts/demo_prefect_tier0futures_old.py
# scripts/if2016.yaml
# scripts/pltest1.yaml
# scripts/pltest1_old.yaml
# scripts/polcal.yaml
# scripts/run_prefect_pltest1.py
# scripts/run_prefect_pltest1.sh
# scripts/run_prefect_server.sh
# scripts/run_pytest.sh
# test_hang.py
#
