Code commits
Pipeline
-
Rui Xue b035ba6bc554bc352a36d2fa7c564fed72fb3716
PIPE-3073: demo_prefect_tier0futures: update config and robustness fixes
- Update dataset paths to 2016.1.00053.S on NAASC lustre and introduce
store_dir variable so ppr_file, raw_dir, and work_dir share a common
base prefix
- Add prefect_api_url config variable; set PREFECT_API_URL before pipeline
imports so flow runs are visible in the running Prefect server instead
of going to the ephemeral in-process API
- Fix rawdata symlink handling: always recreate the symlink (unlink if
it exists); if rawdata is a real directory, rename it with a timestamp
backup before symlinking
- Override SIGINT handler after imports to call os._exit(130), bypassing
the dask_jobqueue SIGINT→sys.exit→atexit→cluster.close() deadlock that
caused Ctrl+C to hang
pipeline/config.yaml:
- Increase SLURM job memory from 10gb to 20gb
pipeline/infrastructure/daskhelpers.py:
- Raise default local worker cap from 4 to 6