Pull request #1810: PIPE-3085: reduce MS open calls
Merge in PIPE/pipeline from PIPE-3085-improve-the-freq-frame--chan-conversion-speed-within-fluxcalflag to main
* commit 'ad315c171f1d6b940df0fe50f4c392b23effb04e': PIPE-3085: Replace singleton ms tool with MSReader; add type hints PIPE-3085: Extend MS open optimization to `correctedampflag`, `rawflagchans`, `gcorfluxscale` PIPE-3085: Reduce MS open overhead in `hif_applycal` weblog rendering PIPE-3085: hifa_fluxcalflag - cache SPW freq grids and use searchsorted in _get_chanrange
PIPE-3085: Extend MS open optimization to `correctedampflag`, `rawflagchans`, `gcorfluxscale`
Hoist casa_tools.MSReader outside the inner loops in three additional
tasks, following the same pattern established for hif_applycal weblog
rendering:
- hif_correctedampflag: open once for the intent × field × spw loop
in _run_flagging_iteration; thread ms_handle through
_evaluate_heuristic → _evaluate_heuristic_for_baseline_set →
mstools.read_channel_averaged_data_from_ms.
- hif_rawflagchans: open once for the spw loop in
RawflagchansData.prepare; call openms.reset() between iterations
instead of opening a fresh handle per SPW.
- hifa_gfluxscale: open once for the field × spw loop in
_derive_calvis_flux; pass ms_handle to mstools.compute_mean_flux.
Also: bare except: → except Exception:, lazy % log formatting.
PIPE-3085: Reduce MS open overhead in `hif_applycal` weblog rendering
Reduce the cost of opening measurement sets in get_brightest_field()
by reusing a single MS handle across all (field, spw) iterations instead
of opening and closing the MS once per combination. On lazy-import data
sets, where the DATA column is ASDM-backed, this cuts the applycal
post-processing time significantly (observed ~50 min saving on a
2496-call dataset).
- renderer.py: open MS once with casa_tools.MSReader and pass handle
down to mstools.compute_mean_flux via the new ms_handle kwarg.
- mstools.py: add keyword-only ms_handle parameter to both
read_channel_averaged_data_from_ms and compute_mean_flux; call
openms.reset() before each reuse to clear prior selection state.
PIPE-3085: hifa_fluxcalflag - cache SPW freq grids and use searchsorted in _get_chanrange
Open the MS once per prepare() call instead of once per line candidate,
and cache the cvelfreqs result per (vis, fieldid, spwid, refframe) via a
new _get_spw_freqs helper. Replace the O(N) linear channel scan with
np.searchsorted for O(log N) boundary lookup. Also convert %-style LOG
calls to lazy-format style and raise ValueError instead of Exception.
Configuration changes
Plan configuration has changed since the last successful build. See the plan audit log for more details.
AssertionError: Expected key s21.hif_applycal.uid___A002_X1199f9e_X7c24.spw_31.qa.metric.phase_vs_freqslope does not match new key s21.hif_applycal.uid___A002_X1199f9e_X7c24.spw_31.qa.metric.phase_vs_freqintercept.
@pytest.mark.seven
@pytest.mark.mpi
def test_2023_1_00228_S__uid___A002_X1199f9e_X7c24__procedure_hifa_calimage_diffgain__regression():
"""Run ALMA cal+image regression on a 7m B2B dataset with differential gain calibration.
Recipe name: procedure_hifa_calimage_diffgain
Dataset: 2023.1.00228.S: uid___A002_X1199f9e_X7c24
(50 more lines...)
Failed: Failed to match 29 result values within tolerances :
s21.hifa_gfluxscale.uid___A002_X10b6f7c_X41d1.field_0.spw_16.qa.metric.score_gfluxscale_k_spw
values differ by > a relative difference of 1e-07
expected: 0.9998218986723444
new: 0.9998227851376125
diff: -8.864652680795615e-07
percent_diff: -8.866231768444877e-05%
s21.hifa_gfluxscale.uid___A002_X10b6f7c_X41d1.field_0.spw_18.I
(432 more lines...)
Luca Di Mascolo
Automatically assigned