Pipeline

• 

  Rui Xue 4 days ago 058d8aeea34d9aecc6b4a6a172339e1e4dea7d4a

  Pull request #1810: PIPE-3085: reduce MS open calls
  Merge in PIPE/pipeline from PIPE-3085-improve-the-freq-frame--chan-conversion-speed-within-fluxcalflag to main
  
  * commit 'ad315c171f1d6b940df0fe50f4c392b23effb04e':
    PIPE-3085: Replace singleton ms tool with MSReader; add type hints
    PIPE-3085: Extend MS open optimization to `correctedampflag`, `rawflagchans`, `gcorfluxscale`
    PIPE-3085: Reduce MS open overhead in `hif_applycal` weblog rendering
    PIPE-3085: hifa_fluxcalflag - cache SPW freq grids and use searchsorted in _get_chanrange
• 

  Rui Xue 4 days ago ad315c171f1d6b940df0fe50f4c392b23effb04e m

  PIPE-3085: Replace singleton ms tool with MSReader; add type hints

  • pipeline/h/tasks/common/mstools.py (version ad315c171f1d6b940df0fe50f4c392b23effb04e) (diffs)
  • pipeline/hifa/tasks/fluxcalflag/fluxcalflag.py (version ad315c171f1d6b940df0fe50f4c392b23effb04e) (diffs)
• 

  Rui Xue 11 May 2026 454b91e80c6962a09f89ae57b094da0fbc1bbc61 m

  PIPE-3085: Extend MS open optimization to `correctedampflag`, `rawflagchans`, `gcorfluxscale`
  Hoist casa_tools.MSReader outside the inner loops in three additional
  tasks, following the same pattern established for hif_applycal weblog
  rendering:
  
  - hif_correctedampflag: open once for the intent × field × spw loop
    in _run_flagging_iteration; thread ms_handle through
    _evaluate_heuristic → _evaluate_heuristic_for_baseline_set →
    mstools.read_channel_averaged_data_from_ms.
  - hif_rawflagchans: open once for the spw loop in
    RawflagchansData.prepare; call openms.reset() between iterations
    instead of opening a fresh handle per SPW.
  - hifa_gfluxscale: open once for the field × spw loop in
    _derive_calvis_flux; pass ms_handle to mstools.compute_mean_flux.
  
  Also: bare except: → except Exception:, lazy % log formatting.

  • pipeline/hif/tasks/correctedampflag/correctedampflag.py (version 454b91e80c6962a09f89ae57b094da0fbc1bbc61) (diffs)
  • pipeline/hif/tasks/rawflagchans/rawflagchans.py (version 454b91e80c6962a09f89ae57b094da0fbc1bbc61) (diffs)
  • pipeline/hifa/tasks/fluxscale/gcorfluxscale.py (version 454b91e80c6962a09f89ae57b094da0fbc1bbc61) (diffs)
• 

  Rui Xue 11 May 2026 7673073843ca00326b758194ef083e23c7573149 m

  PIPE-3085: Reduce MS open overhead in `hif_applycal` weblog rendering
  Reduce the cost of opening measurement sets in get_brightest_field()
  by reusing a single MS handle across all (field, spw) iterations instead
  of opening and closing the MS once per combination. On lazy-import data
  sets, where the DATA column is ASDM-backed, this cuts the applycal
  post-processing time significantly (observed ~50 min saving on a
  2496-call dataset).
  
  - renderer.py: open MS once with casa_tools.MSReader and pass handle
    down to mstools.compute_mean_flux via the new ms_handle kwarg.
  - mstools.py: add keyword-only ms_handle parameter to both
    read_channel_averaged_data_from_ms and compute_mean_flux; call
    openms.reset() before each reuse to clear prior selection state.

  • pipeline/h/tasks/applycal/renderer.py (version 7673073843ca00326b758194ef083e23c7573149) (diffs)
  • pipeline/h/tasks/common/mstools.py (version 7673073843ca00326b758194ef083e23c7573149) (diffs)
• 

  Rui Xue 04 May 2026 50fed352aa45cc77be74ca0dc00d3c028d594bfc m

  PIPE-3085: hifa_fluxcalflag - cache SPW freq grids and use searchsorted in _get_chanrange
  Open the MS once per prepare() call instead of once per line candidate,
  and cache the cvelfreqs result per (vis, fieldid, spwid, refframe) via a
  new _get_spw_freqs helper. Replace the O(N) linear channel scan with
  np.searchsorted for O(log N) boundary lookup. Also convert %-style LOG
  calls to lazy-format style and raise ValueError instead of Exception.

  • pipeline/hifa/tasks/fluxcalflag/fluxcalflag.py (version 50fed352aa45cc77be74ca0dc00d3c028d594bfc) (diffs)