Pipeline

• 

  Shawn Booth 03 Feb 2026 d985fec6bf51b70c6b92bf7d729b856940d3ed4e

  Merge remote-tracking branch 'origin/context-serializer' into context-serializer
• 

  Shawn Booth 03 Feb 2026 bd92715f277279b406681e3368fb9aa06fc95ae4 m

  Implement Phase 2: weblog rendering optimization with per-renderer caching
  Optimize pipeline weblog generation through strategic in-memory caching and
  precomputation of expensive plot data. Move computation from render-time to
  execution-time for applycal and importdata tasks.
  
  Key improvements:
  - Per-renderer timing instrumentation (htmlrenderer.py WebLogGenerator.render)
  - T2_1DetailsRenderer: Cache expensive plot charts (IntentVsTime, FieldVsTime, etc)
  - T2_2_3Renderer: Cache antenna setup plots (PlotAntsChart, UVChart)
  - T2_2_4Renderer: Cache sky setup plots (AzElChart, SunTrackChart, etc)
  - Applycal: Precompute calapps/caltypes during merge_with_context, store in metadata
  - Importdata: Precompute summary rows and flux table data, store in metadata
  - ALMA Importdata: Precompute parallactic angle plots for caching
  
  Performance results:
  - Phase 1 (metadata export): 120 min → 57 min (52.5% improvement)
  - Phase 2 (caching optimizations): 57 min → 50 min (12% additional improvement)
  - Total: 58.3% pipeline runtime reduction (70 minutes saved)
  
  Modified files:
  - pipeline/infrastructure/renderer/htmlrenderer.py: Timing and caching
  - pipeline/h/tasks/applycal/applycal.py: _cache_weblog_data() precomputation
  - pipeline/h/tasks/applycal/renderer.py: Cache-aware rendering
  - pipeline/h/tasks/importdata/importdata.py: Summary data precomputation
  - pipeline/h/tasks/importdata/renderer.py: Cache-aware rendering
  - pipeline/h/templates/importdata.mako: Cache support
  - pipeline/hifa/tasks/importdata/almaimportdata.py: Parang plot caching
  - pipeline/hifa/tasks/importdata/renderer.py: Cache-aware rendering
  - pipeline/infrastructure/launcher.py: Metadata export robustness
  - pipeline/infrastructure/serializers.py: Added load_result_metadata()

  • pipeline/h/tasks/applycal/applycal.py (version bd92715f277279b406681e3368fb9aa06fc95ae4) (diffs)
  • pipeline/h/tasks/applycal/renderer.py (version bd92715f277279b406681e3368fb9aa06fc95ae4) (diffs)
  • pipeline/h/tasks/importdata/importdata.py (version bd92715f277279b406681e3368fb9aa06fc95ae4) (diffs)
  • pipeline/h/tasks/importdata/renderer.py (version bd92715f277279b406681e3368fb9aa06fc95ae4) (diffs)
  • pipeline/h/templates/importdata.mako (version bd92715f277279b406681e3368fb9aa06fc95ae4) (diffs)
  • pipeline/hifa/tasks/importdata/almaimportdata.py (version bd92715f277279b406681e3368fb9aa06fc95ae4) (diffs)
  • pipeline/hifa/tasks/importdata/renderer.py (version bd92715f277279b406681e3368fb9aa06fc95ae4) (diffs)
  • pipeline/infrastructure/launcher.py (version bd92715f277279b406681e3368fb9aa06fc95ae4) (diffs)
  • pipeline/infrastructure/renderer/htmlrenderer.py (version bd92715f277279b406681e3368fb9aa06fc95ae4) (diffs)
  • pipeline/infrastructure/serializers.py (version bd92715f277279b406681e3368fb9aa06fc95ae4) (diffs)