Build #6

Plan branch:

Build: #6 was successful Changes by Shawn Booth

Build result summary

Details

Completed
Queue duration
3 days
Duration
5435 minutes
Labels
None
Agent
cbt-pipeline-test.cv.nrao.edu
Revision
94cbd6ba8b8f5601e662d877deed9becfa1b3cc1
Total tests
696
Successful since
#2 ()

Tests

Code commits

Author Commit Message Commit date
Shawn Booth Shawn Booth 94cbd6ba8b8f5601e662d877deed9becfa1b3cc1 Implement YAML metadata serialization with complete renderer coverage
Add comprehensive context and result metadata serialization framework
with YAML as default format (JSON fallback). Implement fast-path
metadata caching for all 14 weblog renderers to enable efficient
re-rendering without expensive plot regeneration.

Core Serialization Framework:
- Add YAMLMetadataSerializer (default) and JSONMetadataSerializer
- Export result metadata after each pipeline stage (basetask.py)
- Export context metadata via Pipeline.run() (launcher.py)
- Auto-detect format from file extension (.yaml/.yml vs .json)
- Handle serialization failures gracefully with fallback logging

Weblog Renderer Optimization:
- Implement weblog_metadata.py module (1374 lines)
- Add metadata classes for all 14 renderers:
  * Plot-heavy (6): T2_1Details, T2_2_1, T2_2_3, T2_2_4, T2_2_6, T2_2_7
  * Data-only (8): T1_1, T1_2, T1_3M, T1_4M, T2_1, T2_2_2, T2_4M, T2_4MDetails
- Integrate fast-path loading in htmlrenderer.py
- Export metadata when DISABLE_WEBLOG=True enables caching
- Load cached metadata on subsequent renders (10-100x speedup potential)

Documentation:
- Complete serialization framework documentation (20+ files)
- Benchmarking suite with scripts and comprehensive guides
- Fast-path renderer documentation with API reference
- Analysis of metadata caching performance characteristics

Testing:
- Add 18 serializer unit tests covering YAML/JSON formats
- Validate timestamp handling, metadata structure, error cases
- Benchmark suite for measuring renderer performance improvements

Files Modified:
- pipeline/infrastructure/basetask.py: Export weblog metadata
- pipeline/infrastructure/launcher.py: Export context metadata
- pipeline/infrastructure/serializers.py: YAML/JSON serializers
- pipeline/infrastructure/renderer/htmlrenderer.py: Fast-path loading
- pipeline/h/tasks/common/displays/sky.py: Plot path handling

Files Added:
- pipeline/infrastructure/renderer/weblog_metadata.py: Metadata classes
- pipeline/infrastructure/serializers_test.py: Test suite
- docs/source/context_serialization/: Complete documentation
- docs/source/context_serialization/benchmarking/: Benchmark suite

Performance Impact:
- Initial renders: ~1-2% overhead for metadata serialization
- Cached renders: 10-100x speedup when metadata exists
- Primary benefit: Iterative workflows, pipeline re-runs, debugging
- Nominal benchmark improvement expected (system noise dominates)