Build #9

Plan branch:

Build: #9 failed Changes by Shawn Booth

Build result summary

Details

Completed
Queue duration
< 1 second
Duration
2 minutes
Labels
None
Agent
cbt-pipeline-test.cv.nrao.edu
Revision
35c7e299b3a096a0f12a1c26ce3ba5b9a8f83a73
Fixed in
#10 (Changes by Shawn Booth)
No failed test found. A possible compilation error occurred.

Responsible

  • Shawn Booth Shawn Booth Automatically assigned

Code commits

Author Commit Message Commit date
Shawn Booth Shawn Booth 35c7e299b3a096a0f12a1c26ce3ba5b9a8f83a73 Phase 4: Implement results slimming and delta metadata with investigation tooling
IMPLEMENTATION
--------------
Results Slimming Infrastructure:
- Added Results.slim_for_pickle() method to remove transient fields before serialization
- Implemented _transient_fields support on 5 flagging task result classes:
  * BandpassflagResults, GfluxscaleflagResults, PolcalflagResults
  * TargetflagResults, TsysflagResults
- Modified ResultsProxy.write() to automatically slim results during pickle dump

Metadata Delta Tracking:
- Implemented ContextDeltaMetadata dataclass for per-stage progression tracking
- Added export_context_delta_metadata() to generate stage-level YAML deltas
- Created Context.save_metadata() API for lightweight metadata-only snapshots
- Per-stage exports: context-stageN-delta.yaml, context-stageN-metadata.yaml

Investigation Tooling:
- audit_storage_reduction.py: Component-by-component storage analysis
- check_dataset_size.py: Dataset suitability checker for optimization testing
- drill_imaging_products.py: Detailed imaging product size breakdown
- estimate_results_slimming.py: Post-hoc slimming benefit estimation
- debug_pickle_contents.py: Pickle content inspection utility

TESTING & VALIDATION
--------------------
Controlled testing performed on multiple datasets:
- Small dataset (ALMA-IF, 914 KB context, 48 stages): No measurable improvement
- Large dataset (44 MB context, 56 stages, 24 GB results): 0% storage savings

Result pickle slimming analysis on production data:
- 21 GB stage 4 pickle: 0.0% reduction
- 1.5 GB stage 7 pickle: 0.0% reduction
- 944 MB stages 13/19: 0.1% reduction (within measurement noise)

Findings: Large result pickles are dominated by calibration data/arrays, not plots.
Phase 4 optimizations provide no measurable benefit on tested datasets.

DOCUMENTATION
-------------
- Updated EXECUTIVE_SUMMARY.md with Phase 4 status and testing results
- Updated INVESTIGATION_SUMMARY.md with honest assessment
- Updated CONTEXT_STORAGE_INVESTIGATION.md with implementation details
- Updated COMPLETE_IMPLEMENTATION_SUMMARY.md with negative results section
- Created PHASE4_RESULTS.md with comprehensive testing metrics

FILES CHANGED
-------------
Core implementation: 5 result objects + 3 infrastructure files (242 lines)
Investigation tools: 5 new utilities (934 lines)
Documentation: 5 markdown files updated (268 lines)
Total: 18 files, 1,444 insertions

RECOMMENDATION
--------------
While Phase 4 infrastructure is correctly implemented with zero functional
regressions, controlled testing reveals no storage or performance improvements
on representative datasets. This represents honest negative results from a
properly executed investigation.

The investigation tooling and testing methodology developed during this phase
remain valuable for future context optimization work.

Error summary for Pipeline PR Test 6.6.6

The job generated some errors, drill down into the full build log for more details.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   463  100   463    0     0   2102      0 --:--:-- --:--:-- --:--:--  2153
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0  966M    0  7716    0     0  32895      0  8:33:43 --:--:--  8:33:43 33547
  6  966M    6 64.4M    0     0  53.1M      0  0:00:18  0:00:01  0:00:17 53.3M
15  966M   15  147M    0     0  66.8M      0  0:00:14  0:00:02  0:00:12 66.9M
23  966M   23  225M    0     0  70.2M      0  0:00:13  0:00:03  0:00:10 70.3M
32  966M   32  311M    0     0  74.0M      0  0:00:13  0:00:04  0:00:09 74.1M
40  966M   40  395M    0     0  75.9M      0  0:00:12  0:00:05  0:00:07 79.5M
49  966M   49  482M    0     0  77.6M      0  0:00:12  0:00:06  0:00:06 83.5M
58  966M   58  568M    0     0  78.8M      0  0:00:12  0:00:07  0:00:05 84.1M
67  966M   67  651M    0     0  79.3M      0  0:00:12  0:00:08  0:00:04 85.1M
75  966M   75  734M    0     0  79.7M      0  0:00:12  0:00:09  0:00:03 84.5M
84  966M   84  817M    0     0  80.0M      0  0:00:12  0:00:10  0:00:02 84.2M
93  966M   93  899M    0     0  80.2M      0  0:00:12  0:00:11  0:00:01 83.4M
100  966M  100  966M    0     0  80.4M      0  0:00:12  0:00:12 --:--:-- 82.8M
Cloning into 'casa-build-utils'...
  WARNING: The script distro is installed in '/home/casatest/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: You are using pip version 20.2.3; however, version 25.0.1 is available.
You should consider upgrading via the '/usr/local/bin/python3.8 -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "testpipeline.py", line 62, in <module>
    raise Exception("Merge failed")
Exception: Merge failed
mkdir: cannot create directory 'workdir': File exists
cp: cannot stat '/home/casatest/casa-build-utils/pipeline/workdir/pipeline_test_result.xml': No such file or directory
chown: cannot access '/pkgout/*': No such file or directory
Error response from daemon: No such container: pipeline-test
Error response from daemon: No such container: pipeline-test