Build: #67 failed
Job: Test ManyLinux 2.28 Python 3.12 failed
13a 537 calibration PPr regression: Test case result
The below summarizes the result of the test " 13a 537 calibration PPr regression" in build 67 of PIPESPECS - Test Pipeline main with Casa master - Test ManyLinux 2.28 Python 3.12.
- Description
- 13a 537 calibration PPr regression
- Test class
- tests.regression.fast.vla_fast_test
- Method
- test_13A_537__calibration__PPR__regression
- Duration
- 58 mins
- Status
- Failed (New Failure)
Error Log
Failed: Failed to match 23 result values within tolerances :
s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 15177536
new: 15178738
diff: -1202
percent_diff: -0.007919599070626483%
s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 11498690
new: 11499292
diff: -602
percent_diff: -0.005235378986649783%
s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 3678846
new: 3679446
diff: -600
percent_diff: -0.016309462260719804%
s12.hifv_fluxboot.13A-537.sb24066356.eb24324502.56514.05971091435.flux_densities.spw_0
values differ by > a relative difference of 1e-07
expected: 0.7150094554333849
new: 0.7155448522556886
diff: -0.0005353968223037464
percent_diff: -0.07487968421050169%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 48816984
new: 48818194
diff: -1210
percent_diff: -0.0024786455468039565%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.before
values differ by > a relative difference of 1e-07
expected: 48585580
new: 48586790
diff: -1210
percent_diff: -0.002490450870402288%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 7526954
new: 7526948
diff: 6
percent_diff: 7.971352023673852e-05%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.before
values differ by > a relative difference of 1e-07
expected: 7526954
new: 7526948
diff: 6
percent_diff: 7.971352023673852e-05%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 6902474
new: 6902488
diff: -14
percent_diff: -0.00020282582737725634%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.before
values differ by > a relative difference of 1e-07
expected: 6902474
new: 6902488
diff: -14
percent_diff: -0.00020282582737725634%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 11498690
new: 11499292
diff: -602
percent_diff: -0.005235378986649783%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.before
values differ by > a relative difference of 1e-07
expected: 11498690
new: 11499292
diff: -602
percent_diff: -0.005235378986649783%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 3678846
new: 3679446
diff: -600
percent_diff: -0.016309462260719804%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.before
values differ by > a relative difference of 1e-07
expected: 3678846
new: 3679446
diff: -600
percent_diff: -0.016309462260719804%
s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 29069541
new: 29070651
diff: -1110
percent_diff: -0.003818429744040334%
s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_5.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 9127246
new: 9128970
diff: -1724
percent_diff: -0.018888501526090125%
s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_6.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 7154924
new: 7155223
diff: -299
percent_diff: -0.00417894026547312%
s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_7.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 7755435
new: 7754522
diff: 913
percent_diff: 0.011772389298601561%
s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.mean
values differ by > a relative difference of 1e-07
expected: 0.6294125096532232
new: 0.6287809749109473
diff: 0.0006315347422758411
percent_diff: 0.10033717674658026%
s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.variance
values differ by > a relative difference of 1e-07
expected: 0.06275818592512647
new: 0.06260832106673087
diff: 0.00014986485839560415
percent_diff: 0.2387973077717704%
s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 14429428
new: 14429436
diff: -8
percent_diff: -5.544225315099115e-05%
s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 7526954
new: 7526948
diff: 6
percent_diff: 7.971352023673852e-05%
s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 6902474
new: 6902488
diff: -14
percent_diff: -0.00020282582737725634%
Worst absolute diff, s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_5.num_rows_flagged.after: -1724
Worst percentage diff, s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.variance: 0.2387973077717704%
def test_13A_537__calibration__PPR__regression():
"""Run VLA calibration regression with a PPR file.
PPR name: PPR_13A-537.xml
Dataset: 13A-537/13A-537.sb24066356.eb24324502.56514.05971091435
"""
ref_directory = 'pl-regressiontest/13A-537'
pt = PipelineTester(
visname=['13A-537.sb24066356.eb24324502.56514.05971091435'],
ppr=f'{ref_directory}/PPR_13A-537.xml',
input_dir=ref_directory,
output_dir='13A_537__calibration__PPR__regression',
expectedoutput_dir=ref_directory,
)
> pt.run(telescope='vla', omp_num_threads=1)
tests/regression/fast/vla_fast_test.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/testing_utils.py:376: in run
self.__compare_results(new_file, default_relative_tolerance)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.testing_utils.PipelineTester object at 0x7fed277f26f0>
new_file = '13A-537.sb24066356.eb24324502.56514.05971091435.NEW.results.txt'
relative_tolerance = 1e-07
def __compare_results(self, new_file: str, relative_tolerance: float) -> None:
"""
Compare results between new one loaded from file and old one.
Args:
new_file : file path of new results
relative_tolerance : relative tolerance of output value
"""
with open(self.expectedoutput_file) as expected_fd, open(new_file) as new_fd:
expected_results = expected_fd.readlines()
new_results = new_fd.readlines()
errors = []
worst_diff = (0, 0)
worst_percent_diff = (0, 0)
for old, new in zip(expected_results, new_results):
try:
oldkey, oldval, tol = self.__sanitize_results_string(old)
newkey, newval, _ = self.__sanitize_results_string(new)
except ValueError as e:
errorstr = "The results: {0} could not be parsed. Error: {1}".format(new, str(e))
errors.append(errorstr)
continue
assert oldkey == newkey, f"Expected key {oldkey} does not match new key {newkey}."
tolerance = tol if tol else relative_tolerance
if newval is not None:
LOG.info('Comparing %s to %s with a rel. tolerance of %s', oldval, newval, tolerance)
if oldval != pytest.approx(newval, rel=tolerance):
diff = oldval-newval
percent_diff = (oldval-newval)/oldval * 100 if oldval != 0 else 100
if abs(diff) > abs(worst_diff[0]):
worst_diff = diff, oldkey
if abs(percent_diff) > abs(worst_percent_diff[0]):
worst_percent_diff = percent_diff, oldkey
errorstr = f"{oldkey}\n\tvalues differ by > a relative difference of {tolerance}\n\texpected: {oldval}\n\tnew: {newval}\n\tdiff: {diff}\n\tpercent_diff: {percent_diff}%"
errors.append(errorstr)
elif oldval is not None:
# If only the new value is None, fail
errorstr = f"{oldkey}\n\tvalue is None\n\texpected: {oldval}\n\tnew: {newval}"
errors.append(errorstr)
else:
# If old and new values are both None, this is expected, so pass
LOG.info('Comparing %s and %s... both values are None.', oldval, newval)
[LOG.warning(x) for x in errors]
n_errors = len(errors)
if n_errors > 0:
summary_str = f"Worst absolute diff, {worst_diff[1]}: {worst_diff[0]}\nWorst percentage diff, {worst_percent_diff[1]}: {worst_percent_diff[0]}%"
errors.append(summary_str)
> pytest.fail("Failed to match {0} result value{1} within tolerance{1} :\n{2}".format(
n_errors, '' if n_errors == 1 else 's', '\n'.join(errors)), pytrace=True)
E Failed: Failed to match 23 result values within tolerances :
E s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 15177536
E new: 15178738
E diff: -1202
E percent_diff: -0.007919599070626483%
E s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 11498690
E new: 11499292
E diff: -602
E percent_diff: -0.005235378986649783%
E s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 3678846
E new: 3679446
E diff: -600
E percent_diff: -0.016309462260719804%
E s12.hifv_fluxboot.13A-537.sb24066356.eb24324502.56514.05971091435.flux_densities.spw_0
E values differ by > a relative difference of 1e-07
E expected: 0.7150094554333849
E new: 0.7155448522556886
E diff: -0.0005353968223037464
E percent_diff: -0.07487968421050169%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 48816984
E new: 48818194
E diff: -1210
E percent_diff: -0.0024786455468039565%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.before
E values differ by > a relative difference of 1e-07
E expected: 48585580
E new: 48586790
E diff: -1210
E percent_diff: -0.002490450870402288%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 7526954
E new: 7526948
E diff: 6
E percent_diff: 7.971352023673852e-05%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.before
E values differ by > a relative difference of 1e-07
E expected: 7526954
E new: 7526948
E diff: 6
E percent_diff: 7.971352023673852e-05%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 6902474
E new: 6902488
E diff: -14
E percent_diff: -0.00020282582737725634%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.before
E values differ by > a relative difference of 1e-07
E expected: 6902474
E new: 6902488
E diff: -14
E percent_diff: -0.00020282582737725634%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 11498690
E new: 11499292
E diff: -602
E percent_diff: -0.005235378986649783%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.before
E values differ by > a relative difference of 1e-07
E expected: 11498690
E new: 11499292
E diff: -602
E percent_diff: -0.005235378986649783%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 3678846
E new: 3679446
E diff: -600
E percent_diff: -0.016309462260719804%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.before
E values differ by > a relative difference of 1e-07
E expected: 3678846
E new: 3679446
E diff: -600
E percent_diff: -0.016309462260719804%
E s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 29069541
E new: 29070651
E diff: -1110
E percent_diff: -0.003818429744040334%
E s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_5.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 9127246
E new: 9128970
E diff: -1724
E percent_diff: -0.018888501526090125%
E s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_6.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 7154924
E new: 7155223
E diff: -299
E percent_diff: -0.00417894026547312%
E s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_7.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 7755435
E new: 7754522
E diff: 913
E percent_diff: 0.011772389298601561%
E s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.mean
E values differ by > a relative difference of 1e-07
E expected: 0.6294125096532232
E new: 0.6287809749109473
E diff: 0.0006315347422758411
E percent_diff: 0.10033717674658026%
E s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.variance
E values differ by > a relative difference of 1e-07
E expected: 0.06275818592512647
E new: 0.06260832106673087
E diff: 0.00014986485839560415
E percent_diff: 0.2387973077717704%
E s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 14429428
E new: 14429436
E diff: -8
E percent_diff: -5.544225315099115e-05%
E s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 7526954
E new: 7526948
E diff: 6
E percent_diff: 7.971352023673852e-05%
E s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 6902474
E new: 6902488
E diff: -14
E percent_diff: -0.00020282582737725634%
E Worst absolute diff, s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_5.num_rows_flagged.after: -1724
E Worst percentage diff, s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.variance: 0.2387973077717704%
tests/testing_utils.py:435: Failed