Build: #5 failed
Job: Test Many Linux 2.28 failed
13a 537 calibration PPr regression: Test case result
The below summarizes the result of the test " 13a 537 calibration PPr regression" in build 5 of Pipeline - Pipeline Main with Casa 6.6.6 test - cvpost - release-6.6.6 - Test Many Linux 2.28.
- Description
- 13a 537 calibration PPr regression
- Test class
- pipeline.infrastructure.utils.regression-tester
- Method
- test_13A_537__calibration__PPR__regression
- Duration
- 50 mins
- Status
- Failed (Existing Failure)
Error Log
Failed: Failed to match 23 result values within tolerances :
s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 15177530
new: 15177502
diff: 28
percent_diff: 0.00018448324595635784%
s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 11498684
new: 11498668
diff: 16
percent_diff: 0.0001391463579658333%
s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 3678846
new: 3678834
diff: 12
percent_diff: 0.00032618924521439605%
s12.hifv_fluxboot.13A-537.sb24066356.eb24324502.56514.05971091435.flux_densities.spw_0
values differ by > a relative difference of 1e-07
expected: 0.7150194812182905
new: 0.7150122744292318
diff: 7.2067890587446115e-06
percent_diff: 0.001007915063581943%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 48816924
new: 48816938
diff: -14
percent_diff: -2.8678578765020097e-05%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.before
values differ by > a relative difference of 1e-07
expected: 48585520
new: 48585534
diff: -14
percent_diff: -2.881516962255421e-05%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 7526936
new: 7526946
diff: -10
percent_diff: -0.00013285618477425608%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.before
values differ by > a relative difference of 1e-07
expected: 7526936
new: 7526946
diff: -10
percent_diff: -0.00013285618477425608%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 6902438
new: 6902470
diff: -32
percent_diff: -0.00046360430908615186%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.before
values differ by > a relative difference of 1e-07
expected: 6902438
new: 6902470
diff: -32
percent_diff: -0.00046360430908615186%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 11498684
new: 11498668
diff: 16
percent_diff: 0.0001391463579658333%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.before
values differ by > a relative difference of 1e-07
expected: 11498684
new: 11498668
diff: 16
percent_diff: 0.0001391463579658333%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 3678846
new: 3678834
diff: 12
percent_diff: 0.00032618924521439605%
s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.before
values differ by > a relative difference of 1e-07
expected: 3678846
new: 3678834
diff: 12
percent_diff: 0.00032618924521439605%
s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 29068855
new: 29069200
diff: -345
percent_diff: -0.0011868372524476798%
s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_5.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 9126917
new: 9126923
diff: -6
percent_diff: -6.573961393535189e-05%
s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_6.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 7154834
new: 7154943
diff: -109
percent_diff: -0.001523445547443868%
s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_7.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 7755168
new: 7755398
diff: -230
percent_diff: -0.002965764249078808%
s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.mean
values differ by > a relative difference of 1e-07
expected: 0.6293920371440289
new: 0.629412100079294
diff: -2.0062935265130477e-05
percent_diff: -0.0031876690649232523%
s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.variance
values differ by > a relative difference of 1e-07
expected: 0.06275352250234698
new: 0.06275879799932889
diff: -5.275496981912986e-06
percent_diff: -0.008406694591073644%
s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 14429374
new: 14429416
diff: -42
percent_diff: -0.00029107291834004715%
s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 7526936
new: 7526946
diff: -10
percent_diff: -0.00013285618477425608%
s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.after
values differ by > a relative difference of 1e-07
expected: 6902438
new: 6902470
diff: -32
percent_diff: -0.00046360430908615186%
Worst absolute diff, s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after: -345
Worst percentage diff, s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.variance: -0.008406694591073644%
@pytest.mark.vla
@pytest.mark.fast
def test_13A_537__calibration__PPR__regression():
"""Run VLA calibration regression with a PPR file.
PPR name: PPR_13A-537.xml
Dataset: 13A-537/13A-537.sb24066356.eb24324502.56514.05971091435
"""
input_dir = 'pl-regressiontest/13A-537'
pr = PipelineRegression(input_dir=input_dir,
visname=['13A-537.sb24066356.eb24324502.56514.05971091435'],
expectedoutput_dir=input_dir,
output_dir='13A_537__calibration__PPR__regression')
> pr.run(ppr=f'{input_dir}/PPR_13A-537.xml', telescope='vla', omp_num_threads=1)
pipeline/infrastructure/utils/regression-tester.py:625:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pipeline/infrastructure/utils/regression-tester.py:224: in run
self.__compare_results(new_file, default_relative_tolerance)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pipeline.infrastructure.utils.regression-tester.PipelineRegression object at 0x7fc370f915e0>
new_file = '13A-537.sb24066356.eb24324502.56514.05971091435.NEW.results.txt'
relative_tolerance = 1e-07
def __compare_results(self, new_file: str, relative_tolerance: float):
"""
Compare results between new one loaded from file and old one.
Args:
new_file : file path of new results
relative_tolerance : relative tolerance of output value
"""
with open(self.expectedoutput_file) as expected_fd, open(new_file) as new_fd:
expected_results = expected_fd.readlines()
new_results = new_fd.readlines()
errors = []
worst_diff = (0, 0)
worst_percent_diff = (0, 0)
for old, new in zip(expected_results, new_results):
try:
oldkey, oldval, tol = self.__sanitize_regression_string(old)
newkey, newval, _ = self.__sanitize_regression_string(new)
except ValueError as e:
errorstr = "The results: {0} could not be parsed. Error: {1}".format(new, str(e))
errors.append(errorstr)
continue
assert oldkey == newkey
tolerance = tol if tol else relative_tolerance
if newval is not None:
LOG.info(f'Comparing {oldval} to {newval} with a rel. tolerance of {tolerance}')
if oldval != pytest.approx(newval, rel=tolerance):
diff = oldval-newval
percent_diff = (oldval-newval)/oldval * 100
if abs(diff) > abs(worst_diff[0]):
worst_diff = diff, oldkey
if abs(percent_diff) > abs(worst_percent_diff[0]):
worst_percent_diff = percent_diff, oldkey
errorstr = f"{oldkey}\n\tvalues differ by > a relative difference of {tolerance}\n\texpected: {oldval}\n\tnew: {newval}\n\tdiff: {diff}\n\tpercent_diff: {percent_diff}%"
errors.append(errorstr)
elif oldval is not None:
# If only the new value is None, fail
errorstr = f"{oldkey}\n\tvalue is None\n\texpected: {oldval}\n\tnew: {newval}"
errors.append(errorstr)
else:
# If old and new values are both None, this is expected, so pass
LOG.info(f'Comparing {oldval} and {newval}... both values are None.')
[LOG.warning(x) for x in errors]
n_errors = len(errors)
if n_errors > 0:
summary_str = f"Worst absolute diff, {worst_diff[1]}: {worst_diff[0]}\nWorst percentage diff, {worst_percent_diff[1]}: {worst_percent_diff[0]}%"
errors.append(summary_str)
> pytest.fail("Failed to match {0} result value{1} within tolerance{1} :\n{2}".format(
n_errors, '' if n_errors == 1 else 's', '\n'.join(errors)), pytrace=True)
E Failed: Failed to match 23 result values within tolerances :
E s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 15177530
E new: 15177502
E diff: 28
E percent_diff: 0.00018448324595635784%
E s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 11498684
E new: 11498668
E diff: 16
E percent_diff: 0.0001391463579658333%
E s10.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 3678846
E new: 3678834
E diff: 12
E percent_diff: 0.00032618924521439605%
E s12.hifv_fluxboot.13A-537.sb24066356.eb24324502.56514.05971091435.flux_densities.spw_0
E values differ by > a relative difference of 1e-07
E expected: 0.7150194812182905
E new: 0.7150122744292318
E diff: 7.2067890587446115e-06
E percent_diff: 0.001007915063581943%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 48816924
E new: 48816938
E diff: -14
E percent_diff: -2.8678578765020097e-05%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.before
E values differ by > a relative difference of 1e-07
E expected: 48585520
E new: 48585534
E diff: -14
E percent_diff: -2.881516962255421e-05%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 7526936
E new: 7526946
E diff: -10
E percent_diff: -0.00013285618477425608%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.before
E values differ by > a relative difference of 1e-07
E expected: 7526936
E new: 7526946
E diff: -10
E percent_diff: -0.00013285618477425608%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 6902438
E new: 6902470
E diff: -32
E percent_diff: -0.00046360430908615186%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.before
E values differ by > a relative difference of 1e-07
E expected: 6902438
E new: 6902470
E diff: -32
E percent_diff: -0.00046360430908615186%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 11498684
E new: 11498668
E diff: 16
E percent_diff: 0.0001391463579658333%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_4.num_rows_flagged.before
E values differ by > a relative difference of 1e-07
E expected: 11498684
E new: 11498668
E diff: 16
E percent_diff: 0.0001391463579658333%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 3678846
E new: 3678834
E diff: 12
E percent_diff: 0.00032618924521439605%
E s14.hifv_applycals.13A-537.sb24066356.eb24324502.56514.05971091435.scan_8.num_rows_flagged.before
E values differ by > a relative difference of 1e-07
E expected: 3678846
E new: 3678834
E diff: 12
E percent_diff: 0.00032618924521439605%
E s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 29068855
E new: 29069200
E diff: -345
E percent_diff: -0.0011868372524476798%
E s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_5.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 9126917
E new: 9126923
E diff: -6
E percent_diff: -6.573961393535189e-05%
E s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_6.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 7154834
E new: 7154943
E diff: -109
E percent_diff: -0.001523445547443868%
E s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_7.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 7755168
E new: 7755398
E diff: -230
E percent_diff: -0.002965764249078808%
E s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.mean
E values differ by > a relative difference of 1e-07
E expected: 0.6293920371440289
E new: 0.629412100079294
E diff: -2.0062935265130477e-05
E percent_diff: -0.0031876690649232523%
E s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.variance
E values differ by > a relative difference of 1e-07
E expected: 0.06275352250234698
E new: 0.06275879799932889
E diff: -5.275496981912986e-06
E percent_diff: -0.008406694591073644%
E s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 14429374
E new: 14429416
E diff: -42
E percent_diff: -0.00029107291834004715%
E s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_2.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 7526936
E new: 7526946
E diff: -10
E percent_diff: -0.00013285618477425608%
E s8.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.scan_3.num_rows_flagged.after
E values differ by > a relative difference of 1e-07
E expected: 6902438
E new: 6902470
E diff: -32
E percent_diff: -0.00046360430908615186%
E Worst absolute diff, s15.hifv_checkflag.13A-537.sb24066356.eb24324502.56514.05971091435.num_rows_flagged.after: -345
E Worst percentage diff, s16.hifv_statwt.13A-537.sb24066356.eb24324502.56514.05971091435.variance: -0.008406694591073644%
pipeline/infrastructure/utils/regression-tester.py:290: Failed