Build: #112 failed

Job: Test ManyLinux 2.28 Python 3.12 failed

Stages & jobs

  1. Test

csv 3899 eb2 small procedure hifa calimage regression: Test case result

The below summarizes the result of the test " csv 3899 eb2 small procedure hifa calimage regression" in build 112 of PIPESPECS - Test Pipeline main with Casa 6.7.4 - Test ManyLinux 2.28 Python 3.12.
Description
csv 3899 eb2 small procedure hifa calimage regression
Test class
tests.regression.fast.alma_if_fast_test
Method
test_csv_3899_eb2_small__procedure_hifa_calimage__regression
Duration
138 mins
Status
Failed (New Failure)

Error Log

Failed: Failed to match 18 result values within tolerances :
s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.num_rows_flagged.after
	values differ by > a relative difference of 1e-07
	expected: 5103176
	new:      5081336
	diff: 21840
	percent_diff: 0.42796877865862354%
s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.num_rows_flagged.before
	values differ by > a relative difference of 1e-07
	expected: 5101736
	new:      5081336
	diff: 20400
	percent_diff: 0.3998638894682124%
s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.qa.metric.CorrectedampflagQAHandler
	values differ by > a relative difference of 1e-07
	expected: 0.0053688072299937365
	new:      0.0
	diff: 0.0053688072299937365
	percent_diff: 100.0%
s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.qa.score.CorrectedampflagQAHandler
	values differ by > a relative difference of 1e-07
	expected: 0.9946311927700062
	new:      1.0
	diff: -0.005368807229993777
	percent_diff: -0.539778690736802%
s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_1.spw_24.I
	values differ by > a relative difference of 1e-07
	expected: 0.33639694604470255
	new:      0.3363920394822608
	diff: 4.906562441731044e-06
	percent_diff: 0.00145856331319935%
s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_1.spw_24.qa.metric.score_gfluxscale_k_spw
	values differ by > a relative difference of 1e-07
	expected: 0.9954974538624377
	new:      0.9954829339017919
	diff: 1.4519960645831276e-05
	percent_diff: 0.0014585633131953455%
s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_16.I
	values differ by > a relative difference of 1e-07
	expected: 0.007922074070068001
	new:      0.007922920772211503
	diff: -8.46702143501854e-07
	percent_diff: -0.010687884713183276%
s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_22.I
	values differ by > a relative difference of 1e-07
	expected: 0.009403204584910561
	new:      0.009092581426194022
	diff: 0.00031062315871653937
	percent_diff: 3.3033755238613036%
s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_22.qa.metric.score_gfluxscale_k_spw
	values differ by > a relative difference of 1e-07
	expected: 1.1082090210872806
	new:      1.0714861963220799
	diff: 0.03672282476520072
	percent_diff: 3.313709243150846%
s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_22.qa.score.score_gfluxscale_k_spw
	values differ by > a relative difference of 1e-07
	expected: 0.75
	new:      1.0
	diff: -0.25
	percent_diff: -33.33333333333333%
s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_24.I
	values differ by > a relative difference of 1e-07
	expected: 0.014421274744359853
	new:      0.014419697875696509
	diff: 1.5768686633440415e-06
	percent_diff: 0.010934322320991447%
s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_24.qa.metric.score_gfluxscale_k_spw
	values differ by > a relative difference of 1e-07
	expected: 1.672256908989825
	new:      1.6718953687798206
	diff: 0.0003615402100043408
	percent_diff: 0.021619896324586847%
s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.qa.metric.score_derived_fluxes_snr
	values differ by > a relative difference of 1e-07
	expected: 2.0439023901874442
	new:      2.0402774242377393
	diff: 0.0036249659497049613
	percent_diff: 0.17735514020180385%
s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.qa.score.score_derived_fluxes_snr
	values differ by > a relative difference of 1e-07
	expected: 0.8372152443898132
	new:      0.8372279101112245
	diff: -1.2665721411342368e-05
	percent_diff: -0.0015128393201408455%
s21.hif_applycal.uid___A002_X1181695_X1c6a4_8ant.num_rows_flagged.after
	values differ by > a relative difference of 1e-07
	expected: 4845536
	new:      4840736
	diff: 4800
	percent_diff: 0.0990602484431031%
s21.hif_applycal.uid___A002_X1181695_X1c6a4_8ant.num_rows_flagged.before
	values differ by > a relative difference of 1e-07
	expected: 4845536
	new:      4840736
	diff: 4800
	percent_diff: 0.0990602484431031%
s21.hif_applycal.uid___A002_X1181695_X1c6a4_8ant.scan_7.num_rows_flagged.after
	values differ by > a relative difference of 1e-07
	expected: 109440
	new:      104640
	diff: 4800
	percent_diff: 4.385964912280701%
s21.hif_applycal.uid___A002_X1181695_X1c6a4_8ant.scan_7.num_rows_flagged.before
	values differ by > a relative difference of 1e-07
	expected: 109440
	new:      104640
	diff: 4800
	percent_diff: 4.385964912280701%
Worst absolute diff, s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.num_rows_flagged.after: 21840
Worst percentage diff, s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.qa.metric.CorrectedampflagQAHandler: 100.0%
@pytest.mark.twelve
    def test_csv_3899_eb2_small__procedure_hifa_calimage__regression():
        """PIPE-2245: Run small ALMA cal+image regression to cover various heuristics
    
        Recipe name:                procedure_hifa_calimage
        Dataset:                    CSV-3899-EB2-small
        """
        ref_directory = 'pl-regressiontest/CSV-3899-EB2-small'
    
        pt = PipelineTester(
            visname=['uid___A002_X1181695_X1c6a4_8ant.ms'],
            recipe='procedure_hifa_calimage.xml',
            input_dir=ref_directory,
            expectedoutput_dir=ref_directory,
        )
    
>       pt.run(omp_num_threads=1)

tests/regression/fast/alma_if_fast_test.py:181: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/testing_utils.py:381: in run
    self.__compare_results(new_file, default_relative_tolerance)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.testing_utils.PipelineTester object at 0x7fa7b5a687a0>
new_file = 'uid___A002_X1181695_X1c6a4_8ant.ms.NEW.results.txt'
relative_tolerance = 1e-07

    def __compare_results(self, new_file: str, relative_tolerance: float) -> None:
        """
        Compare results between new one loaded from file and old one.
    
        Args:
            new_file : file path of new results
            relative_tolerance : relative tolerance of output value
        """
        with open(self.expectedoutput_file) as expected_fd, open(new_file) as new_fd:
            expected_results = expected_fd.readlines()
            new_results = new_fd.readlines()
            errors = []
            worst_diff = (0, 0)
            worst_percent_diff = (0, 0)
            for old, new in zip(expected_results, new_results):
                try:
                    oldkey, oldval, tol = self.__sanitize_results_string(old)
                    newkey, newval, _ = self.__sanitize_results_string(new)
                except ValueError as e:
                    errorstr = "The results: {0} could not be parsed. Error: {1}".format(new, str(e))
                    errors.append(errorstr)
                    continue
    
                assert oldkey == newkey, f"Expected key {oldkey} does not match new key {newkey}."
                tolerance = tol if tol else relative_tolerance
                if newval is not None:
                    LOG.info('Comparing %s to %s with a rel. tolerance of %s', oldval, newval, tolerance)
                    if oldval != pytest.approx(newval, rel=tolerance):
                        diff = oldval-newval
                        percent_diff = (oldval-newval)/oldval * 100 if oldval != 0 else 100
                        if abs(diff) > abs(worst_diff[0]):
                            worst_diff = diff, oldkey
                        if abs(percent_diff) > abs(worst_percent_diff[0]):
                            worst_percent_diff = percent_diff, oldkey
                        errorstr = f"{oldkey}\n\tvalues differ by > a relative difference of {tolerance}\n\texpected: {oldval}\n\tnew:      {newval}\n\tdiff: {diff}\n\tpercent_diff: {percent_diff}%"
                        errors.append(errorstr)
                elif oldval is not None:
                    # If only the new value is None, fail
                    errorstr = f"{oldkey}\n\tvalue is None\n\texpected: {oldval}\n\tnew:      {newval}"
                    errors.append(errorstr)
                else:
                    # If old and new values are both None, this is expected, so pass
                    LOG.info('Comparing %s and %s... both values are None.', oldval, newval)
    
            [LOG.warning(x) for x in errors]
            n_errors = len(errors)
            if n_errors > 0:
                summary_str = f"Worst absolute diff, {worst_diff[1]}: {worst_diff[0]}\nWorst percentage diff, {worst_percent_diff[1]}: {worst_percent_diff[0]}%"
                errors.append(summary_str)
>               pytest.fail("Failed to match {0} result value{1} within tolerance{1} :\n{2}".format(
                    n_errors, '' if n_errors == 1 else 's', '\n'.join(errors)), pytrace=True)
E               Failed: Failed to match 18 result values within tolerances :
E               s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.num_rows_flagged.after
E               	values differ by > a relative difference of 1e-07
E               	expected: 5103176
E               	new:      5081336
E               	diff: 21840
E               	percent_diff: 0.42796877865862354%
E               s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.num_rows_flagged.before
E               	values differ by > a relative difference of 1e-07
E               	expected: 5101736
E               	new:      5081336
E               	diff: 20400
E               	percent_diff: 0.3998638894682124%
E               s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.qa.metric.CorrectedampflagQAHandler
E               	values differ by > a relative difference of 1e-07
E               	expected: 0.0053688072299937365
E               	new:      0.0
E               	diff: 0.0053688072299937365
E               	percent_diff: 100.0%
E               s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.qa.score.CorrectedampflagQAHandler
E               	values differ by > a relative difference of 1e-07
E               	expected: 0.9946311927700062
E               	new:      1.0
E               	diff: -0.005368807229993777
E               	percent_diff: -0.539778690736802%
E               s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_1.spw_24.I
E               	values differ by > a relative difference of 1e-07
E               	expected: 0.33639694604470255
E               	new:      0.3363920394822608
E               	diff: 4.906562441731044e-06
E               	percent_diff: 0.00145856331319935%
E               s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_1.spw_24.qa.metric.score_gfluxscale_k_spw
E               	values differ by > a relative difference of 1e-07
E               	expected: 0.9954974538624377
E               	new:      0.9954829339017919
E               	diff: 1.4519960645831276e-05
E               	percent_diff: 0.0014585633131953455%
E               s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_16.I
E               	values differ by > a relative difference of 1e-07
E               	expected: 0.007922074070068001
E               	new:      0.007922920772211503
E               	diff: -8.46702143501854e-07
E               	percent_diff: -0.010687884713183276%
E               s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_22.I
E               	values differ by > a relative difference of 1e-07
E               	expected: 0.009403204584910561
E               	new:      0.009092581426194022
E               	diff: 0.00031062315871653937
E               	percent_diff: 3.3033755238613036%
E               s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_22.qa.metric.score_gfluxscale_k_spw
E               	values differ by > a relative difference of 1e-07
E               	expected: 1.1082090210872806
E               	new:      1.0714861963220799
E               	diff: 0.03672282476520072
E               	percent_diff: 3.313709243150846%
E               s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_22.qa.score.score_gfluxscale_k_spw
E               	values differ by > a relative difference of 1e-07
E               	expected: 0.75
E               	new:      1.0
E               	diff: -0.25
E               	percent_diff: -33.33333333333333%
E               s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_24.I
E               	values differ by > a relative difference of 1e-07
E               	expected: 0.014421274744359853
E               	new:      0.014419697875696509
E               	diff: 1.5768686633440415e-06
E               	percent_diff: 0.010934322320991447%
E               s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.field_3.spw_24.qa.metric.score_gfluxscale_k_spw
E               	values differ by > a relative difference of 1e-07
E               	expected: 1.672256908989825
E               	new:      1.6718953687798206
E               	diff: 0.0003615402100043408
E               	percent_diff: 0.021619896324586847%
E               s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.qa.metric.score_derived_fluxes_snr
E               	values differ by > a relative difference of 1e-07
E               	expected: 2.0439023901874442
E               	new:      2.0402774242377393
E               	diff: 0.0036249659497049613
E               	percent_diff: 0.17735514020180385%
E               s17.hifa_gfluxscale.uid___A002_X1181695_X1c6a4_8ant.qa.score.score_derived_fluxes_snr
E               	values differ by > a relative difference of 1e-07
E               	expected: 0.8372152443898132
E               	new:      0.8372279101112245
E               	diff: -1.2665721411342368e-05
E               	percent_diff: -0.0015128393201408455%
E               s21.hif_applycal.uid___A002_X1181695_X1c6a4_8ant.num_rows_flagged.after
E               	values differ by > a relative difference of 1e-07
E               	expected: 4845536
E               	new:      4840736
E               	diff: 4800
E               	percent_diff: 0.0990602484431031%
E               s21.hif_applycal.uid___A002_X1181695_X1c6a4_8ant.num_rows_flagged.before
E               	values differ by > a relative difference of 1e-07
E               	expected: 4845536
E               	new:      4840736
E               	diff: 4800
E               	percent_diff: 0.0990602484431031%
E               s21.hif_applycal.uid___A002_X1181695_X1c6a4_8ant.scan_7.num_rows_flagged.after
E               	values differ by > a relative difference of 1e-07
E               	expected: 109440
E               	new:      104640
E               	diff: 4800
E               	percent_diff: 4.385964912280701%
E               s21.hif_applycal.uid___A002_X1181695_X1c6a4_8ant.scan_7.num_rows_flagged.before
E               	values differ by > a relative difference of 1e-07
E               	expected: 109440
E               	new:      104640
E               	diff: 4800
E               	percent_diff: 4.385964912280701%
E               Worst absolute diff, s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.num_rows_flagged.after: 21840
E               Worst percentage diff, s16.hifa_gfluxscaleflag.uid___A002_X1181695_X1c6a4_8ant.qa.metric.CorrectedampflagQAHandler: 100.0%

tests/testing_utils.py:440: Failed