Pipeline

• 

  Takeshi Nakazato 15 Oct 2024 aa8b5873729174202c020ebfa4395c0b8a67bfeb

  Pull request #1385: PIPE-2264 improve numpy 2 compatibility
  Merge in PIPE/pipeline from PIPE-2264-improve-numpy-2-compatibility to main
  
  * commit 'e2cad6514b1d16995ba318197fdb9a89b1811455': (34 commits)
    Do not import bdsf on CASA 6.6.6+
    Bump minimum casa version
    Increment astropy version
    Constrain input value to 1-d list
    Revert temporal change on hsd/tasks/imaging/worker.py
    Porting findContinuum version 8.2
    porting almarenorm v2.1
    properly display observing pattern in pointing plot
    Fix scorecalculator unit test failure
    Fix mswrapper unit test failure
    Additoinal bug fix for tb.query issue
    Use np.prod instead of deprecated np.product
    set interpolation method only when it's available as tsdimaging parameter
    Avoid contamination from previouw plots
    Explicitly cast field id into Python int
    temporarily use nearest interpolation for imaging
    change to fix weblog rendering error in hsd_imaiging
    use tolist method to convert numpy array into Python list
    Removed unnecessary log handling from applycal qa code.
    Use ndarray.tolist method to convert all list elements into Python builtin types
    ...
• 

  Takeshi Nakazato 09 Oct 2024 e2cad6514b1d16995ba318197fdb9a89b1811455

  Merge branch 'main' into PIPE-2264-improve-numpy-2-compatibility
• 

  Takeshi Nakazato 07 Oct 2024 9787a3400178e00ec60f92b16061f057077063ec m

  Do not import bdsf on CASA 6.6.6+
  Environment marker suggested by Rui (see PIPE-2359)

  • requirements.txt (version 9787a3400178e00ec60f92b16061f057077063ec) (diffs)
• 

  Takeshi Nakazato 06 Oct 2024 bbd80b45c92ed1cad4865d63f50c993f25544b72 m

  Bump minimum casa version

  • pipeline/infrastructure/launcher.py (version bbd80b45c92ed1cad4865d63f50c993f25544b72) (diffs)
  • requirements.txt (version bbd80b45c92ed1cad4865d63f50c993f25544b72) (diffs)
• 

  Takeshi Nakazato 06 Oct 2024 a618278abd771a7f63577e813fe7ae973cd97791 m

  Increment astropy version

  • requirements.txt (version a618278abd771a7f63577e813fe7ae973cd97791) (diffs)
• 

  Takeshi Nakazato 04 Oct 2024 605267cf9d461fe82005923d9d88948cb22c57df m

  Constrain input value to 1-d list
  Although there is no mechanism to constrain numpy array input, constraint on the list was introduced to type hint. Implementation checks every list/ndarray elements if they are "value" (number or string) instead of list/ndarray.

  • pipeline/infrastructure/utils/utils.py (version 605267cf9d461fe82005923d9d88948cb22c57df) (diffs)
  • pipeline/infrastructure/utils/utils_test.py (version 605267cf9d461fe82005923d9d88948cb22c57df) (diffs)
• 

  Takeshi Nakazato 01 Oct 2024 c204a649e36f069715fbe5af4f33d6de1b56b3f8 m

  Revert temporal change on hsd/tasks/imaging/worker.py
  This reverts commits 3a0a2c2f9176383418a06d3c5c830b406437d107 and 0a6d900e8b307e4640608f0a82670d7cc8eb694c.
  
  By this commit, image products generated by this branch will change
  compared with the ones generated by CASA 6.6.1 (PL2024) due to the
  change of frequency interpolation method for imaging from "nearest"
  to "linear".

  • pipeline/hsd/tasks/imaging/worker.py (version c204a649e36f069715fbe5af4f33d6de1b56b3f8) (diffs)
• 

  Takeshi Nakazato 29 Sep 2024 e20b2aaa9c4d4156e8ecded4051fc6caf971c1db m

  Porting findContinuum version 8.2

  • pipeline/extern/findContinuum.py (version e20b2aaa9c4d4156e8ecded4051fc6caf971c1db) (diffs)
• 

  Takeshi Nakazato 27 Sep 2024 2914d156ab7a832deb852facbf06f82674b0e984 m

  porting almarenorm v2.1
  Ported from almasw repository.
  
  branch URL: https://bitbucket.alma.cl/projects/ALMA/repos/almasw/browse?at=refs%2Fheads%2FICT-23760%2Frenorm
  commit: 1b5a3ca13e2

  • pipeline/extern/almarenorm.py (version 2914d156ab7a832deb852facbf06f82674b0e984) (diffs)
• 

  Takeshi Nakazato 27 Sep 2024 516b5a4a4acac16d3e4a3692b8f1d236b0ff904d m

  properly display observing pattern in pointing plot

  • pipeline/infrastructure/displays/pointing.py (version 516b5a4a4acac16d3e4a3692b8f1d236b0ff904d) (diffs)
• 

  Takeshi Nakazato 27 Sep 2024 d624f7e5ca4c1a1dd256039943fe15f5d9d47948 m

  Fix scorecalculator unit test failure
  It seems that builtin sum in Python 3.10 tries to keep data type of the values to be summed up even if it causes overflow. On the other hand, np.sum flexibly changes data type of resulting value when data type is integer and the resulting value causes overflow with the input data type.

  • pipeline/qa/scorecalculator.py (version d624f7e5ca4c1a1dd256039943fe15f5d9d47948) (diffs)
• 

  Takeshi Nakazato 27 Sep 2024 d0031a98d9a7834f0d3f7ed0225797917896a002 m

  Fix mswrapper unit test failure

  • pipeline/hifa/tasks/applycal/mswrapper_test.py (version d0031a98d9a7834f0d3f7ed0225797917896a002) (diffs)
• 

  Takeshi Nakazato 26 Sep 2024 bb17a59a35ea4042cc3c2972ed9aac09c714c556 m

  Additoinal bug fix for tb.query issue

  • pipeline/extern/almarenorm.py (version bb17a59a35ea4042cc3c2972ed9aac09c714c556) (diffs)
  • pipeline/extern/almarenorm_pl2023.py (version bb17a59a35ea4042cc3c2972ed9aac09c714c556) (diffs)
• 

  Takeshi Nakazato 26 Sep 2024 19da091504c508ff7ca204c6f6c8686207e6dca6 m

  Use np.prod instead of deprecated np.product
  numpy.product is deprecated since 1.25.0.

  • pipeline/hifv/tasks/syspower/display.py (version 19da091504c508ff7ca204c6f6c8686207e6dca6) (diffs)
• 

  Takeshi Nakazato 25 Sep 2024 0a6d900e8b307e4640608f0a82670d7cc8eb694c m

  set interpolation method only when it's available as tsdimaging parameter

  • pipeline/hsd/tasks/imaging/worker.py (version 0a6d900e8b307e4640608f0a82670d7cc8eb694c) (diffs)
• 

  Takeshi Nakazato 25 Sep 2024 07feee9880afb6a86f6213df93e591e6d06a2a6a m

  Avoid contamination from previouw plots
  This is done by clearing Figure object explicitly before drawing

  • pipeline/h/tasks/common/displays/slice.py (version 07feee9880afb6a86f6213df93e591e6d06a2a6a) (diffs)
• 

  Takeshi Nakazato 25 Sep 2024 1da9b7aad8091ff1581e06213bf5c665339ca2b6 m

  Explicitly cast field id into Python int

  • pipeline/infrastructure/displays/pointing.py (version 1da9b7aad8091ff1581e06213bf5c665339ca2b6) (diffs)
• 

  Takeshi Nakazato 25 Sep 2024 3a0a2c2f9176383418a06d3c5c830b406437d107 m

  temporarily use nearest interpolation for imaging
  This must eventually be reverted to linear (default) once verification has been done.

  • pipeline/hsd/tasks/imaging/worker.py (version 3a0a2c2f9176383418a06d3c5c830b406437d107) (diffs)
• 

  Takeshi Nakazato 25 Sep 2024 ceb3afc6e3c743114920a868fff11115bb75fd52 m

  change to fix weblog rendering error in hsd_imaiging

  • pipeline/hsd/tasks/imaging/display.py (version ceb3afc6e3c743114920a868fff11115bb75fd52) (diffs)
• 

  Takeshi Nakazato 24 Sep 2024 3ca671064ddcba9f2efa41105a36439b81d839ae m

  use tolist method to convert numpy array into Python list

  • pipeline/extern/almarenorm.py (version 3ca671064ddcba9f2efa41105a36439b81d839ae) (diffs)
  • pipeline/extern/almarenorm_pl2023.py (version 3ca671064ddcba9f2efa41105a36439b81d839ae) (diffs)
• 

  Takeshi Nakazato 24 Sep 2024 a02624416a4e8bdaf4e1e951a1c22826d22e97c5 m

  Removed unnecessary log handling from applycal qa code.
  A piece of code was wrapped by "with" statement to ignore numpy ComplexWarning. However, that code should not emit ComplexWarning according to the nature of the operation. So it should be safe to remove "with" block.

  • pipeline/hifa/tasks/applycal/ampphase_vs_freq_qa.py (version a02624416a4e8bdaf4e1e951a1c22826d22e97c5) (diffs)
• 

  Takeshi Nakazato 23 Sep 2024 361e3cc397980e169e26c8dc5ee87c3682804181 m

  Use ndarray.tolist method to convert all list elements into Python builtin types

  • pipeline/infrastructure/utils/utils.py (version 361e3cc397980e169e26c8dc5ee87c3682804181) (diffs)
  • pipeline/infrastructure/utils/utils_test.py (version 361e3cc397980e169e26c8dc5ee87c3682804181) (diffs)
• 

  Takeshi Nakazato 23 Sep 2024 0985a522388d7e4547006f6871948aba0558b4b2 m

  Use callable to avoid too long test case description

  • pipeline/infrastructure/utils/utils_test.py (version 0985a522388d7e4547006f6871948aba0558b4b2) (diffs)
• 

  Takeshi Nakazato 23 Sep 2024 3b7f099b90cc91f9de14d8056a65fd2f7784750f m

  use isinstance instead of comparing type

  • pipeline/extern/findContinuum.py (version 3b7f099b90cc91f9de14d8056a65fd2f7784750f) (diffs)
• 

  Takeshi Nakazato 23 Sep 2024 bcfa315d79585f7dace48c4ee67b505c49100ea6 m

  Refactoring and supplemental comments

  • pipeline/infrastructure/utils/utils.py (version bcfa315d79585f7dace48c4ee67b505c49100ea6) (diffs)
• 

  Takeshi Nakazato 23 Sep 2024 206046ff3eeaac8fceb69e590313518a0a212e6e m

  Updated upstream repository information for extern/SDcalatmcorr.py

  • pipeline/extern/SDcalatmcorr.py (version 206046ff3eeaac8fceb69e590313518a0a212e6e) (diffs)
• 

  Takeshi Nakazato 20 Sep 2024 9a4015f918bf90c3268b450807d6bd301cf705da m

  support converting long list

  • pipeline/infrastructure/utils/utils.py (version 9a4015f918bf90c3268b450807d6bd301cf705da) (diffs)
  • pipeline/infrastructure/utils/utils_test.py (version 9a4015f918bf90c3268b450807d6bd301cf705da) (diffs)
• 

  Takeshi Nakazato 20 Sep 2024 af8936c1fed844f0f9c9a520c63fa5e7ae867179 m

  Switch astropy version according to python version
  Effectively, it installs astropy 5.2.2 on CASA 6.6.1 and 6.1.3 on CASA 6.6.6.

  • requirements.txt (version af8936c1fed844f0f9c9a520c63fa5e7ae867179) (diffs)
• 

  Takeshi Nakazato 20 Sep 2024 3e9da803ee87115d33177c1076838bc14b9e0d03 m

  Use np.str_ instead of deprecated np.unicode_
  This is the change for extern module

  • pipeline/extern/SDcalatmcorr.py (version 3e9da803ee87115d33177c1076838bc14b9e0d03) (diffs)
• 

  Takeshi Nakazato 20 Sep 2024 dfff6087eaaa5c3fc8728f4a5720d38d98c896b7 m

  Replace removed aliases with valid names
  np.NaN -> np.nan
  np.mat -> np.asmatrix

  • pipeline/hifa/tasks/fluxscale/qa.py (version dfff6087eaaa5c3fc8728f4a5720d38d98c896b7) (diffs)
  • pipeline/hifv/tasks/syspower/syspower.py (version dfff6087eaaa5c3fc8728f4a5720d38d98c896b7) (diffs)
• 

  Takeshi Nakazato 20 Sep 2024 58a8bbe9dd9ebe43ba85b00e5d1e6eaa1d93acbc m

  Replace deprecated np.str_ with np.bytes_
  This is the change for extern module

  • pipeline/extern/findContinuum.py (version 58a8bbe9dd9ebe43ba85b00e5d1e6eaa1d93acbc) (diffs)
• 

  Takeshi Nakazato 20 Sep 2024 6beabd5ab15a9a7fed6ec4852dd5ac778fe8aada m

  convert list into string safely for table query
  This is the change for extern module

  • pipeline/extern/almarenorm.py (version 6beabd5ab15a9a7fed6ec4852dd5ac778fe8aada) (diffs)
  • pipeline/extern/almarenorm_pl2023.py (version 6beabd5ab15a9a7fed6ec4852dd5ac778fe8aada) (diffs)
• 

  Takeshi Nakazato 20 Sep 2024 27750658a0800eda3959299102d4f237a6c0dedd m

  use utility function to convert list/ndarray into string
  In Python 3.10 with NumPy 2.0.1, print(list(ndaray)) gives elements with data types such as '[np.int64(0), np.int64(1), np.int64(2)]'. Utility function was defined to avoid such unexpected string.

  • pipeline/domain/datatable.py (version 27750658a0800eda3959299102d4f237a6c0dedd) (diffs)
  • pipeline/domain/measurementset.py (version 27750658a0800eda3959299102d4f237a6c0dedd) (diffs)
  • pipeline/hifa/heuristics/wvrgcal.py (version 27750658a0800eda3959299102d4f237a6c0dedd) (diffs)
  • pipeline/hifa/tasks/wvrgcal/wvrg_qa.py (version 27750658a0800eda3959299102d4f237a6c0dedd) (diffs)
  • pipeline/hsd/tasks/common/utils.py (version 27750658a0800eda3959299102d4f237a6c0dedd) (diffs)
  • pipeline/hsd/tasks/imaging/weighting.py (version 27750658a0800eda3959299102d4f237a6c0dedd) (diffs)
  • pipeline/hsd/tasks/importdata/reader.py (version 27750658a0800eda3959299102d4f237a6c0dedd) (diffs)
  • pipeline/hsd/tasks/skycal/renderer.py (version 27750658a0800eda3959299102d4f237a6c0dedd) (diffs)
  • pipeline/infrastructure/utils/weblog.py (version 27750658a0800eda3959299102d4f237a6c0dedd) (diffs)
  • pipeline/qa/scorecalculator.py (version 27750658a0800eda3959299102d4f237a6c0dedd) (diffs)
• 

  Takeshi Nakazato 20 Sep 2024 4453f7cdac1986974271151ed5e8eafedb291c28 m

  define list_to_str

  • pipeline/infrastructure/utils/utils.py (version 4453f7cdac1986974271151ed5e8eafedb291c28) (diffs)
  • pipeline/infrastructure/utils/utils_test.py (version 4453f7cdac1986974271151ed5e8eafedb291c28) (diffs)
• 

  Takeshi Nakazato 20 Sep 2024 4c6e1b801b022b64f691f97d1fe00e531473a680 m

  import numpy as np
  This is for eval to recognize "np".

  • pipeline/domain/datatable.py (version 4c6e1b801b022b64f691f97d1fe00e531473a680) (diffs)
• 

  Takeshi Nakazato 20 Sep 2024 b5f31478139a4e7082aa13cf3a1a0bcbc26d0bb2 m

  Upgrade astropy to 6.1.3

  • requirements.txt (version b5f31478139a4e7082aa13cf3a1a0bcbc26d0bb2) (diffs)
• 

  Takeshi Nakazato 20 Sep 2024 580c190b732b748b2f561a7925ba50671aa4e0ba m

  Merge branch 'PIPE-1719-improve-python-3.10-compatbility' into PIPE-2264-improve-numpy-2-compatibility