simple 01-Jun-2017 16:08:57 Build CASA - Branch Package - feature-CAS-10009 - Tag Branch for Packages #2 (CASA-CP90-TBFP-2) started building on agent cbt-el7-4.cv.nrao.edu simple 01-Jun-2017 16:08:57 Remote agent on host cbt-el7-4.cv.nrao.edu simple 01-Jun-2017 16:08:57 Build working directory is /export/home/cbt-el7-4/bamboohome/xml-data/build-dir/CASA-CP90-TBFP simple 01-Jun-2017 16:08:57 Executing build CASA - Branch Package - feature-CAS-10009 - Tag Branch for Packages #2 (CASA-CP90-TBFP-2) simple 01-Jun-2017 16:08:57 Skipping execution of task 'Checkout Default Repository' as it is disabled simple 01-Jun-2017 16:08:57 Running pre-build action: VCS Version Collector simple 01-Jun-2017 16:08:57 Skipping execution of task 'Checkout Default Repository' as it is disabled command 01-Jun-2017 16:08:57 Substituting variable: ${bamboo.repository.branch.name} with feature/CAS-10009 command 01-Jun-2017 16:08:57 Substituting variable: ${bamboo.buildNumber} with 2 command 01-Jun-2017 16:08:57 Substituting variable: ${bamboo.planRepository.revision} with 205cd80ac6fd7d512c22dc25996b25c2e8a669e8 command 01-Jun-2017 16:08:57 Substituting variable: ${bamboo.planRepository.repositoryUrl} with ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git command 01-Jun-2017 16:08:57 Substituting variable: ${bamboo.planRepository.repositoryUrl} with ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git command 01-Jun-2017 16:08:57 Substituting variable: ${bamboo.build.working.directory} with /export/home/cbt-el7-4/bamboohome/xml-data/build-dir/CASA-CP90-TBFP command 01-Jun-2017 16:08:57 Substituting variable: ${bamboo.planRepository.repositoryUrl} with ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git command 01-Jun-2017 16:08:57 Substituting variable: ${bamboo.repository.branch.name} with feature/CAS-10009 simple 01-Jun-2017 16:08:57 Starting task 'Script' of type 'com.atlassian.bamboo.plugins.scripttask:task.builder.script' command 01-Jun-2017 16:08:57 Beginning to execute external process for build 'CASA - Branch Package - feature-CAS-10009 - Tag Branch for Packages #2 (CASA-CP90-TBFP-2)'\n ... running command line: \n/bin/sh /export/home/cbt-el7-4/bamboohome/temp/CASA-CP90-TBFP-2-ScriptBuildTask-716888204078185411.sh\n ... in: /export/home/cbt-el7-4/bamboohome/xml-data/build-dir/CASA-CP90-TBFP\n ... using extra environment variables: \nbamboo_planRepository_1_branch=feature/CAS-10009\nbamboo_capability_buildmachinetype=build\nbamboo_dependency_parent_0=CASA-CTS1G273-3\nbamboo_planRepository_2_branch=master\nbamboo_repository_6619139_branch_name=master\nbamboo_planRepository_1_branchDisplayName=feature/CAS-10009\nbamboo_dependency_parent_total=1\nbamboo_repository_revision_number=205cd80ac6fd7d512c22dc25996b25c2e8a669e8\nbamboo_resultsUrl=https://open-bamboo.nrao.edu/browse/CASA-CP90-TBFP-2\nbamboo_repository_15368248_git_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git\nbamboo_repository_6619139_git_username=\nbamboo_planRepository_1_name=OPEN-CASA-FULL\nbamboo_build_working_directory=/export/home/cbt-el7-4/bamboohome/xml-data/build-dir/CASA-CP90-TBFP\nbamboo_buildKey=CASA-CP90-TBFP\nbamboo_repository_15368248_branch_name=feature/CAS-10009\nbamboo_shortPlanName=feature-CAS-10009\nbamboo_capability_hostname=cbt-el7-4.cv.nrao.edu\nbamboo_planRepository_2_username=\nbamboo_planRepository_name=OPEN-CASA-FULL\nbamboo_buildNumber=2\nbamboo_planRepository_2_branchDisplayName=master\nbamboo_capability_system_jdk_JDK_1_8_0_131=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-2.b11.el7_3.x86_64\nbamboo_shortJobName=Tag Branch for Packages\nbamboo_buildResultsUrl=https://open-bamboo.nrao.edu/browse/CASA-CP90-TBFP-2\nbamboo_repository_15368248_git_username=\nbamboo_planRepository_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git\nbamboo_agentId=6160386\nbamboo_planName=CASA - Branch Package - feature-CAS-10009\nbamboo_shortPlanKey=CP90\nbamboo_shortJobKey=TBFP\nbamboo_planRepository_2_name=OPEN-CASA-PKG\nbamboo_planRepository_revision=205cd80ac6fd7d512c22dc25996b25c2e8a669e8\nbamboo_repository_previous_revision_number=642f791962ab7c51dcf2a4b510fed555073def87\nbamboo_buildTimeStamp=2017-06-01T15:42:16.034-04:00\nbamboo_casaadm_password=********\nbamboo_repository_15368248_git_branch=feature/CAS-10009\nbamboo_repository_6619139_revision_number=a5033aef1fd7dc1bbcb6cec79f754f5e47d37874\nbamboo_buildResultKey=CASA-CP90-TBFP-2\nbamboo_repository_git_branch=feature/CAS-10009\nbamboo_repository_branch_name=feature/CAS-10009\nbamboo_buildPlanName=CASA - Branch Package - feature-CAS-10009 - Tag Branch for Packages\nbamboo_capability_system_jdk_JDK_1_8_0_102=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-1.b14.el7_2.x86_64\nbamboo_repository_15368248_previous_revision_number=642f791962ab7c51dcf2a4b510fed555073def87\nbamboo_planRepository_1_revision=205cd80ac6fd7d512c22dc25996b25c2e8a669e8\nbamboo_capability_system_hg_executable=/bin/hg\nbamboo_repository_name=OPEN-CASA-PKG\nbamboo_repository_6619139_git_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa-pkg.git\nbamboo_buildFailed=false\nbamboo_capability_isCIMachine=true\nbamboo_planRepository_branch=feature/CAS-10009\nbamboo_repository_6619139_previous_revision_number=a5033aef1fd7dc1bbcb6cec79f754f5e47d37874\nbamboo_agentWorkingDirectory=/export/home/cbt-el7-4/bamboohome/xml-data/build-dir\nbamboo_planRepository_2_revision=a5033aef1fd7dc1bbcb6cec79f754f5e47d37874\nbamboo_capability_system_git_executable=/bin/git\nbamboo_repository_6619139_git_branch=master\nbamboo_capability_tag=true\nbamboo_capability_system_jdk_JDK_1_8_0_111=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64\nbamboo_repository_git_username=\nbamboo_capability_operating_system=EL7\nbamboo_planRepository_branchDisplayName=feature/CAS-10009\nbamboo_planRepository_1_type=bbserver\nbamboo_planRepository_branchName=feature/CAS-10009\nbamboo_repository_15368248_name=OPEN-CASA-FULL\nbamboo_capability_project=CASA\nbamboo_repository_6619139_name=OPEN-CASA-PKG\nbamboo_planRepository_2_type=bbserver\nbamboo_capability_system_jdk_JDK=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-1.b14.el7_2.x86_64\nbamboo_plan_storageTag=plan-15171829\nbamboo_planRepository_2_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa-pkg.git\nbamboo_planRepository_type=bbserver\nbamboo_planRepository_1_username=\nbamboo_repository_git_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git\nbamboo_planRepository_2_branchName=master\nbamboo_capability_system_builder_ant_Ant=/usr\nbamboo_working_directory=/export/home/cbt-el7-4/bamboohome/xml-data/build-dir/CASA-CP90-TBFP\nbamboo_planKey=CASA-CP90\nbamboo_repository_15368248_revision_number=205cd80ac6fd7d512c22dc25996b25c2e8a669e8\nbamboo_planRepository_1_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git\nbamboo_planRepository_username=\nbamboo_capability_system_jdk_JDK_1_8=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-1.b14.el7_2.x86_64\nbamboo_planRepository_1_branchName=feature/CAS-10009\n build 01-Jun-2017 16:08:57 2 build 01-Jun-2017 16:08:57 205cd80ac6fd7d512c22dc25996b25c2e8a669e8 build 01-Jun-2017 16:08:57 ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git build 01-Jun-2017 16:08:57 Fetching origin build 01-Jun-2017 16:08:58 Already up-to-date. build 01-Jun-2017 16:08:59 workdir: /export/home/cbt-el7-4/bamboohome/xml-data/build-dir/CASA-CP90-TBFP/taggitworkdir build 01-Jun-2017 16:08:59 sourcedir: /export/home/cbt-el7-4/bamboohome/xml-data/build-dir/CASA-CP90-TBFP/taggitworkdir/casa build 01-Jun-2017 16:08:59 gitremote: ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git build 01-Jun-2017 16:08:59 branch: feature/CAS-10009 build 01-Jun-2017 16:08:59 Push option: true error 01-Jun-2017 16:09:00 Switched to branch 'master' build 01-Jun-2017 16:09:00 build 01-Jun-2017 16:09:00 Fetching origin error 01-Jun-2017 16:09:00 From ssh://open-bitbucket.nrao.edu:7999/casa/casa error 01-Jun-2017 16:09:00 * [new branch] bugfix/CAS-10218 -> origin/bugfix/CAS-10218 error 01-Jun-2017 16:09:00 * [new branch] bugfix/CAS-10230 -> origin/bugfix/CAS-10230 error 01-Jun-2017 16:09:00 12b6847..882ebb0 carta+imager -> origin/carta+imager error 01-Jun-2017 16:09:00 642f791..205cd80 feature/CAS-10009 -> origin/feature/CAS-10009 error 01-Jun-2017 16:09:00 08f7a85..dd3c016 release/5.0.0 -> origin/release/5.0.0 error 01-Jun-2017 16:09:00 * [new tag] 5.1.0-10-bugfix-CAS-10230-3 -> 5.1.0-10-bugfix-CAS-10230-3 error 01-Jun-2017 16:09:00 * [new tag] 5.1.0-mas-11 -> 5.1.0-mas-11 error 01-Jun-2017 16:09:00 From ssh://open-bitbucket.nrao.edu:7999/casa/casa error 01-Jun-2017 16:09:00 * [new tag] 5.1.0-10-bugfix-CAS-10230-1 -> 5.1.0-10-bugfix-CAS-10230-1 error 01-Jun-2017 16:09:00 * [new tag] 5.1.0-10-bugfix-CAS-10230-2 -> 5.1.0-10-bugfix-CAS-10230-2 build 01-Jun-2017 16:09:00 Already up-to-date. build 01-Jun-2017 16:09:00 build 01-Jun-2017 16:09:00 getLastTag called with build 01-Jun-2017 16:09:00 branch: feature/CAS-10009 build 01-Jun-2017 16:09:00 env: [PATH=/bin:/usr/bin:/bin, bamboo_planRepository_1_branch=feature/CAS-10009, bamboo_planRepository_2_branch=master, bamboo_planRepository_1_branchDisplayName=feature/CAS-10009, bamboo_repository_revision_number=205cd80ac6fd7d512c22dc25996b25c2e8a669e8, WRAPPER_BITS=32, bamboo_buildKey=CASA-CP90-TBFP, bamboo_repository_15368248_branch_name=feature/CAS-10009, PWD=/export/home/cbt-el7-4/bamboohome/xml-data/build-dir/CASA-CP90-TBFP/casa-pkg, bamboo_planRepository_2_branchDisplayName=master, bamboo_capability_system_jdk_JDK_1_8_0_131=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-2.b11.el7_3.x86_64, bamboo_shortJobName=Tag Branch for Packages, bamboo_buildResultsUrl=https://open-bamboo.nrao.edu/browse/CASA-CP90-TBFP-2, bamboo_planRepository_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git, bamboo_agentId=6160386, bamboo_planRepository_revision=205cd80ac6fd7d512c22dc25996b25c2e8a669e8, bamboo_casaadm_password=********, bamboo_repository_branch_name=feature/CAS-10009, bamboo_repository_15368248_previous_revision_number=642f791962ab7c51dcf2a4b510fed555073def87, SHLVL=3, WRAPPER_OS=linux, JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-2.b11.el7_3.x86_64, bamboo_repository_6619139_git_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa-pkg.git, XDG_SESSION_ID=2, bamboo_capability_isCIMachine=true, bamboo_planRepository_branch=feature/CAS-10009, bamboo_repository_6619139_previous_revision_number=a5033aef1fd7dc1bbcb6cec79f754f5e47d37874, bamboo_repository_6619139_git_branch=master, bamboo_capability_tag=true, bamboo_planRepository_1_type=bbserver, bamboo_planRepository_branchName=feature/CAS-10009, WRAPPER_FILE_SEPARATOR=/, bamboo_repository_6619139_name=OPEN-CASA-PKG, bamboo_capability_system_jdk_JDK=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-1.b14.el7_2.x86_64, WRAPPER_PATH_SEPARATOR=:, bamboo_planRepository_1_username=, bamboo_planRepository_type=bbserver, bamboo_planRepository_2_branchName=master, bamboo_capability_system_builder_ant_Ant=/usr, bamboo_planKey=CASA-CP90, bamboo_planRepository_username=, NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat, bamboo_planRepository_1_branchName=feature/CAS-10009, bamboo_capability_buildmachinetype=build, bamboo_dependency_parent_0=CASA-CTS1G273-3, bamboo_repository_6619139_branch_name=master, bamboo_dependency_parent_total=1, bamboo_resultsUrl=https://open-bamboo.nrao.edu/browse/CASA-CP90-TBFP-2, bamboo_repository_15368248_git_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git, bamboo_repository_6619139_git_username=, bamboo_planRepository_1_name=OPEN-CASA-FULL, bamboo_build_working_directory=/export/home/cbt-el7-4/bamboohome/xml-data/build-dir/CASA-CP90-TBFP, bamboo_shortPlanName=feature-CAS-10009, bamboo_capability_hostname=cbt-el7-4.cv.nrao.edu, bamboo_planRepository_2_username=, bamboo_planRepository_name=OPEN-CASA-FULL, LOGNAME=casaci, bamboo_buildNumber=2, bamboo_repository_15368248_git_username=, SHELL=/bin/sh, bamboo_planName=CASA - Branch Package - feature-CAS-10009, bamboo_shortPlanKey=CP90, OLDPWD=/usr/lib/jvm/groovy-2.4.7, bamboo_shortJobKey=TBFP, bamboo_planRepository_2_name=OPEN-CASA-PKG, bamboo_buildTimeStamp=2017-06-01T15:42:16.034-04:00, bamboo_repository_previous_revision_number=642f791962ab7c51dcf2a4b510fed555073def87, bamboo_repository_15368248_git_branch=feature/CAS-10009, bamboo_repository_6619139_revision_number=a5033aef1fd7dc1bbcb6cec79f754f5e47d37874, bamboo_repository_git_branch=feature/CAS-10009, bamboo_buildResultKey=CASA-CP90-TBFP-2, bamboo_buildPlanName=CASA - Branch Package - feature-CAS-10009 - Tag Branch for Packages, bamboo_capability_system_jdk_JDK_1_8_0_102=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-1.b14.el7_2.x86_64, bamboo_planRepository_1_revision=205cd80ac6fd7d512c22dc25996b25c2e8a669e8, bamboo_capability_system_hg_executable=/bin/hg, bamboo_repository_name=OPEN-CASA-PKG, XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt, LANG=en_US.UTF-8, bamboo_buildFailed=false, bamboo_agentWorkingDirectory=/export/home/cbt-el7-4/bamboohome/xml-data/build-dir, bamboo_planRepository_2_revision=a5033aef1fd7dc1bbcb6cec79f754f5e47d37874, bamboo_capability_system_git_executable=/bin/git, bamboo_repository_git_username=, bamboo_capability_system_jdk_JDK_1_8_0_111=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64, bamboo_capability_operating_system=EL7, bamboo_planRepository_branchDisplayName=feature/CAS-10009, bamboo_repository_15368248_name=OPEN-CASA-FULL, bamboo_capability_project=CASA, bamboo_planRepository_2_type=bbserver, bamboo_plan_storageTag=plan-15171829, bamboo_planRepository_2_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa-pkg.git, USER=casaci, bamboo_repository_git_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git, bamboo_working_directory=/export/home/cbt-el7-4/bamboohome/xml-data/build-dir/CASA-CP90-TBFP, bamboo_repository_15368248_revision_number=205cd80ac6fd7d512c22dc25996b25c2e8a669e8, bamboo_planRepository_1_repositoryUrl=ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git, bamboo_capability_system_jdk_JDK_1_8=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-1.b14.el7_2.x86_64, WRAPPER_ARCH=x86, XDG_RUNTIME_DIR=/run/user/9285, HOME=/users/casaci] build 01-Jun-2017 16:09:00 sourceDir: /export/home/cbt-el7-4/bamboohome/xml-data/build-dir/CASA-CP90-TBFP/taggitworkdir/casa build 01-Jun-2017 16:09:00 Resolving feature/bugfix tag build 01-Jun-2017 16:09:00 Branch parts: [feature, CAS-10009] build 01-Jun-2017 16:09:00 Checking for an existing tag for the branch build 01-Jun-2017 16:09:00 tagGrep: grep feature-CAS-10009 build 01-Jun-2017 16:09:00 java.lang.UNIXProcess@581ac8a8 | java.lang.UNIXProcess@6d4e5011 | java.lang.UNIXProcess@57d7f8ca | java.lang.UNIXProcess@76c3e77a build 01-Jun-2017 16:09:00 Checking out feature/CAS-10009 error 01-Jun-2017 16:09:00 Switched to branch 'feature/CAS-10009' build 01-Jun-2017 16:09:00 Your branch is behind 'origin/feature/CAS-10009' by 2 commits, and can be fast-forwarded. build 01-Jun-2017 16:09:00 (use "git pull" to update your local branch) build 01-Jun-2017 16:09:00 build 01-Jun-2017 16:09:00 Updating feature/CAS-10009 build 01-Jun-2017 16:09:01 Updating 642f791..205cd80 build 01-Jun-2017 16:09:01 Fast-forward build 01-Jun-2017 16:09:01 code/mstransform/MSTransform/StatWt.cc | 1 + build 01-Jun-2017 16:09:01 code/mstransform/TVI/StatWtTVI.cc | 127 ++++++++---- build 01-Jun-2017 16:09:01 code/mstransform/TVI/StatWtTVI.h | 10 +- build 01-Jun-2017 16:09:01 code/mstransform/TVI/test/tStatWtTVI.cc | 1 + build 01-Jun-2017 16:09:01 gcwrap/python/scripts/tests/test_statwt2.py | 287 ++++++++++++++++++++-------- build 01-Jun-2017 16:09:01 gcwrap/tasks/statwt2.xml | 149 ++++++++------- build 01-Jun-2017 16:09:01 gcwrap/tools/ms/ms.xml | 13 +- build 01-Jun-2017 16:09:01 7 files changed, 391 insertions(+), 197 deletions(-) build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 Constructing a new tag based on 5.1.0-4-feature-CAS-10009-1 build 01-Jun-2017 16:09:01 Branch: feature/CAS-10009 build 01-Jun-2017 16:09:01 Feature/bugfix detected build 01-Jun-2017 16:09:01 5.1.0-4-feature-CAS-10009-2 build 01-Jun-2017 16:09:01 Last tag: 5.1.0-4-feature-CAS-10009-1 build 01-Jun-2017 16:09:01 Branch: feature/CAS-10009 build 01-Jun-2017 16:09:01 Comparing tag to branch with: git diff --exit-code 5.1.0-4-feature-CAS-10009-1 feature/CAS-10009 build 01-Jun-2017 16:09:01 diff --git a/code/mstransform/MSTransform/StatWt.cc b/code/mstransform/MSTransform/StatWt.cc build 01-Jun-2017 16:09:01 index 4ede74e..e726f9e 100644 build 01-Jun-2017 16:09:01 --- a/code/mstransform/MSTransform/StatWt.cc build 01-Jun-2017 16:09:01 +++ b/code/mstransform/MSTransform/StatWt.cc build 01-Jun-2017 16:09:01 @@ -130,6 +130,7 @@ void StatWt::writeWeights() const { build 01-Jun-2017 16:09:01 config.defineRecord("chanbin", *_chanBinWidthQ); build 01-Jun-2017 16:09:01 } build 01-Jun-2017 16:09:01 config.define("minsamp", _minSample); build 01-Jun-2017 16:09:01 + config.define("combcorr", _combine.contains("corr")); build 01-Jun-2017 16:09:01 vi::StatWtTVILayerFactory statWtLayerFactory(config); build 01-Jun-2017 16:09:01 Vector facts(2); build 01-Jun-2017 16:09:01 facts[0] = &data; build 01-Jun-2017 16:09:01 diff --git a/code/mstransform/TVI/StatWtTVI.cc b/code/mstransform/TVI/StatWtTVI.cc build 01-Jun-2017 16:09:01 index b9f366b..d254df0 100644 build 01-Jun-2017 16:09:01 --- a/code/mstransform/TVI/StatWtTVI.cc build 01-Jun-2017 16:09:01 +++ b/code/mstransform/TVI/StatWtTVI.cc build 01-Jun-2017 16:09:01 @@ -77,6 +77,9 @@ Bool StatWtTVI::_parseConfiguration(const Record& config) { build 01-Jun-2017 16:09:01 if (config.isDefined("minsamp")) { build 01-Jun-2017 16:09:01 config.get("minsamp", _minSamp); build 01-Jun-2017 16:09:01 } build 01-Jun-2017 16:09:01 + if (config.isDefined("combcorr")) { build 01-Jun-2017 16:09:01 + config.get("combcorr", _combineCorr); build 01-Jun-2017 16:09:01 + } build 01-Jun-2017 16:09:01 return True; build 01-Jun-2017 16:09:01 } build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 @@ -188,7 +191,13 @@ void StatWtTVI::weightSpectrum(Cube & newWtsp) const { build 01-Jun-2017 16:09:01 blc[1] = biter->start; build 01-Jun-2017 16:09:01 trc[1] = biter->end; build 01-Jun-2017 16:09:01 blcb.chanBin = *biter; build 01-Jun-2017 16:09:01 - newWtsp(blc, trc) = _weights.find(blcb)->second; build 01-Jun-2017 16:09:01 + auto weights = _weights.find(blcb)->second; build 01-Jun-2017 16:09:01 + auto ncorr = weights.size(); build 01-Jun-2017 16:09:01 + for (uInt corr=0; corr & wtmat) const { build 01-Jun-2017 16:09:01 Cube flagCube; build 01-Jun-2017 16:09:01 weightSpectrum(newWtsp); build 01-Jun-2017 16:09:01 flag(flagCube); build 01-Jun-2017 16:09:01 + IPosition blc(3, 0); build 01-Jun-2017 16:09:01 + IPosition trc = newWtsp.shape() - 1; build 01-Jun-2017 16:09:01 + const auto ncorr = newWtsp.shape()[0]; build 01-Jun-2017 16:09:01 for (Int i=0; i & wtmat) const { build 01-Jun-2017 16:09:01 blcb.baseline = _baseline(ant1[i], ant2[i]); build 01-Jun-2017 16:09:01 blcb.spw = spws[1]; build 01-Jun-2017 16:09:01 blcb.chanBin = bins[0]; build 01-Jun-2017 16:09:01 - wtmat.column(i) = _weights.find(blcb)->second; build 01-Jun-2017 16:09:01 + auto weights = _weights.find(blcb)->second; build 01-Jun-2017 16:09:01 + if (_combineCorr) { build 01-Jun-2017 16:09:01 + wtmat.column(i) = weights[0]; build 01-Jun-2017 16:09:01 + } build 01-Jun-2017 16:09:01 + else { build 01-Jun-2017 16:09:01 + auto corr = 0; build 01-Jun-2017 16:09:01 + for (const auto weight: weights) { build 01-Jun-2017 16:09:01 + wtmat(corr, i) = weight; build 01-Jun-2017 16:09:01 + ++corr; build 01-Jun-2017 16:09:01 + } build 01-Jun-2017 16:09:01 + } build 01-Jun-2017 16:09:01 } build 01-Jun-2017 16:09:01 } build 01-Jun-2017 16:09:01 _newWt = wtmat.copy(); build 01-Jun-2017 16:09:01 @@ -255,6 +297,7 @@ void StatWtTVI::flag(Cube& flagCube) const { build 01-Jun-2017 16:09:01 auto nrows = nRows(); build 01-Jun-2017 16:09:01 IPosition blc(3, 0); build 01-Jun-2017 16:09:01 auto trc = flagCube.shape() - 1; build 01-Jun-2017 16:09:01 + auto ncorr = _combineCorr ? 1 : flagCube.shape()[0]; build 01-Jun-2017 16:09:01 BaselineChanBin blcb; build 01-Jun-2017 16:09:01 for (Int i=0; i& flagCube) const { build 01-Jun-2017 16:09:01 blc[1] = biter->start; build 01-Jun-2017 16:09:01 trc[1] = biter->end; build 01-Jun-2017 16:09:01 blcb.chanBin = *biter; build 01-Jun-2017 16:09:01 - if (_weights.find(blcb)->second == 0) { build 01-Jun-2017 16:09:01 - flagCube(blc, trc) = True; build 01-Jun-2017 16:09:01 + auto weights = _weights.find(blcb)->second; build 01-Jun-2017 16:09:01 + for (uInt corr=0; corrgetVisBuffer(); build 01-Jun-2017 16:09:01 _newRowIDs.resize(vii->nRowsInChunk()); build 01-Jun-2017 16:09:01 @@ -350,7 +399,6 @@ void StatWtTVI::_gatherAndComputeWeights() const { build 01-Jun-2017 16:09:01 const auto flagCube = vb->flagCube(); build 01-Jun-2017 16:09:01 const auto nrows = vb->nRows(); build 01-Jun-2017 16:09:01 const auto npol = dataCube.nrow(); build 01-Jun-2017 16:09:01 - //const auto nchan = dataCube.ncolumn(); build 01-Jun-2017 16:09:01 const auto spws = vb->spectralWindows(); build 01-Jun-2017 16:09:01 blc = dataCubeBLC; build 01-Jun-2017 16:09:01 trc = dataCubeTRC; build 01-Jun-2017 16:09:01 @@ -421,33 +469,46 @@ void StatWtTVI::_computeWeights( build 01-Jun-2017 16:09:01 auto diter = data.begin(); build 01-Jun-2017 16:09:01 auto dend = data.end(); build 01-Jun-2017 16:09:01 auto fiter = flags.begin(); build 01-Jun-2017 16:09:01 + const auto nActCorr = diter->second.shape()[0]; build 01-Jun-2017 16:09:01 + const auto ncorr = _combineCorr ? 1 : nActCorr; build 01-Jun-2017 16:09:01 for (; diter!=dend; ++diter, ++fiter) { build 01-Jun-2017 16:09:01 auto blcb = diter->first; build 01-Jun-2017 16:09:01 auto dataForBLCB = diter->second; build 01-Jun-2017 16:09:01 - const auto npts = dataForBLCB.size(); build 01-Jun-2017 16:09:01 - if (npts < _minSamp) { build 01-Jun-2017 16:09:01 - // not enough points, trivial build 01-Jun-2017 16:09:01 - _weights[blcb] = 0; build 01-Jun-2017 16:09:01 - } build 01-Jun-2017 16:09:01 - else { build 01-Jun-2017 16:09:01 - auto flagsForBLCB = fiter->second; build 01-Jun-2017 16:09:01 - if (nfalse(flagsForBLCB) < _minSamp) { build 01-Jun-2017 16:09:01 + for (uInt corr=0; corrsecond; build 01-Jun-2017 16:09:01 + auto flagChunk = flagsForBLCB(slice); build 01-Jun-2017 16:09:01 + if (nfalse(flagChunk) < _minSamp) { build 01-Jun-2017 16:09:01 + // not enough points, trivial build 01-Jun-2017 16:09:01 + _weights[blcb].push_back(0); build 01-Jun-2017 16:09:01 + } build 01-Jun-2017 16:09:01 + else { build 01-Jun-2017 16:09:01 + // some data not flagged build 01-Jun-2017 16:09:01 + const auto realPart = real(dataChunk); build 01-Jun-2017 16:09:01 + const auto imagPart = imag(dataChunk); build 01-Jun-2017 16:09:01 + const auto mask = ! flagChunk; build 01-Jun-2017 16:09:01 + const auto riter = realPart.begin(); build 01-Jun-2017 16:09:01 + const auto iiter = imagPart.begin(); build 01-Jun-2017 16:09:01 + const auto miter = mask.begin(); build 01-Jun-2017 16:09:01 + csReal.setData(riter, miter, npts); build 01-Jun-2017 16:09:01 + csImag.setData(iiter, miter, npts); build 01-Jun-2017 16:09:01 + auto varSum = csReal.getStatistic(StatisticsData::VARIANCE) build 01-Jun-2017 16:09:01 + + csImag.getStatistic(StatisticsData::VARIANCE); build 01-Jun-2017 16:09:01 + _weights[blcb].push_back(varSum == 0 ? 0 : 2/varSum); build 01-Jun-2017 16:09:01 + } build 01-Jun-2017 16:09:01 } build 01-Jun-2017 16:09:01 } build 01-Jun-2017 16:09:01 } build 01-Jun-2017 16:09:01 diff --git a/code/mstransform/TVI/StatWtTVI.h b/code/mstransform/TVI/StatWtTVI.h build 01-Jun-2017 16:09:01 index a30ff18..66fe8c0 100644 build 01-Jun-2017 16:09:01 --- a/code/mstransform/TVI/StatWtTVI.h build 01-Jun-2017 16:09:01 +++ b/code/mstransform/TVI/StatWtTVI.h build 01-Jun-2017 16:09:01 @@ -68,11 +68,6 @@ private: build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 using Baseline = std::pair; build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - // first is spw ID, second is bin within that spw build 01-Jun-2017 16:09:01 - // using ChanBin = std::pair; build 01-Jun-2017 16:09:01 - build 01-Jun-2017 16:09:01 - //using BaselineChanBin = std::pair; build 01-Jun-2017 16:09:01 - build 01-Jun-2017 16:09:01 struct ChanBin { build 01-Jun-2017 16:09:01 casacore::uInt start = 0; build 01-Jun-2017 16:09:01 casacore::uInt end = 0; build 01-Jun-2017 16:09:01 @@ -114,12 +109,15 @@ private: build 01-Jun-2017 16:09:01 mutable casacore::Cube _newFlag; build 01-Jun-2017 16:09:01 mutable casacore::Vector _newFlagRow; build 01-Jun-2017 16:09:01 mutable casacore::Vector _newRowIDs; build 01-Jun-2017 16:09:01 - mutable std::map _weights; build 01-Jun-2017 16:09:01 + // the vector represents separate correlations, there will be build 01-Jun-2017 16:09:01 + // only one element in the vector if _combineCorr is true build 01-Jun-2017 16:09:01 + mutable std::map> _weights; build 01-Jun-2017 16:09:01 // The key refers to the spw, the value vector refers to the build 01-Jun-2017 16:09:01 // channel numbers within that spw that are the first, last channel pair build 01-Jun-2017 16:09:01 // in their respective bins build 01-Jun-2017 16:09:01 map> _chanBins; build 01-Jun-2017 16:09:01 casacore::uInt _minSamp = 1; build 01-Jun-2017 16:09:01 + casacore::Bool _combineCorr = false; build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 void _gatherAndComputeWeights() const; build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 diff --git a/code/mstransform/TVI/test/tStatWtTVI.cc b/code/mstransform/TVI/test/tStatWtTVI.cc build 01-Jun-2017 16:09:01 index 993fc06..36f8e91 100755 build 01-Jun-2017 16:09:01 --- a/code/mstransform/TVI/test/tStatWtTVI.cc build 01-Jun-2017 16:09:01 +++ b/code/mstransform/TVI/test/tStatWtTVI.cc build 01-Jun-2017 16:09:01 @@ -100,6 +100,7 @@ void StatWtTVITest::testCompareTransformedData() { build 01-Jun-2017 16:09:01 MeasurementSet mstest(testFile_p); build 01-Jun-2017 16:09:01 vi::VisIterImpl2LayerFactory data(&mstest, ipar, True); build 01-Jun-2017 16:09:01 Record config; build 01-Jun-2017 16:09:01 + config.define("combcorr", True); build 01-Jun-2017 16:09:01 vi::StatWtTVILayerFactory statWtLayerFactory(config); build 01-Jun-2017 16:09:01 Vector factsTest(2); build 01-Jun-2017 16:09:01 factsTest[0] = &data; build 01-Jun-2017 16:09:01 diff --git a/gcwrap/python/scripts/tests/test_statwt2.py b/gcwrap/python/scripts/tests/test_statwt2.py build 01-Jun-2017 16:09:01 index 75fb1fd..b2513c3 100644 build 01-Jun-2017 16:09:01 --- a/gcwrap/python/scripts/tests/test_statwt2.py build 01-Jun-2017 16:09:01 +++ b/gcwrap/python/scripts/tests/test_statwt2.py build 01-Jun-2017 16:09:01 @@ -37,6 +37,7 @@ def _get_table_cols(mytb): build 01-Jun-2017 16:09:01 data = mytb.getcol("CORRECTED_DATA") build 01-Jun-2017 16:09:01 return [times, wt, wtsp, flag, frow, data] build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 +# combine correlations build 01-Jun-2017 16:09:01 def _variance(dr, di, flag, row): build 01-Jun-2017 16:09:01 fr = numpy.extract(numpy.logical_not(flag[:,:,row]), dr[:,:,row]) build 01-Jun-2017 16:09:01 fi = numpy.extract(numpy.logical_not(flag[:,:,row]), di[:,:,row]) build 01-Jun-2017 16:09:01 @@ -47,6 +48,18 @@ def _variance(dr, di, flag, row): build 01-Jun-2017 16:09:01 vi = numpy.var(fi, ddof=1) build 01-Jun-2017 16:09:01 return 2/(vr + vi) build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 +# per correlation build 01-Jun-2017 16:09:01 +def _variance2(dr, di, flag, corr, row): build 01-Jun-2017 16:09:01 + fr = numpy.extract(numpy.logical_not(flag[corr,:,row]), dr[corr,:,row]) build 01-Jun-2017 16:09:01 + fi = numpy.extract(numpy.logical_not(flag[corr,:,row]), di[corr,:,row]) build 01-Jun-2017 16:09:01 + if len(fr) <= 1: build 01-Jun-2017 16:09:01 + return 0 build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + vr = numpy.var(fr, ddof=1) build 01-Jun-2017 16:09:01 + vi = numpy.var(fi, ddof=1) build 01-Jun-2017 16:09:01 + return 2/(vr + vi) build 01-Jun-2017 16:09:01 + build 01-Jun-2017 16:09:01 + build 01-Jun-2017 16:09:01 class statwt2_test(unittest.TestCase): build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 def test_algorithm(self): build 01-Jun-2017 16:09:01 @@ -57,37 +70,78 @@ class statwt2_test(unittest.TestCase): build 01-Jun-2017 16:09:01 expfrow = mytb.getcol("FLAG_ROW") build 01-Jun-2017 16:09:01 mytb.done() build 01-Jun-2017 16:09:01 dst = "ngc5921.split.ms" build 01-Jun-2017 16:09:01 - for i in [0,1]: build 01-Jun-2017 16:09:01 - shutil.copytree(src, dst) build 01-Jun-2017 16:09:01 - myms = mstool() build 01-Jun-2017 16:09:01 - if i == 0: build 01-Jun-2017 16:09:01 - myms.open(dst, nomodify=False) build 01-Jun-2017 16:09:01 - myms.statwt2() build 01-Jun-2017 16:09:01 - myms.done() build 01-Jun-2017 16:09:01 - else: build 01-Jun-2017 16:09:01 - statwt2(dst) build 01-Jun-2017 16:09:01 - [wt, wtsp, flag, frow, data] = _get_dst_cols(dst) build 01-Jun-2017 16:09:01 - dr = numpy.real(data) build 01-Jun-2017 16:09:01 - di = numpy.imag(data) build 01-Jun-2017 16:09:01 - myshape = wtsp.shape build 01-Jun-2017 16:09:01 - ncorr = myshape[0] build 01-Jun-2017 16:09:01 - nrow = myshape[2] build 01-Jun-2017 16:09:01 - rtol = 1e-7 build 01-Jun-2017 16:09:01 - for row in range(nrow): build 01-Jun-2017 16:09:01 - expec = _variance(dr, di, flag, row) build 01-Jun-2017 16:09:01 - self.assertTrue( build 01-Jun-2017 16:09:01 - numpy.all(numpy.isclose(wt[:, row], expec, rtol=rtol)), build 01-Jun-2017 16:09:01 - "WEIGHT fail at row" + str(row) + ". got: " + str(wt[:, row]) + " expec " + str(expec) build 01-Jun-2017 16:09:01 - ) build 01-Jun-2017 16:09:01 - self.assertTrue(len(numpy.unique(wtsp[:,:,row])) == 1, "Weight values are not the same") build 01-Jun-2017 16:09:01 - self.assertTrue(numpy.all(numpy.isclose(wtsp[:,:,row], expec, rtol)), "Incorrect weights") build 01-Jun-2017 16:09:01 - if expec == 0: build 01-Jun-2017 16:09:01 - self.assertTrue(numpy.all(flag[:,:,row]), "Not all flags are true") build 01-Jun-2017 16:09:01 - self.assertTrue(frow[row], "FLAG_ROW is not true") build 01-Jun-2017 16:09:01 + rtol = 1e-7 build 01-Jun-2017 16:09:01 + for combine in ["", "corr"]: build 01-Jun-2017 16:09:01 + for i in [0,1]: build 01-Jun-2017 16:09:01 + shutil.copytree(src, dst) build 01-Jun-2017 16:09:01 + myms = mstool() build 01-Jun-2017 16:09:01 + if i == 0: build 01-Jun-2017 16:09:01 + myms.open(dst, nomodify=False) build 01-Jun-2017 16:09:01 + myms.statwt2(combine=combine) build 01-Jun-2017 16:09:01 + myms.done() build 01-Jun-2017 16:09:01 else: build 01-Jun-2017 16:09:01 - self.assertTrue(numpy.all(flag[:,:,row] == expflag[:,:,row]), "FLAGs don't match") build 01-Jun-2017 16:09:01 - self.assertTrue(frow[row] == expfrow[row], "FLAG_ROW doesn't match") build 01-Jun-2017 16:09:01 - shutil.rmtree(dst) build 01-Jun-2017 16:09:01 + statwt2(dst, combine=combine) build 01-Jun-2017 16:09:01 + [wt, wtsp, flag, frow, data] = _get_dst_cols(dst) build 01-Jun-2017 16:09:01 + dr = numpy.real(data) build 01-Jun-2017 16:09:01 + di = numpy.imag(data) build 01-Jun-2017 16:09:01 + myshape = wtsp.shape build 01-Jun-2017 16:09:01 + ncorr = myshape[0] build 01-Jun-2017 16:09:01 + nrow = myshape[2] build 01-Jun-2017 16:09:01 + if (combine == "corr"): build 01-Jun-2017 16:09:01 + for row in range(nrow): build 01-Jun-2017 16:09:01 + expec = _variance(dr, di, flag, row) build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all(numpy.isclose(wt[:, row], expec, rtol=rtol)), build 01-Jun-2017 16:09:01 + "WEIGHT fail at row" + str(row) + ". got: " + str(wt[:, row]) build 01-Jun-2017 16:09:01 + + " expec " + str(expec) build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + len(numpy.unique(wtsp[:,:,row])) == 1, build 01-Jun-2017 16:09:01 + "Weight values are not the same" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all(numpy.isclose(wtsp[:,:,row], expec, rtol)), build 01-Jun-2017 16:09:01 + "Incorrect weights" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + if expec == 0: build 01-Jun-2017 16:09:01 + self.assertTrue(numpy.all(flag[:,:,row]), "Not all flags are true") build 01-Jun-2017 16:09:01 + self.assertTrue(frow[row], "FLAG_ROW is not true") build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all(flag[:,:,row] == expflag[:,:,row]), build 01-Jun-2017 16:09:01 + "FLAGs don't match" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + frow[row] == expfrow[row], "FLAG_ROW doesn't match" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + for row in range(nrow): build 01-Jun-2017 16:09:01 + for corr in range(ncorr): build 01-Jun-2017 16:09:01 + expec = _variance2(dr, di, flag, corr, row) build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.isclose(wt[corr, row], expec, rtol=rtol), build 01-Jun-2017 16:09:01 + "WEIGHT fail at row" + str(row) + ". got: " + str(wt[corr, row]) build 01-Jun-2017 16:09:01 + + " expec " + str(expec) build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + len(numpy.unique(wtsp[corr,:,row])) == 1, build 01-Jun-2017 16:09:01 + "Weight values are not the same" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all( build 01-Jun-2017 16:09:01 + numpy.isclose(wtsp[corr,:,row], expec, rtol)), "Incorrect weights" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + if expec == 0: build 01-Jun-2017 16:09:01 + self.assertTrue(numpy.all(flag[corr,:,row]), "Not all flags are true") build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all(flag[corr,:,row] == expflag[corr,:,row]), "FLAGs don't match" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + if (numpy.all(flag[:,:,row])): build 01-Jun-2017 16:09:01 + self.assertTrue(frow[row], "FLAG_ROW is not true") build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + self.assertFalse(frow[row], "FLAG_ROW is not false") build 01-Jun-2017 16:09:01 + shutil.rmtree(dst) build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 def test_timebin(self): build 01-Jun-2017 16:09:01 """ Test time binning""" build 01-Jun-2017 16:09:01 @@ -95,16 +149,17 @@ class statwt2_test(unittest.TestCase): build 01-Jun-2017 16:09:01 ref = datadir + "ngc5921.timebin300s_1.ms.ref" build 01-Jun-2017 16:09:01 [refwt, refwtsp, refflag, reffrow, refdata] = _get_dst_cols(ref) build 01-Jun-2017 16:09:01 rtol = 1e-7 build 01-Jun-2017 16:09:01 + combine = "corr" build 01-Jun-2017 16:09:01 for timebin in ["300s", 10]: build 01-Jun-2017 16:09:01 for i in [0, 1]: build 01-Jun-2017 16:09:01 shutil.copytree(src, dst) build 01-Jun-2017 16:09:01 myms = mstool() build 01-Jun-2017 16:09:01 if i == 0: build 01-Jun-2017 16:09:01 myms.open(dst, nomodify=False) build 01-Jun-2017 16:09:01 - myms.statwt2(timebin=timebin) build 01-Jun-2017 16:09:01 + myms.statwt2(timebin=timebin, combine=combine) build 01-Jun-2017 16:09:01 myms.done() build 01-Jun-2017 16:09:01 else: build 01-Jun-2017 16:09:01 - statwt2(dst, timebin=timebin) build 01-Jun-2017 16:09:01 + statwt2(dst, timebin=timebin, combine=combine) build 01-Jun-2017 16:09:01 [tstwt, tstwtsp, tstflag, tstfrow, tstdata] = _get_dst_cols(dst) build 01-Jun-2017 16:09:01 self.assertTrue(numpy.all(tstflag == refflag), "FLAGs don't match") build 01-Jun-2017 16:09:01 self.assertTrue(numpy.all(tstfrow == reffrow), "FLAG_ROWs don't match") build 01-Jun-2017 16:09:01 @@ -120,67 +175,129 @@ class statwt2_test(unittest.TestCase): build 01-Jun-2017 16:09:01 expfrow = mytb.getcol("FLAG_ROW") build 01-Jun-2017 16:09:01 mytb.done() build 01-Jun-2017 16:09:01 dst = "ngc5921.split.ms" build 01-Jun-2017 16:09:01 - for i in [0,1]: build 01-Jun-2017 16:09:01 - for chanbin in ["195.312kHz", 8]: build 01-Jun-2017 16:09:01 - shutil.copytree(src, dst) build 01-Jun-2017 16:09:01 - if i == 0: build 01-Jun-2017 16:09:01 - myms = mstool() build 01-Jun-2017 16:09:01 - myms.open(dst, nomodify=False) build 01-Jun-2017 16:09:01 - myms.statwt2(chanbin=chanbin) build 01-Jun-2017 16:09:01 - myms.done() build 01-Jun-2017 16:09:01 - else: build 01-Jun-2017 16:09:01 - statwt2(dst, chanbin=chanbin) build 01-Jun-2017 16:09:01 - [wt, wtsp, flag, frow, data] = _get_dst_cols(dst) build 01-Jun-2017 16:09:01 - dr = numpy.real(data) build 01-Jun-2017 16:09:01 - di = numpy.imag(data) build 01-Jun-2017 16:09:01 - myshape = wtsp.shape build 01-Jun-2017 16:09:01 - ncorr = myshape[0] build 01-Jun-2017 16:09:01 - nrow = myshape[2] build 01-Jun-2017 16:09:01 - rtol = 1e-7 build 01-Jun-2017 16:09:01 - for row in range(nrow): build 01-Jun-2017 16:09:01 - for start in (0, 8, 16, 24, 32, 40, 48, 56): build 01-Jun-2017 16:09:01 - end = start + 8 build 01-Jun-2017 16:09:01 - fr = numpy.extract(numpy.logical_not(flag[:,start:end,row]), dr[:,start:end,row]) build 01-Jun-2017 16:09:01 - fi = numpy.extract(numpy.logical_not(flag[:,start:end,row]), di[:,start:end,row]) build 01-Jun-2017 16:09:01 - if len(fr) <= 1: build 01-Jun-2017 16:09:01 - expec = 0 build 01-Jun-2017 16:09:01 - else: build 01-Jun-2017 16:09:01 - vr = numpy.var(fr, ddof=1) build 01-Jun-2017 16:09:01 - vi = numpy.var(fi, ddof=1) build 01-Jun-2017 16:09:01 - expec = 2/(vr + vi) build 01-Jun-2017 16:09:01 - self.assertTrue(numpy.all(numpy.isclose(wtsp[:,start:end,row], expec, rtol)), "Incorrect weight spectrum") build 01-Jun-2017 16:09:01 - if expec == 0: build 01-Jun-2017 16:09:01 - self.assertTrue(numpy.all(flag[:,start:end,row]), "Not all flags are true") build 01-Jun-2017 16:09:01 - else: build 01-Jun-2017 16:09:01 - self.assertTrue(numpy.all(flag[:,start:end,row] == expflag[:,start:end,row]), "FLAGs don't match") build 01-Jun-2017 16:09:01 - wts = numpy.extract(numpy.logical_not(flag[:,:,row]), wtsp[:,:,row]) build 01-Jun-2017 16:09:01 - if len(wts) > 0: build 01-Jun-2017 16:09:01 - expwt = numpy.median(wts) build 01-Jun-2017 16:09:01 + for combine in ["", "corr"]: build 01-Jun-2017 16:09:01 + for i in [0,1]: build 01-Jun-2017 16:09:01 + for chanbin in ["195.312kHz", 8]: build 01-Jun-2017 16:09:01 + shutil.copytree(src, dst) build 01-Jun-2017 16:09:01 + if i == 0: build 01-Jun-2017 16:09:01 + myms = mstool() build 01-Jun-2017 16:09:01 + myms.open(dst, nomodify=False) build 01-Jun-2017 16:09:01 + myms.statwt2(chanbin=chanbin, combine=combine) build 01-Jun-2017 16:09:01 + myms.done() build 01-Jun-2017 16:09:01 else: build 01-Jun-2017 16:09:01 - expwt = 0 build 01-Jun-2017 16:09:01 - self.assertTrue( build 01-Jun-2017 16:09:01 - numpy.all(numpy.isclose(wt[:, row], expwt, rtol=rtol)), build 01-Jun-2017 16:09:01 - "WEIGHT fail at row" + str(row) + ". got: " + str(wt[:, row]) + " expec " + str(expec) build 01-Jun-2017 16:09:01 - ) build 01-Jun-2017 16:09:01 - if expec == 0: build 01-Jun-2017 16:09:01 - self.assertTrue(frow[row], "FLAG_ROW is not true") build 01-Jun-2017 16:09:01 + statwt2(dst, chanbin=chanbin, combine=combine) build 01-Jun-2017 16:09:01 + [wt, wtsp, flag, frow, data] = _get_dst_cols(dst) build 01-Jun-2017 16:09:01 + dr = numpy.real(data) build 01-Jun-2017 16:09:01 + di = numpy.imag(data) build 01-Jun-2017 16:09:01 + myshape = wtsp.shape build 01-Jun-2017 16:09:01 + ncorr = myshape[0] build 01-Jun-2017 16:09:01 + nrow = myshape[2] build 01-Jun-2017 16:09:01 + rtol = 1e-7 build 01-Jun-2017 16:09:01 + if combine == "corr": build 01-Jun-2017 16:09:01 + # speed it up build 01-Jun-2017 16:09:01 + for row in range(0, nrow, 11): build 01-Jun-2017 16:09:01 + for start in (0, 8, 16, 24, 32, 40, 48, 56): build 01-Jun-2017 16:09:01 + end = start + 8 build 01-Jun-2017 16:09:01 + fr = numpy.extract( build 01-Jun-2017 16:09:01 + numpy.logical_not(flag[:,start:end,row]), dr[:,start:end,row] build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + fi = numpy.extract( build 01-Jun-2017 16:09:01 + numpy.logical_not(flag[:,start:end,row]), di[:,start:end,row] build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + if len(fr) <= 1: build 01-Jun-2017 16:09:01 + expec = 0 build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + vr = numpy.var(fr, ddof=1) build 01-Jun-2017 16:09:01 + vi = numpy.var(fi, ddof=1) build 01-Jun-2017 16:09:01 + expec = 2/(vr + vi) build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all(numpy.isclose(wtsp[:,start:end,row], expec, rtol)), build 01-Jun-2017 16:09:01 + "Incorrect weight spectrum" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + if expec == 0: build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all(flag[:,start:end,row]), "Not all flags are true" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all(flag[:,start:end,row] == expflag[:,start:end,row]), build 01-Jun-2017 16:09:01 + "FLAGs don't match" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + wts = numpy.extract(numpy.logical_not(flag[:,:,row]), wtsp[:,:,row]) build 01-Jun-2017 16:09:01 + if len(wts) > 0: build 01-Jun-2017 16:09:01 + expwt = numpy.median(wts) build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + expwt = 0 build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all( build 01-Jun-2017 16:09:01 + numpy.isclose(wt[:, row], expwt, rtol=rtol)), build 01-Jun-2017 16:09:01 + "WEIGHT fail at row" + str(row) + ". got: " + str(wt[:, row]) build 01-Jun-2017 16:09:01 + + " expec " + str(expec) build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + if expec == 0: build 01-Jun-2017 16:09:01 + self.assertTrue(frow[row], "FLAG_ROW is not true") build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + self.assertFalse(frow[row], "FLAG_ROW is not false") build 01-Jun-2017 16:09:01 else: build 01-Jun-2017 16:09:01 - self.assertFalse(frow[row], "FLAG_ROW is not false") build 01-Jun-2017 16:09:01 - shutil.rmtree(dst) build 01-Jun-2017 16:09:01 + for row in range(0, nrow, 11): build 01-Jun-2017 16:09:01 + for start in (0, 8, 16, 24, 32, 40, 48, 56): build 01-Jun-2017 16:09:01 + end = start + 8 build 01-Jun-2017 16:09:01 + for corr in range(ncorr): build 01-Jun-2017 16:09:01 + fr = numpy.extract( build 01-Jun-2017 16:09:01 + numpy.logical_not(flag[corr,start:end,row]), dr[corr,start:end,row] build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + fi = numpy.extract( build 01-Jun-2017 16:09:01 + numpy.logical_not(flag[corr,start:end,row]), di[corr,start:end,row] build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + if len(fr) <= 1: build 01-Jun-2017 16:09:01 + expec = 0 build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + vr = numpy.var(fr, ddof=1) build 01-Jun-2017 16:09:01 + vi = numpy.var(fi, ddof=1) build 01-Jun-2017 16:09:01 + expec = 2/(vr + vi) build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all(numpy.isclose(wtsp[corr,start:end,row], expec, rtol)), build 01-Jun-2017 16:09:01 + "Incorrect weight spectrum" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + if expec == 0: build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all(flag[corr,start:end,row]), "Not all flags are true" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.all(flag[corr,start:end,row] == expflag[corr,start:end,row]), build 01-Jun-2017 16:09:01 + "FLAGs don't match" build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + wts = numpy.extract(numpy.logical_not(flag[corr,:,row]), wtsp[corr,:,row]) build 01-Jun-2017 16:09:01 + if len(wts) > 0: build 01-Jun-2017 16:09:01 + expwt = numpy.median(wts) build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + expwt = 0 build 01-Jun-2017 16:09:01 + self.assertTrue( build 01-Jun-2017 16:09:01 + numpy.isclose(wt[corr, row], expwt, rtol=rtol), build 01-Jun-2017 16:09:01 + "WEIGHT fail at row" + str(row) + ". got: " + str(wt[:, row]) build 01-Jun-2017 16:09:01 + + " expec " + str(expec) build 01-Jun-2017 16:09:01 + ) build 01-Jun-2017 16:09:01 + if numpy.all(flag[:,:,row]): build 01-Jun-2017 16:09:01 + self.assertTrue(frow[row], "FLAG_ROW is not true") build 01-Jun-2017 16:09:01 + else: build 01-Jun-2017 16:09:01 + self.assertFalse(frow[row], "FLAG_ROW is not false") build 01-Jun-2017 16:09:01 + shutil.rmtree(dst) build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 def test_minsamp(self): build 01-Jun-2017 16:09:01 """Test minimum number of points""" build 01-Jun-2017 16:09:01 dst = "ngc5921.split.minsamp.ms" build 01-Jun-2017 16:09:01 + combine = "corr" build 01-Jun-2017 16:09:01 for i in [0,1]: build 01-Jun-2017 16:09:01 for minsamp in [60, 80]: build 01-Jun-2017 16:09:01 shutil.copytree(src, dst) build 01-Jun-2017 16:09:01 if i == 0: build 01-Jun-2017 16:09:01 myms = mstool() build 01-Jun-2017 16:09:01 myms.open(dst, nomodify=False) build 01-Jun-2017 16:09:01 - myms.statwt2(minsamp=minsamp) build 01-Jun-2017 16:09:01 + myms.statwt2(minsamp=minsamp, combine=combine) build 01-Jun-2017 16:09:01 myms.done() build 01-Jun-2017 16:09:01 else: build 01-Jun-2017 16:09:01 - statwt2(dst, minsamp=minsamp) build 01-Jun-2017 16:09:01 + statwt2(dst, minsamp=minsamp, combine=combine) build 01-Jun-2017 16:09:01 [wt, wtsp, flag, frow, data] = _get_dst_cols(dst) build 01-Jun-2017 16:09:01 if minsamp == 60: build 01-Jun-2017 16:09:01 self.assertTrue((wt[:, 30] > 0).all(), "Incorrect weight row 30") build 01-Jun-2017 16:09:01 @@ -197,11 +314,12 @@ class statwt2_test(unittest.TestCase): build 01-Jun-2017 16:09:01 def test_fieldsel(self): build 01-Jun-2017 16:09:01 """Test field selection""" build 01-Jun-2017 16:09:01 dst = "ngc5921.split.fieldsel.ms" build 01-Jun-2017 16:09:01 + combine = "corr" build 01-Jun-2017 16:09:01 [origwt, origwtsp, origflag, origfrow, origdata] = _get_dst_cols(src) build 01-Jun-2017 16:09:01 rtol = 1e-7 build 01-Jun-2017 16:09:01 for field in ["2", "N5921_2"]: build 01-Jun-2017 16:09:01 shutil.copytree(src, dst) build 01-Jun-2017 16:09:01 - statwt2(dst, field=field) build 01-Jun-2017 16:09:01 + statwt2(dst, field=field, combine=combine) build 01-Jun-2017 16:09:01 [wt, wtsp, flag, frow, data, field_id] = _get_dst_cols(dst, "FIELD_ID") build 01-Jun-2017 16:09:01 nrow = len(frow) build 01-Jun-2017 16:09:01 dr = numpy.real(data) build 01-Jun-2017 16:09:01 @@ -233,12 +351,13 @@ class statwt2_test(unittest.TestCase): build 01-Jun-2017 16:09:01 def test_spwsel(self): build 01-Jun-2017 16:09:01 """Test spw selection""" build 01-Jun-2017 16:09:01 dst = "ngc5921.split.spwsel.ms" build 01-Jun-2017 16:09:01 + combine = "corr" build 01-Jun-2017 16:09:01 [origwt, origwtsp, origflag, origfrow, origdata] = _get_dst_cols(src) build 01-Jun-2017 16:09:01 rtol = 1e-7 build 01-Jun-2017 16:09:01 spw="0" build 01-Jun-2017 16:09:01 # data set only has one spw build 01-Jun-2017 16:09:01 shutil.copytree(src, dst) build 01-Jun-2017 16:09:01 - statwt2(dst, spw=spw) build 01-Jun-2017 16:09:01 + statwt2(dst, spw=spw, combine=combine) build 01-Jun-2017 16:09:01 [wt, wtsp, flag, frow, data] = _get_dst_cols(dst) build 01-Jun-2017 16:09:01 nrow = len(frow) build 01-Jun-2017 16:09:01 dr = numpy.real(data) build 01-Jun-2017 16:09:01 @@ -266,7 +385,7 @@ class statwt2_test(unittest.TestCase): build 01-Jun-2017 16:09:01 # there are three field_ids, and there is a change in field_id when build 01-Jun-2017 16:09:01 # there is a change in scan number, so specifying combine="field" in the build 01-Jun-2017 16:09:01 # absence of "scan" will give the same result as combine="" build 01-Jun-2017 16:09:01 - for combine in ["", "field"]: build 01-Jun-2017 16:09:01 + for combine in ["corr", "corr,field"]: build 01-Jun-2017 16:09:01 for i in [0, 1]: build 01-Jun-2017 16:09:01 shutil.copytree(src, dst) build 01-Jun-2017 16:09:01 if i == 0: build 01-Jun-2017 16:09:01 @@ -290,7 +409,7 @@ class statwt2_test(unittest.TestCase): build 01-Jun-2017 16:09:01 ref = datadir + "ngc5921.no_scan_bounds.ms.ref" build 01-Jun-2017 16:09:01 rtol = 1e-7 build 01-Jun-2017 16:09:01 [expwt, expwtsp, expflag, expfrow, expdata] = _get_dst_cols(ref) build 01-Jun-2017 16:09:01 - combine = "scan" build 01-Jun-2017 16:09:01 + combine = "corr, scan" build 01-Jun-2017 16:09:01 for i in [0, 1]: build 01-Jun-2017 16:09:01 shutil.copytree(src, dst) build 01-Jun-2017 16:09:01 if i == 0: build 01-Jun-2017 16:09:01 @@ -314,7 +433,7 @@ class statwt2_test(unittest.TestCase): build 01-Jun-2017 16:09:01 ref = datadir + "ngc5921.no_scan_nor_field_bounds.ms.ref" build 01-Jun-2017 16:09:01 rtol = 1e-7 build 01-Jun-2017 16:09:01 [expwt, expwtsp, expflag, expfrow, expdata] = _get_dst_cols(ref) build 01-Jun-2017 16:09:01 - for combine in ["scan,field", "field,scan"]: build 01-Jun-2017 16:09:01 + for combine in ["corr,scan,field", "corr,field,scan"]: build 01-Jun-2017 16:09:01 for i in [0, 1]: build 01-Jun-2017 16:09:01 shutil.copytree(src, dst) build 01-Jun-2017 16:09:01 if i == 0: build 01-Jun-2017 16:09:01 diff --git a/gcwrap/tasks/statwt2.xml b/gcwrap/tasks/statwt2.xml build 01-Jun-2017 16:09:01 index 10ab68f..0dbf7fe 100644 build 01-Jun-2017 16:09:01 --- a/gcwrap/tasks/statwt2.xml build 01-Jun-2017 16:09:01 +++ b/gcwrap/tasks/statwt2.xml build 01-Jun-2017 16:09:01 @@ -62,96 +62,103 @@ file:///opt/casa/code/xmlcasa/xml/casa.xsd"> build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - THIS APPLICATION IS UNDER DEVELOPMENT AND CURRENTLY FOR TESTING ONLY! USE AT YOUR OWN RISK! build 01-Jun-2017 16:09:01 + THIS APPLICATION IS UNDER DEVELOPMENT AND CURRENTLY FOR TESTING ONLY! USE AT YOUR OWN RISK! build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - THIS APPLICATION MODIFIES THE WEIGHT, WEIGHT SPECTRUM, FLAG, AND FLAG_ROW COLUMNS OF THE INPUT build 01-Jun-2017 16:09:01 - MS. IF YOU WANT A PRISTINE COPY OF THE INPUT MS TO BE PRESERVED, MAKE A COPY OF IT BEFORE build 01-Jun-2017 16:09:01 - RUNNING THIS APPLICATION. build 01-Jun-2017 16:09:01 + THIS APPLICATION MODIFIES THE WEIGHT, WEIGHT SPECTRUM, FLAG, AND FLAG_ROW COLUMNS OF THE INPUT build 01-Jun-2017 16:09:01 + MS. IF YOU WANT A PRISTINE COPY OF THE INPUT MS TO BE PRESERVED, MAKE A COPY OF IT BEFORE build 01-Jun-2017 16:09:01 + RUNNING THIS APPLICATION. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - This application computes weights for the WEIGHT and WEIGHT_SPECTRUM (if present) columns build 01-Jun-2017 16:09:01 - based on the variance of values in the CORRECTED_DATA column. If the MS has no CORRECTED_DATA build 01-Jun-2017 16:09:01 - column, the application will fail. The following algorithm is used: build 01-Jun-2017 16:09:01 + This application computes weights for the WEIGHT and WEIGHT_SPECTRUM (if present) columns build 01-Jun-2017 16:09:01 + based on the variance of values in the CORRECTED_DATA column. If the MS has no CORRECTED_DATA build 01-Jun-2017 16:09:01 + column, the application will fail. The following algorithm is used: build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - 1. For unflagged data, create two sets of values, one set is composed solely of the real part build 01-Jun-2017 16:09:01 - of the data values, the other set is composed solely of the imaginary part of the data build 01-Jun-2017 16:09:01 - values. build 01-Jun-2017 16:09:01 - 2. Compute the variance of each of these sets, vr and vi. build 01-Jun-2017 16:09:01 - 3. Compute veq = (vr + vi)/2. build 01-Jun-2017 16:09:01 - 4. The associated weight is just the reciprocal of veq. The weight will have unit build 01-Jun-2017 16:09:01 - of (data unit)^(-2), eg Jy^(-2). build 01-Jun-2017 16:09:01 + 1. For unflagged data, create two sets of values, one set is composed solely of the real part build 01-Jun-2017 16:09:01 + of the data values, the other set is composed solely of the imaginary part of the data build 01-Jun-2017 16:09:01 + values. build 01-Jun-2017 16:09:01 + 2. Compute the variance of each of these sets, vr and vi. build 01-Jun-2017 16:09:01 + 3. Compute veq = (vr + vi)/2. build 01-Jun-2017 16:09:01 + 4. The associated weight is just the reciprocal of veq. The weight will have unit build 01-Jun-2017 16:09:01 + of (data unit)^(-2), eg Jy^(-2). build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - Data are aggregated on a per-baseline, per-data description ID basis. Data are aggregated build 01-Jun-2017 16:09:01 - in bins determined by the specified values of the timebin and chanbin parameters. build 01-Jun-2017 16:09:01 + Data are aggregated on a per-baseline, per-data description ID basis. Data are aggregated build 01-Jun-2017 16:09:01 + in bins determined by the specified values of the timebin and chanbin parameters. By default, build 01-Jun-2017 16:09:01 + data for separate correlations are aggregated separately. This behavior can be overriden build 01-Jun-2017 16:09:01 + by specifying combine="corr" (see below). build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - TIME BINNING build 01-Jun-2017 16:09:01 + TIME BINNING build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - If the value of timebin is an integer, it means that the specified value should be build 01-Jun-2017 16:09:01 - multiplied by the representative integration time in the MS. This integration is the build 01-Jun-2017 16:09:01 - median value of all the values in the INTERVAL column. Flags are not considered in build 01-Jun-2017 16:09:01 - the integration time computation. If either extrema in the INTERVAL column differs from build 01-Jun-2017 16:09:01 - the median by more than 25%, the application will fail because the values vary too much build 01-Jun-2017 16:09:01 - for there to be a single, representative, integration time. The timebin parameter can build 01-Jun-2017 16:09:01 - also be specified as a quantity (string) that must have time conformant units. build 01-Jun-2017 16:09:01 + If the value of timebin is an integer, it means that the specified value should be build 01-Jun-2017 16:09:01 + multiplied by the representative integration time in the MS. This integration is the build 01-Jun-2017 16:09:01 + median value of all the values in the INTERVAL column. Flags are not considered in build 01-Jun-2017 16:09:01 + the integration time computation. If either extrema in the INTERVAL column differs from build 01-Jun-2017 16:09:01 + the median by more than 25%, the application will fail because the values vary too much build 01-Jun-2017 16:09:01 + for there to be a single, representative, integration time. The timebin parameter can build 01-Jun-2017 16:09:01 + also be specified as a quantity (string) that must have time conformant units. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - The time bins are not necessarily contiguous and are not necessarily the same width. The start build 01-Jun-2017 16:09:01 - of a bin is always coincident with a value from the TIME column, So for example, if values build 01-Jun-2017 16:09:01 - from the time column are [20, 60, 100, 140, 180, 230], and the width of the bins is chosen build 01-Jun-2017 16:09:01 - to be 110s, the first bin would start at 20s and run to 130s, so that data from timestamps build 01-Jun-2017 16:09:01 - 20, 60, and 100 will be included in the first bin. The second bin would start at 140s, so that build 01-Jun-2017 16:09:01 - data for timestamps 140, 180, and 230 would be included in the second bin. Also, time binning build 01-Jun-2017 16:09:01 - does not span scan boundaries, so that data associated with different scan numbers will build 01-Jun-2017 16:09:01 - always be binned separately; changes in SCAN_NUMBER will cause a new time bin to be created, build 01-Jun-2017 16:09:01 - with its starting value coincident with the time of the new SCAN_NUMBER. Similar behavior can build 01-Jun-2017 16:09:01 - be expected for changes in FIELD_ID and ARRAY_ID. One can override this behavior for some build 01-Jun-2017 16:09:01 - columns by specifying the combine parameter (see below). build 01-Jun-2017 16:09:01 + The time bins are not necessarily contiguous and are not necessarily the same width. The start build 01-Jun-2017 16:09:01 + of a bin is always coincident with a value from the TIME column, So for example, if values build 01-Jun-2017 16:09:01 + from the time column are [20, 60, 100, 140, 180, 230], and the width of the bins is chosen build 01-Jun-2017 16:09:01 + to be 110s, the first bin would start at 20s and run to 130s, so that data from timestamps build 01-Jun-2017 16:09:01 + 20, 60, and 100 will be included in the first bin. The second bin would start at 140s, so that build 01-Jun-2017 16:09:01 + data for timestamps 140, 180, and 230 would be included in the second bin. Also, time binning build 01-Jun-2017 16:09:01 + does not span scan boundaries, so that data associated with different scan numbers will build 01-Jun-2017 16:09:01 + always be binned separately; changes in SCAN_NUMBER will cause a new time bin to be created, build 01-Jun-2017 16:09:01 + with its starting value coincident with the time of the new SCAN_NUMBER. Similar behavior can build 01-Jun-2017 16:09:01 + be expected for changes in FIELD_ID and ARRAY_ID. One can override this behavior for some build 01-Jun-2017 16:09:01 + columns by specifying the combine parameter (see below). build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - CHANNEL BINNING build 01-Jun-2017 16:09:01 + CHANNEL BINNING build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - The width of channel bins is specified via the chanbin parameter. Channel binning occurs within build 01-Jun-2017 16:09:01 - individual spectral windows; bins never span multiple spectral windows. Each channel will build 01-Jun-2017 16:09:01 - be included in exactly one bin. build 01-Jun-2017 16:09:01 + The width of channel bins is specified via the chanbin parameter. Channel binning occurs within build 01-Jun-2017 16:09:01 + individual spectral windows; bins never span multiple spectral windows. Each channel will build 01-Jun-2017 16:09:01 + be included in exactly one bin. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - The default value "spw" indicates that all channels in each spectral window are to be build 01-Jun-2017 16:09:01 - included in a single bin. build 01-Jun-2017 16:09:01 + The default value "spw" indicates that all channels in each spectral window are to be build 01-Jun-2017 16:09:01 + included in a single bin. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - Any other string value is interpreted as a quantity, and so should have frequency units, eg build 01-Jun-2017 16:09:01 - "1MHz". In this case, the channel frequencies from the CHAN_FREQ column of the SPECTRAL_WINDOW build 01-Jun-2017 16:09:01 - subtable of the MS are used to determine the bins. The first bin starts at the channel frequency build 01-Jun-2017 16:09:01 - of the 0th channel in the spectral window. Channels with frequencies that differ by less than build 01-Jun-2017 16:09:01 - the value specified by the chanbin parameter are included in this bin. The next bin starts at build 01-Jun-2017 16:09:01 - the frequency of the first channel outside the first bin, and the process is repeated until all build 01-Jun-2017 16:09:01 - channels have been binned. build 01-Jun-2017 16:09:01 + Any other string value is interpreted as a quantity, and so should have frequency units, eg build 01-Jun-2017 16:09:01 + "1MHz". In this case, the channel frequencies from the CHAN_FREQ column of the SPECTRAL_WINDOW build 01-Jun-2017 16:09:01 + subtable of the MS are used to determine the bins. The first bin starts at the channel frequency build 01-Jun-2017 16:09:01 + of the 0th channel in the spectral window. Channels with frequencies that differ by less than build 01-Jun-2017 16:09:01 + the value specified by the chanbin parameter are included in this bin. The next bin starts at build 01-Jun-2017 16:09:01 + the frequency of the first channel outside the first bin, and the process is repeated until all build 01-Jun-2017 16:09:01 + channels have been binned. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - If specified as an integer, the value is interpreted as the number of channels to include in build 01-Jun-2017 16:09:01 - each bin. The final bin in the spectral window may not necessarily contain this number of build 01-Jun-2017 16:09:01 - channels. For example, if a spectral window has 15 channels, and chanbin is specified to be 6, build 01-Jun-2017 16:09:01 - then channels 0-5 will comprise the first bin, channels 6-11 the second, and channels 12-14 the build 01-Jun-2017 16:09:01 - third, so that only three channels will comprise the final bin. build 01-Jun-2017 16:09:01 + If specified as an integer, the value is interpreted as the number of channels to include in build 01-Jun-2017 16:09:01 + each bin. The final bin in the spectral window may not necessarily contain this number of build 01-Jun-2017 16:09:01 + channels. For example, if a spectral window has 15 channels, and chanbin is specified to be 6, build 01-Jun-2017 16:09:01 + then channels 0-5 will comprise the first bin, channels 6-11 the second, and channels 12-14 the build 01-Jun-2017 16:09:01 + third, so that only three channels will comprise the final bin. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - MINIMUM REQUIRED NUMBER OF VISIBILITIES build 01-Jun-2017 16:09:01 + MINIMUM REQUIRED NUMBER OF VISIBILITIES build 01-Jun-2017 16:09:01 + build 01-Jun-2017 16:09:01 + The minsamp parameter allows the user to specify the minimum number of unflagged visibilities that build 01-Jun-2017 16:09:01 + must be present in a sample for that sample's weight to be computed. If a sample has less than build 01-Jun-2017 16:09:01 + this number of unflagged points, the associated weights of all the points in the sample are build 01-Jun-2017 16:09:01 + set to zero, and all the points in the sample are flagged. build 01-Jun-2017 16:09:01 + build 01-Jun-2017 16:09:01 + AGGREGATING DATA ACROSS BOUNDARIES build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - The minsamp parameter allows the user to specify the minimum number of unflagged visibilities that build 01-Jun-2017 16:09:01 - must be present in a sample for that sample's weight to be computed. If a sample has less than build 01-Jun-2017 16:09:01 - this number of unflagged points, the associated weights of all the points in the sample are build 01-Jun-2017 16:09:01 - set to zero, and all the points in the sample are flagged. build 01-Jun-2017 16:09:01 + By default, data are not aggregated across changes in values in the columns ARRAY_ID, build 01-Jun-2017 16:09:01 + SCAN_NUMBER, STATE_ID, FIELD_ID, and DATA_DESC_ID. One can override this behavior for build 01-Jun-2017 16:09:01 + SCAN_NUMBER, STATE_ID, and FIELD_ID by specifying the combine parameter. For example, build 01-Jun-2017 16:09:01 + specifying combine="scan" will ignore scan boundaries when aggregating data. Specifying build 01-Jun-2017 16:09:01 + combine="field, scan" will ignore both scan and field boundaries when aggregating data. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - AGGREGATING DATA ACROSS BOUNDARIES build 01-Jun-2017 16:09:01 + Also by default, data for separate correlations are aggregated separately. Data for all build 01-Jun-2017 16:09:01 + correlations within each spectral window can be aggregated together by specifying build 01-Jun-2017 16:09:01 + "corr" in the combine parameter. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - By default, data are not aggregated across changes in values in the columns ARRAY_ID, build 01-Jun-2017 16:09:01 - SCAN_NUMBER, STATE_ID, FIELD_ID, and DATA_DESC_ID. One can override this behavior for build 01-Jun-2017 16:09:01 - SCAN_NUMBER, STATE_ID, and FIELD_ID by specifying the combine parameter. For example, build 01-Jun-2017 16:09:01 - specifying combine="scan" will ignore scan boundaries when aggregating data. Specifying build 01-Jun-2017 16:09:01 - combine="field, scan" will ignore both scan and field boundaries when aggregating data. build 01-Jun-2017 16:09:01 - Any combination and permutation of "scan", "field", and "state" are supported by the build 01-Jun-2017 16:09:01 - combine parameter. build 01-Jun-2017 16:09:01 + Any combination and permutation of "scan", "field", "state", and "corr" are supported build 01-Jun-2017 16:09:01 + by the combine parameter. Other values will be silently ignored. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - OTHER CONSIDERATIONS build 01-Jun-2017 16:09:01 + OTHER CONSIDERATIONS build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - Flagged values are not used in computing the weights, although the associated weights of build 01-Jun-2017 16:09:01 - these values are updated. build 01-Jun-2017 16:09:01 + Flagged values are not used in computing the weights, although the associated weights of build 01-Jun-2017 16:09:01 + these values are updated. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 - If the variance for a set of data is 0, all associated flags for that data are set to True, build 01-Jun-2017 16:09:01 - and the corresponding weights are set to 0. build 01-Jun-2017 16:09:01 + If the variance for a set of data is 0, all associated flags for that data are set to True, build 01-Jun-2017 16:09:01 + and the corresponding weights are set to 0. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 EXAMPLE build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 diff --git a/gcwrap/tools/ms/ms.xml b/gcwrap/tools/ms/ms.xml build 01-Jun-2017 16:09:01 index d8171ed..b50029a 100644 build 01-Jun-2017 16:09:01 --- a/gcwrap/tools/ms/ms.xml build 01-Jun-2017 16:09:01 +++ b/gcwrap/tools/ms/ms.xml build 01-Jun-2017 16:09:01 @@ -1156,7 +1156,9 @@ the ms definition itself. --> build 01-Jun-2017 16:09:01 of (data unit)^(-2), eg Jy^(-2). build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 Data are aggregated on a per-baseline, per-data description ID basis. Data are aggregated build 01-Jun-2017 16:09:01 - in bins determined by the specified values of the timebin and chanbin parameters. build 01-Jun-2017 16:09:01 + in bins determined by the specified values of the timebin and chanbin parameters. By default, build 01-Jun-2017 16:09:01 + data for separate correlations are aggregated separately. This behavior can be overriden build 01-Jun-2017 16:09:01 + by specifying combine="corr" (see below). build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 TIME BINNING build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 @@ -1217,8 +1219,13 @@ the ms definition itself. --> build 01-Jun-2017 16:09:01 SCAN_NUMBER, STATE_ID, and FIELD_ID by specifying the combine parameter. For example, build 01-Jun-2017 16:09:01 specifying combine="scan" will ignore scan boundaries when aggregating data. Specifying build 01-Jun-2017 16:09:01 combine="field, scan" will ignore both scan and field boundaries when aggregating data. build 01-Jun-2017 16:09:01 - Any combination and permutation of "scan", "field", and "state" are supported by the build 01-Jun-2017 16:09:01 - combine parameter. build 01-Jun-2017 16:09:01 + build 01-Jun-2017 16:09:01 + Also by default, data for separate correlations are aggregated separately. Data for all build 01-Jun-2017 16:09:01 + correlations within each spectral window can be aggregated together by specifying build 01-Jun-2017 16:09:01 + "corr" in the combine parameter. build 01-Jun-2017 16:09:01 + build 01-Jun-2017 16:09:01 + Any combination and permutation of "scan", "field", "state", and "corr" are supported build 01-Jun-2017 16:09:01 + by the combine parameter. Other values will be silently ignored. build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 OTHER CONSIDERATIONS build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 Comparison returned: 1 build 01-Jun-2017 16:09:01 Checkout: git checkout feature/CAS-10009 error 01-Jun-2017 16:09:01 Already on 'feature/CAS-10009' build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 [git, tag, -a, 5.1.0-4-feature-CAS-10009-2, -m 'Automatic tag'] build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 Done tagging 5.1.0-4-feature-CAS-10009-2 build 01-Jun-2017 16:09:01 Pushing 5.1.0-4-feature-CAS-10009-2 to origin error 01-Jun-2017 16:09:01 To ssh://git@open-bitbucket.nrao.edu:7999/casa/casa.git error 01-Jun-2017 16:09:01 * [new tag] 5.1.0-4-feature-CAS-10009-2 -> 5.1.0-4-feature-CAS-10009-2 build 01-Jun-2017 16:09:01 build 01-Jun-2017 16:09:01 Pushed 5.1.0-4-feature-CAS-10009-2 to origin simple 01-Jun-2017 16:09:01 Finished task 'Script' with result: Success simple 01-Jun-2017 16:09:01 Running post build plugin 'Docker Container Cleanup' simple 01-Jun-2017 16:09:01 Running post build plugin 'NCover Results Collector' simple 01-Jun-2017 16:09:01 Running post build plugin 'Clover Results Collector' simple 01-Jun-2017 16:09:01 Running post build plugin 'npm Cache Cleanup' simple 01-Jun-2017 16:09:01 Running post build plugin 'Artifact Copier' simple 01-Jun-2017 16:09:01 Finalising the build... simple 01-Jun-2017 16:09:01 Stopping timer. simple 01-Jun-2017 16:09:01 Build CASA-CP90-TBFP-2 completed. simple 01-Jun-2017 16:09:01 Running on server: post build plugin 'NCover Results Collector' simple 01-Jun-2017 16:09:01 Running on server: post build plugin 'Build Hanging Detection Configuration' simple 01-Jun-2017 16:09:01 Running on server: post build plugin 'Clover Delta Calculator' simple 01-Jun-2017 16:09:01 Running on server: post build plugin 'Maven Dependencies Postprocessor' simple 01-Jun-2017 16:09:01 All post build plugins have finished simple 01-Jun-2017 16:09:01 Generating build results summary... simple 01-Jun-2017 16:09:01 Saving build results to disk... simple 01-Jun-2017 16:09:01 Logging substituted variables... simple 01-Jun-2017 16:09:02 Indexing build results... simple 01-Jun-2017 16:09:02 Finished building CASA-CP90-TBFP-2.