Build log

The build generated 3,834 lines of output.The output is too long and has been truncated to the last 1,000 lines. Download or view full build log.

17-Jul-2019 16:08:08	+ @param output: This is the dictionary that the key-value pairs will be appended to
17-Jul-2019 16:08:08	+ @param filename: This is the name of the test script file
17-Jul-2019 16:08:08	+ @param dataset: This is the name of the dataset used when executing this test case
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ @return: Nothing is returned, the output dict is modified by this function
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ import inspect
17-Jul-2019 16:08:08	+ frame = inspect.stack()[1]
17-Jul-2019 16:08:08	+ module = inspect.getmodule(frame[0])
17-Jul-2019 16:08:08	+ filename = module.__file__
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ testcase = unittest.TestCase.id(self)
17-Jul-2019 16:08:08	+ test_split = testcase.split('.')
17-Jul-2019 16:08:08	+ test_case = test_split[-1]
17-Jul-2019 16:08:08	+ taskname = test_split[1].split('_')[0]
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if (sys.version_info > (3, 3)):
17-Jul-2019 16:08:08	+ rerun = "python {} {}.{}".format(filename, test_split[1], test_split[2])
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ filename = "{}.py".format(filename.split('.')[0])
17-Jul-2019 16:08:08	+ casapath = os.environ.get('CASAPATH').split()[0]
17-Jul-2019 16:08:08	+ rerun = "casa -c {}/lib/python2.7/runUnitTest.py {}".format(casapath, filename.split('.')[0])
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ current_case = None
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ func_calls = []
17-Jul-2019 16:08:08	+ values = {key:kwargs[key] for key in kwargs}
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ with open(filename, 'r') as file:
17-Jul-2019 16:08:08	+ for line in file:
17-Jul-2019 16:08:08	+ line = line.strip()
17-Jul-2019 16:08:08	+ if line.startswith('def test_'):
17-Jul-2019 16:08:08	+ if line.split()[1][:-7].endswith(test_case):
17-Jul-2019 16:08:08	+ current_case = test_case
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ current_case = None
17-Jul-2019 16:08:08	+ #casa6tasks, miscellaneous_tasks
17-Jul-2019 16:08:08	+ for i in casa6tasks.union(miscellaneous_tasks):
17-Jul-2019 16:08:08	+ if current_case == test_case:
17-Jul-2019 16:08:08	+ if "{}(".format(i) in line:
17-Jul-2019 16:08:08	+ params = line.split(',')[1::]
17-Jul-2019 16:08:08	+ call = "{}({},{})".format(taskname, dataset, ','.join(params))
17-Jul-2019 16:08:08	+ #func_calls.append(call)
17-Jul-2019 16:08:08	+ func_calls.append(line)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ values['runtime'] = -1.0
17-Jul-2019 16:08:08	+ #This is a temp error value
17-Jul-2019 16:08:08	+ values['status'] = status
17-Jul-2019 16:08:08	+ if test_case not in output.keys():
17-Jul-2019 16:08:08	+ output[test_case]= {}
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ for key in values.keys():
17-Jul-2019 16:08:08	+ if test_case in output.keys():
17-Jul-2019 16:08:08	+ print(output[test_case].keys())
17-Jul-2019 16:08:08	+ if key in output[test_case].keys():
17-Jul-2019 16:08:08	+ values[key] = output[test_case][key].append(values[key])
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ print('FIRST')
17-Jul-2019 16:08:08	+ output[test_case][key] = [values[key]]
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ #output[test_case] = values
17-Jul-2019 16:08:08	+ output[test_case]['taskcall'] = func_calls
17-Jul-2019 16:08:08	+ output[test_case]['rerun'] = rerun
17-Jul-2019 16:08:08	+ output[test_case]['description'] = unittest.TestCase.shortDescription(self)
17-Jul-2019 16:08:08	+ output[test_case]['images'] = [ ]
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def topickle(input_dict, picklefile):
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ Add a new dictionary into the existing pickle file
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ @param input_dict: The dictionary object to add to the pickle file
17-Jul-2019 16:08:08	+ @param picklefile: The picklefile containing a dictionary to be appended to
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ @return: Nothing is returned by this function
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ pickle_read = open(picklefile, 'rb')
17-Jul-2019 16:08:08	+ pickle_dict = pickle.load(pickle_read)
17-Jul-2019 16:08:08	+ # Make sure that the pickle file contains a dictionary
17-Jul-2019 16:08:08	+ if type(pickle_dict) != type({}):
17-Jul-2019 16:08:08	+ logging.warning('The pickle file is not a dictionary')
17-Jul-2019 16:08:08	+ # Add to the dictionary in the pickle file
17-Jul-2019 16:08:08	+ for item in list(input_dict.keys()):
17-Jul-2019 16:08:08	+ pickle_dict[item] = input_dict[item]
17-Jul-2019 16:08:08	+ # Re-write the pickle file with the new dictionary
17-Jul-2019 16:08:08	+ with open(picklefile, 'wb') as fout:
17-Jul-2019 16:08:08	+ pickle.dump(pickle_dict, fout)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def default_CASA_tasks():
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ default_CASA_tasks - Default Casa Tasks
17-Jul-2019 16:08:08	+ Delete all *.last files and restore tasks to default
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ Returns
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ logging.debug("Executing: default_CASA_tasks")
17-Jul-2019 16:08:08	+ # Get a list of all files in directory
17-Jul-2019 16:08:08	+ for rootDir, subdirs, filenames in os.walk(os.getcwd()):
17-Jul-2019 16:08:08	+ # Find the files that matches the given patterm
17-Jul-2019 16:08:08	+ for filename in fnmatch.filter(filenames, '*.last'):
17-Jul-2019 16:08:08	+ try:
17-Jul-2019 16:08:08	+ os.remove(os.path.join(rootDir, filename))
17-Jul-2019 16:08:08	+ except OSError:
17-Jul-2019 16:08:08	+ logging.error("Error while deleting file")
17-Jul-2019 16:08:08	+ if casa5:
17-Jul-2019 16:08:08	+ for task in casa6tasks:
17-Jul-2019 16:08:08	+ logging.debug("Defaulting Task: {}".format(task))
17-Jul-2019 16:08:08	+ default(task)
17-Jul-2019 16:08:08	+ for task in miscellaneous_tasks:
17-Jul-2019 16:08:08	+ logging.debug("Defaulting Task: {}".format(task))
17-Jul-2019 16:08:08	+ default(task)
17-Jul-2019 16:08:08	+ return
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def get_directory_size(directory):
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ get_directory_size - Return the size of a directory in bytes
17-Jul-2019 16:08:08	+ directory --> the directory which is to be summed
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ Returns Return the size, in bytes, of directory
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ logging.debug("Executing: get_directory_size(directory = {})".format(directory))
17-Jul-2019 16:08:08	+ total_size = 0
17-Jul-2019 16:08:08	+ for dirpath, dirnames, filenames in os.walk(directory):
17-Jul-2019 16:08:08	+ for filename in filenames:
17-Jul-2019 16:08:08	+ fp = os.path.join(dirpath, filename)
17-Jul-2019 16:08:08	+ total_size += os.path.getsize(fp)
17-Jul-2019 16:08:08	+ logging.debug("Directory: {}, Size: {} Bytes ( {}MB, {}GB) ".format(directory, total_size, (total_size/(1024.02)), float(total_size/(1024.03))))
17-Jul-2019 16:08:08	+ return total_size
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def get_table_column(table, colname):
17-Jul-2019 16:08:08	+ '''Return the requested variable column
17-Jul-2019 16:08:08	+ table --> name of table or MS
17-Jul-2019 16:08:08	+ colname --> column name
17-Jul-2019 16:08:08	+ Return the column as a dictionary
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ col = {}
17-Jul-2019 16:08:08	+ tb.open(table)
17-Jul-2019 16:08:08	+ if tb.isvarcol(colname):
17-Jul-2019 16:08:08	+ col = tb.getvarcol(colname)
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ logging.error("Error Returning Column {}".format(colname))
17-Jul-2019 16:08:08	+ return None
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ tb.close()
17-Jul-2019 16:08:08	+ return col
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def get_caltable_column(caltable, colname='CPARAM'):
17-Jul-2019 16:08:08	+ ''' Open a caltable and get the provided column
17-Jul-2019 16:08:08	+ caltable --> name of cal table
17-Jul-2019 16:08:08	+ colname --> column name
17-Jul-2019 16:08:08	+ Return the column as a dictionary
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ tb.open(caltable)
17-Jul-2019 16:08:08	+ outtable = tb.getcol(colname)
17-Jul-2019 16:08:08	+ tb.close()
17-Jul-2019 16:08:08	+ return outtable
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def get_column_shape(tab,col,start_row=0,nrow=1,row_inc=1):
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ Get the shape of the given column.
17-Jul-2019 16:08:08	+ Keyword arguments:
17-Jul-2019 16:08:08	+ tab -- input table or MS
17-Jul-2019 16:08:08	+ col -- column to get the shape
17-Jul-2019 16:08:08	+ start_row -- start row (default 0)
17-Jul-2019 16:08:08	+ nrow -- number of rows to read (default 1)
17-Jul-2019 16:08:08	+ row_inc -- increment of rows to read (default 1)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ Return a list of strings with the shape of each row in the column.
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ col_shape = []
17-Jul-2019 16:08:08	+ try:
17-Jul-2019 16:08:08	+ try:
17-Jul-2019 16:08:08	+ tb.open(tab)
17-Jul-2019 16:08:08	+ col_shape = tb.getcolshapestring(col,start_row,nrow,row_inc)
17-Jul-2019 16:08:08	+ except:
17-Jul-2019 16:08:08	+ print('Cannot get shape of col %s from table %s '%(col,tab))
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ finally:
17-Jul-2019 16:08:08	+ tb.close()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ return col_shape
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_plotfile(plotfileName, min_size, max_size=None):
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ Check if plotfile generated is cprrect size
17-Jul-2019 16:08:08	+ plotfileName --> Name of plotted Image
17-Jul-2019 16:08:08	+ min_size -- > Min Size of image
17-Jul-2019 16:08:08	+ max_size --> Max Size of image
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ Return : True if image size > min_size ( and < max_size if max_size is provided )
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ val = False
17-Jul-2019 16:08:08	+ if os.path.isfile(plotfileName):
17-Jul-2019 16:08:08	+ plotSize = os.path.getsize(plotfileName) # Return the size, in bytes, of path.
17-Jul-2019 16:08:08	+ logging.info( '{} file size is: {}'.format( plotfileName, plotSize))
17-Jul-2019 16:08:08	+ if plotSize > min_size:
17-Jul-2019 16:08:08	+ val = True
17-Jul-2019 16:08:08	+ if max_size is not None:
17-Jul-2019 16:08:08	+ if not plotSize < max_size:
17-Jul-2019 16:08:08	+ val = False
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ logging.critical("Plot was not created")
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ return val
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def generate_weblog(task,dictionary):
17-Jul-2019 16:08:08	+ """Generate Test Summary Weblog
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ Example:
17-Jul-2019 16:08:08	+ generate_weblog("taskname", dictionary)
17-Jul-2019 16:08:08	+ """
17-Jul-2019 16:08:08	+ global html
17-Jul-2019 16:08:08	+ html = open("test_{}_weblog.html".format(task.lower()), 'w')
17-Jul-2019 16:08:08	+ Weblog(task, dictionary).generate_weblog()
17-Jul-2019 16:08:08	+ html.close()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+############################################################################################
17-Jul-2019 16:08:08	+################################## imagerhelpers ###############################
17-Jul-2019 16:08:08	+############################################################################################
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_model(msname=""):
17-Jul-2019 16:08:08	+ hasmodcol = False
17-Jul-2019 16:08:08	+ modsum=0.0
17-Jul-2019 16:08:08	+ hasvirmod = False
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ tb.open( msname )
17-Jul-2019 16:08:08	+ hasmodcol = ( (tb.colnames()).count('MODEL_DATA')>0 )
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if hasmodcol:
17-Jul-2019 16:08:08	+ model_data = tb.getcol('MODEL_DATA')
17-Jul-2019 16:08:08	+ modsum = model_data.sum()
17-Jul-2019 16:08:08	+ tb.close()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ tb.open( msname+'/SOURCE' )
17-Jul-2019 16:08:08	+ keys = tb.getkeywords()
17-Jul-2019 16:08:08	+ if len(keys)>0:
17-Jul-2019 16:08:08	+ hasvirmod=True
17-Jul-2019 16:08:08	+ tb.close()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ tb.open( msname )
17-Jul-2019 16:08:08	+ keys = tb.getkeywords()
17-Jul-2019 16:08:08	+ for key in keys:
17-Jul-2019 16:08:08	+ if key.count("model_")>0:
17-Jul-2019 16:08:08	+ hasvirmod=True
17-Jul-2019 16:08:08	+ tb.close()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ logging.info("MS Name: {}, modelcol= {}, modsum = {}, virmod = {}".format( msname, hasmodcol, modsum, hasvirmod ))
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ return hasmodcol, modsum, hasvirmod
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def get_max(imname):
17-Jul-2019 16:08:08	+ """Get Image max"""
17-Jul-2019 16:08:08	+ ia.open(imname)
17-Jul-2019 16:08:08	+ stat = ia.statistics()
17-Jul-2019 16:08:08	+ ia.close()
17-Jul-2019 16:08:08	+ return stat['max'],stat['maxpos']
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def get_pix(imname,pos):
17-Jul-2019 16:08:08	+ """Get Image val"""
17-Jul-2019 16:08:08	+ ia.open(imname)
17-Jul-2019 16:08:08	+ apos = ia.pixelvalue(pos)
17-Jul-2019 16:08:08	+ ia.close()
17-Jul-2019 16:08:08	+ if apos == {}:
17-Jul-2019 16:08:08	+ return None
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ return apos['value']['value']
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def get_pixmask(imname,pos):
17-Jul-2019 16:08:08	+ """Get Image Mask val"""
17-Jul-2019 16:08:08	+ ia.open(imname)
17-Jul-2019 16:08:08	+ apos = ia.pixelvalue(pos)
17-Jul-2019 16:08:08	+ ia.close()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if apos == {}:
17-Jul-2019 16:08:08	+ return None
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ return apos['mask']
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_beam_compare(image1, image2, op=operator.le):
17-Jul-2019 16:08:08	+ """Compare all plane of cube beam image1 operator op than image1"""
17-Jul-2019 16:08:08	+ ia.open(image1)
17-Jul-2019 16:08:08	+ nchan = ia.shape()[3]
17-Jul-2019 16:08:08	+ beam1 = numpy.zeros(nchan)
17-Jul-2019 16:08:08	+ for k in range(nchan):
17-Jul-2019 16:08:08	+ beam1[k]= ia.beamarea(k,0)['arcsec2']
17-Jul-2019 16:08:08	+ ia.close()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ ia.open(image2)
17-Jul-2019 16:08:08	+ if(nchan != ia.shape()[3]):
17-Jul-2019 16:08:08	+ return False
17-Jul-2019 16:08:08	+ beam2 = numpy.zeros(nchan)
17-Jul-2019 16:08:08	+ for k in range(nchan):
17-Jul-2019 16:08:08	+ beam2[k] = ia.beamarea(k,0)['arcsec2']
17-Jul-2019 16:08:08	+ ia.close()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ return numpy.alltrue(op(beam1, beam2))
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def exists(imname):
17-Jul-2019 16:08:08	+ """ Image exists """
17-Jul-2019 16:08:08	+ return os.path.exists(imname)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def get_peak_res(summ):
17-Jul-2019 16:08:08	+ if summ.has_key('summaryminor'):
17-Jul-2019 16:08:08	+ reslist = summ['summaryminor'][1,:]
17-Jul-2019 16:08:08	+ peakres = reslist[ len(reslist)-1 ]
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ peakres = None
17-Jul-2019 16:08:08	+ return peakres
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_peak_res(summ,correctres, epsilon=0.05):
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ peakres = get_peak_res(summ)
17-Jul-2019 16:08:08	+ out = True
17-Jul-2019 16:08:08	+ if correctres == None and peakres != None:
17-Jul-2019 16:08:08	+ out = False
17-Jul-2019 16:08:08	+ return out,peakres
17-Jul-2019 16:08:08	+ if correctres != None and peakres == None:
17-Jul-2019 16:08:08	+ out = False
17-Jul-2019 16:08:08	+ return out,peakres
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if out==True and peakres != None:
17-Jul-2019 16:08:08	+ if abs(correctres - peakres)/abs(correctres) > epsilon:
17-Jul-2019 16:08:08	+ out=False
17-Jul-2019 16:08:08	+ return out,peakres
17-Jul-2019 16:08:08	+ return out,peakres
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def get_mod_flux(summ):
17-Jul-2019 16:08:08	+ if summ.has_key('summaryminor'):
17-Jul-2019 16:08:08	+ modlist = summ['summaryminor'][2,:]
17-Jul-2019 16:08:08	+ modflux = modlist[ len(modlist)-1 ]
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ modflux = None
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ return modflux
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_mod_flux(summ,correctmod, epsilon=0.05):
17-Jul-2019 16:08:08	+ modflux = get_mod_flux(summ)
17-Jul-2019 16:08:08	+ out = True
17-Jul-2019 16:08:08	+ if correctmod == None and modflux != None:
17-Jul-2019 16:08:08	+ out = False
17-Jul-2019 16:08:08	+ return out,peakres
17-Jul-2019 16:08:08	+ if correctmod != None and modflux == None:
17-Jul-2019 16:08:08	+ out = False
17-Jul-2019 16:08:08	+ return out,peakres
17-Jul-2019 16:08:08	+ if out==True and modflux != None:
17-Jul-2019 16:08:08	+ if abs(correctmod - modflux)/abs(correctmod) > epsilon:
17-Jul-2019 16:08:08	+ out=False
17-Jul-2019 16:08:08	+ return out,peakres
17-Jul-2019 16:08:08	+ return out,modflux
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def get_iter_done(summ):
17-Jul-2019 16:08:08	+ if summ.has_key('iterdone'):
17-Jul-2019 16:08:08	+ iters = summ['iterdone']
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ iters = None
17-Jul-2019 16:08:08	+ return iters
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def verdict(boolval):
17-Jul-2019 16:08:08	+ return "Pass" if boolval else "Fail"
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_ret( summ,correctres,correctmod):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ if casa5:
17-Jul-2019 16:08:08	+ testname = inspect.stack()[1][3] # Make Sure this is correct
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ testname = "TODO"
17-Jul-2019 16:08:08	+ retres, peakres = check_peak_res(summ,correctres)
17-Jul-2019 16:08:08	+ retmod, modflux = check_mod_flux(summ,correctmod)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ pstr_peak = "[ {} ] PeakRes is {} ( {} : should be {} + )\n".format(testname, str(peakres), verdict(retres) , str(correctres))
17-Jul-2019 16:08:08	+ pstr_mod = "[ {} ] Modflux is {} ( {} : should be {} + )".format(testname, str(modflux), verdict(retmod) , str(correctmod))
17-Jul-2019 16:08:08	+ pstr = pstr_peak + pstr_mod
17-Jul-2019 16:08:08	+ logging.info(pstr)
17-Jul-2019 16:08:08	+ if retres==False or retmod==False:
17-Jul-2019 16:08:08	+ return False, pstr
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ return True, pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_val(val, correctval, valname='Value', exact=False, epsilon=0.05):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ if casa5:
17-Jul-2019 16:08:08	+ testname = inspect.stack()[2][3] # Make Sure this is correct
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ testname = "TODO"
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ out = True
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if numpy.isnan(val) or numpy.isinf(val):
17-Jul-2019 16:08:08	+ out=False
17-Jul-2019 16:08:08	+ if correctval == None and val != None:
17-Jul-2019 16:08:08	+ out = False
17-Jul-2019 16:08:08	+ if correctval != None and val == None:
17-Jul-2019 16:08:08	+ out = False
17-Jul-2019 16:08:08	+ if out==True and val != None:
17-Jul-2019 16:08:08	+ if exact==True:
17-Jul-2019 16:08:08	+ if correctval != val:
17-Jul-2019 16:08:08	+ out=False
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ if abs(correctval - val)/abs(correctval) > epsilon:
17-Jul-2019 16:08:08	+ out=False
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ pstr = "[ {} ] {} is {} ( {} : should be {} )\n".format(testname, valname, str(val), verdict(out), str(correctval) )
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ logging.info(pstr)
17-Jul-2019 16:08:08	+ return out, pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_ims(imlist,truth):
17-Jul-2019 16:08:08	+ if casa5:
17-Jul-2019 16:08:08	+ testname = inspect.stack()[2][3]
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ testname = "TODO"
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ imex=[]
17-Jul-2019 16:08:08	+ out=True
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ for imname in imlist:
17-Jul-2019 16:08:08	+ ondisk = exists(imname)
17-Jul-2019 16:08:08	+ imex.append( ondisk )
17-Jul-2019 16:08:08	+ if ondisk != truth:
17-Jul-2019 16:08:08	+ out=False
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ pstr = "[ {} ] Image made : {} = {} ( {} : should all be {} )\n".format(testname, str(imlist), str(imex), verdict(out),str(truth))
17-Jul-2019 16:08:08	+ logging.info(pstr)
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_keywords(imlist):
17-Jul-2019 16:08:08	+ """
17-Jul-2019 16:08:08	+ Keyword related checks (presence/absence of records and entries in these records,
17-Jul-2019 16:08:08	+ in the keywords of the image table).
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ :param imlist: names of the images produced by a test execution.
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ :returns: the usual (test_imager_helper) string with success/error messages.
17-Jul-2019 16:08:08	+ """
17-Jul-2019 16:08:08	+ # Keeping the general approach. This is fragile!
17-Jul-2019 16:08:08	+ if casa5:
17-Jul-2019 16:08:08	+ testname = inspect.stack()[2][3]
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ testname = "TODO"
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ # accumulator of error strings
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ for imname in imlist:
17-Jul-2019 16:08:08	+ if os.path.exists(imname):
17-Jul-2019 16:08:08	+ issues = check_im_keywords(imname, check_misc=True, check_extended=True)
17-Jul-2019 16:08:08	+ if issues:
17-Jul-2019 16:08:08	+ pstr += '[{0}] {1}: {2}'.format(testname, imname, issues)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if not pstr:
17-Jul-2019 16:08:08	+ pstr += 'All expected keywords in imageinfo, miscinfo, and coords found.\n'
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_im_keywords(imname, check_misc=True, check_extended=True):
17-Jul-2019 16:08:08	+ """
17-Jul-2019 16:08:08	+ Checks several lists of expected and forbidden keywords and entries of these
17-Jul-2019 16:08:08	+ keywords.
17-Jul-2019 16:08:08	+ Forbidden keywords lists introduced with CAS-9231 (prevent duplication of
17-Jul-2019 16:08:08	+ TELESCOP and OBJECT).
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ Note that if imname is the top level of a refconcat image, there's no table to open
17-Jul-2019 16:08:08	+ to look for its keywords. In these cases nothing is checked. We would not have the
17-Jul-2019 16:08:08	+ 'imageinfo' keywords, only the MiscInfo that goes in imageconcat.json and I'm not
17-Jul-2019 16:08:08	+ sure yet how that one is supposed to behave.
17-Jul-2019 16:08:08	+ Tests should check the 'getNParts() from imname' to make sure the components of
17-Jul-2019 16:08:08	+ the refconcat image exist, have the expected keywords, etc.
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ :param imname: image name (output image from tclean)
17-Jul-2019 16:08:08	+ :param check_misc: whether to check miscinfo in addition to imageinfo'
17-Jul-2019 16:08:08	+ :param check_extended: can leave enabled for images other than .tt?, .alpha, etc.
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ :returns: the usual (test_imager_helper) string with success/error messages.
17-Jul-2019 16:08:08	+ Errors marked with '(Fail' as per self.verdict().
17-Jul-2019 16:08:08	+ """
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ try:
17-Jul-2019 16:08:08	+ tbt.open(imname)
17-Jul-2019 16:08:08	+ keys = tbt.getkeywords()
17-Jul-2019 16:08:08	+ except RuntimeError as exc:
17-Jul-2019 16:08:08	+ if os.path.isfile(os.path.join(os.path.abspath(imname), 'imageconcat.json')):
17-Jul-2019 16:08:08	+ # Looks like a refconcat image, nothing to check
17-Jul-2019 16:08:08	+ #return ''
17-Jul-2019 16:08:08	+ # make a bit more informative
17-Jul-2019 16:08:08	+ pstr = 'Looks like it is a refconcat image. Skipping the imageinfo keywords check.'
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ pstr = 'Cannot open image table to check keywords: {0}'.format(imname)
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+ finally:
17-Jul-2019 16:08:08	+ tbt.close()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ if len(keys) <= 0:
17-Jul-2019 16:08:08	+ pstr += ('No keywords found ({0})'.format(verdict(False)))
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ # Records that need to be present
17-Jul-2019 16:08:08	+ imageinfo = 'imageinfo'
17-Jul-2019 16:08:08	+ miscinfo = 'miscinfo'
17-Jul-2019 16:08:08	+ coords = 'coords'
17-Jul-2019 16:08:08	+ mandatory_recs = [imageinfo, coords]
17-Jul-2019 16:08:08	+ if check_misc:
17-Jul-2019 16:08:08	+ mandatory_recs.append(miscinfo)
17-Jul-2019 16:08:08	+ for rec in mandatory_recs:
17-Jul-2019 16:08:08	+ if rec not in keys:
17-Jul-2019 16:08:08	+ pstr += ('{0} record not found ({1})\n'.format(rec, verdict(False)))
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if len(pstr) > 0:
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ mandatory_imageinfo = ['objectname', 'imagetype']
17-Jul-2019 16:08:08	+ pstr += check_expected_entries(mandatory_imageinfo, imageinfo, keys)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if check_misc:
17-Jul-2019 16:08:08	+ if check_extended:
17-Jul-2019 16:08:08	+ mandatory_miscinfo = ['INSTRUME', 'distance']
17-Jul-2019 16:08:08	+ pstr += check_expected_entries(mandatory_miscinfo, miscinfo, keys)
17-Jul-2019 16:08:08	+ forbidden_miscinfo = ['OBJECT', 'TELESCOP']
17-Jul-2019 16:08:08	+ pstr += check_forbidden_entries(forbidden_miscinfo, miscinfo, keys)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ mandatory_coords = ['telescope']
17-Jul-2019 16:08:08	+ pstr += check_expected_entries(mandatory_coords, coords, keys)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_expected_entries( entries, record, keys):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ for entry in entries:
17-Jul-2019 16:08:08	+ if entry not in keys[record]:
17-Jul-2019 16:08:08	+ pstr += ('entry {0} not found in record {1} ({2})\n'.format(entry, record, verdict(False)))
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ # TODO: many tests leave 'distance' empty. Assume that's acceptable...
17-Jul-2019 16:08:08	+ if entry != 'distance' and not keys[record][entry]:
17-Jul-2019 16:08:08	+ pstr += ('entry {0} is found in record {1} but it is empty ({2})\n'.format(entry, record, verdict(False)))
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_forbidden_entries( entries, record, keys):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ for entry in entries:
17-Jul-2019 16:08:08	+ if entry in keys[record]:
17-Jul-2019 16:08:08	+ pstr += ('entry {0} should not be in record {1} ({2})\n'.format(entry, record, verdict(False)))
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_pix_val(imname,theval=0, thepos=[0,0,0,0], exact=False, epsilon=0.05):
17-Jul-2019 16:08:08	+ if casa5:
17-Jul-2019 16:08:08	+ testname = inspect.stack()[2][3]
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ testname = "TODO"
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ readval = get_pix(imname,thepos)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ res=True
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if readval==None:
17-Jul-2019 16:08:08	+ res=False
17-Jul-2019 16:08:08	+ elif numpy.isnan(readval) or numpy.isinf(readval):
17-Jul-2019 16:08:08	+ res=False
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ if abs(theval) > epsilon:
17-Jul-2019 16:08:08	+ if exact==False:
17-Jul-2019 16:08:08	+ if abs(readval - theval)/abs(theval) > epsilon:
17-Jul-2019 16:08:08	+ res = False
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ res = True
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ if abs(readval - theval) > 0.0:
17-Jul-2019 16:08:08	+ res = False
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ res = True
17-Jul-2019 16:08:08	+ else: ## this is to guard against exact zero... sort of.
17-Jul-2019 16:08:08	+ if abs(readval - theval) > epsilon:
17-Jul-2019 16:08:08	+ res = False
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ res = True
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ pstr = "[ {} ] {} : Value is {} at {} ( {} : should be {} )\n".format(testname, imname, str(readval), str(thepos), verdict(res), str(theval))
17-Jul-2019 16:08:08	+ logging.info(pstr)
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_pixmask(imname,theval=True, thepos=[0,0,0,0]):
17-Jul-2019 16:08:08	+ if casa5:
17-Jul-2019 16:08:08	+ testname = inspect.stack()[2][3]
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ testname = "TODO"
17-Jul-2019 16:08:08	+ readval = get_pixmask(imname,thepos)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ res=True
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if readval==None:
17-Jul-2019 16:08:08	+ res=False
17-Jul-2019 16:08:08	+ elif numpy.isnan(readval) or numpy.isinf(readval) or type(readval)!=bool:
17-Jul-2019 16:08:08	+ res=False
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ if readval == theval:
17-Jul-2019 16:08:08	+ res = True
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ res = False
17-Jul-2019 16:08:08	+ pstr = "[ {} ] {} : Mask is {} at {} ( {} : should be {} )\n".format(testname, imname, str(readval), str(thepos), verdict(res), str(theval))
17-Jul-2019 16:08:08	+ logging.info(pstr)
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_ref_freq(imname,theval=0, epsilon=0.05):
17-Jul-2019 16:08:08	+ testname = inspect.stack()[2][3]
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ retres=True
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ ia.open(imname)
17-Jul-2019 16:08:08	+ csys = ia.coordsys()
17-Jul-2019 16:08:08	+ ia.close()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ reffreq = csys.referencevalue()['numeric'][3]
17-Jul-2019 16:08:08	+ if abs(reffreq - theval)/theval > epsilon :
17-Jul-2019 16:08:08	+ retres=False
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ retres=True
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ pstr = "[ {} ] Ref-Freq is {} ( {} : should be {} )\n".format(testname , str(reffreq) , verdict(retres), str(theval))
17-Jul-2019 16:08:08	+ logging.info(pstr)
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+###################################
17-Jul-2019 16:08:08	+def check_imexist(imexist):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ if imexist != None:
17-Jul-2019 16:08:08	+ if type(imexist)==list:
17-Jul-2019 16:08:08	+ pstr += check_ims(imexist, True)
17-Jul-2019 16:08:08	+ print "pstr after checkims=",pstr
17-Jul-2019 16:08:08	+ pstr += check_keywords(imexist)
17-Jul-2019 16:08:08	+ print "pstr after check_keywords=",pstr
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_imexistnot(imexistnot):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ if imexistnot != None:
17-Jul-2019 16:08:08	+ if type(imexistnot)==list:
17-Jul-2019 16:08:08	+ pstr += check_ims(imexistnot, False)
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_imval(imval):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ if imval != None:
17-Jul-2019 16:08:08	+ if type(imval)==list:
17-Jul-2019 16:08:08	+ for ii in imval:
17-Jul-2019 16:08:08	+ if type(ii)==tuple and len(ii)==3:
17-Jul-2019 16:08:08	+ pstr += check_pix_val(ii[0],ii[1],ii[2])
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_imvalexact(imvalexact):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ if imvalexact != None:
17-Jul-2019 16:08:08	+ if type(imvalexact)==list:
17-Jul-2019 16:08:08	+ for ii in imvalexact:
17-Jul-2019 16:08:08	+ if type(ii)==tuple and len(ii)==3:
17-Jul-2019 16:08:08	+ pstr += check_pix_val(ii[0],ii[1],ii[2], exact=True)
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_immask(immask):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ if immask != None:
17-Jul-2019 16:08:08	+ if type(immask)==list:
17-Jul-2019 16:08:08	+ for ii in immask:
17-Jul-2019 16:08:08	+ if type(ii)==tuple and len(ii)==3:
17-Jul-2019 16:08:08	+ pstr += check_pixmask(ii[0],ii[1],ii[2])
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_tabcache(tabcache):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ if tabcache==True:
17-Jul-2019 16:08:08	+ opentabs = tb.showcache()
17-Jul-2019 16:08:08	+ if len(opentabs)>0 :
17-Jul-2019 16:08:08	+ pstr += "["+inspect.stack()[1][3]+"] " + verdict(False) + ": Found open tables after run "
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_stopcode(stopcode):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ if stopcode != None:
17-Jul-2019 16:08:08	+ if type(stopcode)==int:
17-Jul-2019 16:08:08	+ stopstr = "["+inspect.stack()[1][3]+"] Stopcode is " + str(ret['stopcode']) + " (" + verdict(ret['stopcode']==stopcode) + " : should be " + str(stopcode) + ")\n"
17-Jul-2019 16:08:08	+ print stopstr
17-Jul-2019 16:08:08	+ pstr += stopstr
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def check_reffreq(reffreq):
17-Jul-2019 16:08:08	+ pstr = ''
17-Jul-2019 16:08:08	+ if reffreq != None:
17-Jul-2019 16:08:08	+ if type(reffreq)==list:
17-Jul-2019 16:08:08	+ for ii in reffreq:
17-Jul-2019 16:08:08	+ if type(ii)==tuple and len(ii)==2:
17-Jul-2019 16:08:08	+ pstr += check_ref_freq(ii[0],ii[1])
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def checkall( ret=None, peakres=None, modflux=None, iterdone=None, nmajordone=None, imexist=None, imexistnot=None, imval=None, imvalexact=None, immask=None, tabcache=True, stopcode=None, reffreq=None ):
17-Jul-2019 16:08:08	+ """
17-Jul-2019 16:08:08	+ ret=None,
17-Jul-2019 16:08:08	+ peakres=None, # a float
17-Jul-2019 16:08:08	+ modflux=None, # a float
17-Jul-2019 16:08:08	+ iterdone=None, # an int
17-Jul-2019 16:08:08	+ nmajordone=None, # an int
17-Jul-2019 16:08:08	+ imexist=None, # list of image names
17-Jul-2019 16:08:08	+ imexistnot=None, # list of image names
17-Jul-2019 16:08:08	+ imval=None, # list of tuples of (imagename,val,pos)
17-Jul-2019 16:08:08	+ imvalexact=None, # list of tuples of (imagename,val,pos)
17-Jul-2019 16:08:08	+ immask=None, #list of tuples to check mask value
17-Jul-2019 16:08:08	+ tabcache=True,
17-Jul-2019 16:08:08	+ stopcode=None,
17-Jul-2019 16:08:08	+ reffreq=None # list of tuples of (imagename, reffreq)
17-Jul-2019 16:08:08	+ """
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ pstr = ""
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if ret != None and type(ret)==dict:
17-Jul-2019 16:08:08	+ try:
17-Jul-2019 16:08:08	+ if peakres != None:
17-Jul-2019 16:08:08	+ pstr += check_val( val=get_peak_res(ret), correctval=peakres, valname="peak res" )
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if modflux != None:
17-Jul-2019 16:08:08	+ pstr += check_val( val=get_mod_flux(ret), correctval=modflux, valname="mod flux" )
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if iterdone != None:
17-Jul-2019 16:08:08	+ pstr += check_val( val=ret['iterdone'], correctval=iterdone, valname="iterdone", exact=True )
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ if nmajordone != None:
17-Jul-2019 16:08:08	+ pstr += check_val( val=ret['nmajordone'], correctval=nmajordone, valname="nmajordone", exact=True )
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ except Exception as e:
17-Jul-2019 16:08:08	+ logging.info(ret)
17-Jul-2019 16:08:08	+ raise
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ pstr += check_imexist(imexist)
17-Jul-2019 16:08:08	+ pstr += check_imexistnot(imexistnot)
17-Jul-2019 16:08:08	+ pstr += check_imval(imval)
17-Jul-2019 16:08:08	+ pstr += check_imvalexact(imvalexact)
17-Jul-2019 16:08:08	+ pstr += check_immask(immask)
17-Jul-2019 16:08:08	+ pstr += check_tabcache(tabcache)
17-Jul-2019 16:08:08	+ pstr += check_stopcode(stopcode)
17-Jul-2019 16:08:08	+ pstr += check_reffreq(reffreq)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ return pstr
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+############################################################################################
17-Jul-2019 16:08:08	+################################## Decorators ##################################
17-Jul-2019 16:08:08	+############################################################################################
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+#import casaTestHelper
17-Jul-2019 16:08:08	+#@casaTestHelper.skipIfMissingModule
17-Jul-2019 16:08:08	+def skipIfMissingModule(required_module,strict=False):
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ Decorator: skip test if specified module is not avaliable
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ Example:
17-Jul-2019 16:08:08	+ @casaTestHelper.skipIfMissingModule('astropy')
17-Jul-2019 16:08:08	+ def test_test(self):
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ import os
17-Jul-2019 16:08:08	+ try:
17-Jul-2019 16:08:08	+ __import__(required_module)
17-Jul-2019 16:08:08	+ flag = True
17-Jul-2019 16:08:08	+ except ImportError:
17-Jul-2019 16:08:08	+ flag = False
17-Jul-2019 16:08:08	+ def deco(function):
17-Jul-2019 16:08:08	+ if not CASA6:
17-Jul-2019 16:08:08	+ return deco
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ def wrapper(self, args, *kwargs):
17-Jul-2019 16:08:08	+ if not flag:
17-Jul-2019 16:08:08	+ # If there is a strict flag run the tests as normal
17-Jul-2019 16:08:08	+ print(sys.argv)
17-Jul-2019 16:08:08	+ if strict:
17-Jul-2019 16:08:08	+ function(self)
17-Jul-2019 16:08:08	+ pass
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ # Module ImportError and no strict flag
17-Jul-2019 16:08:08	+ self.skipTest("ModuleNotFoundError: No module named '{}'".format(required_module))
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ function(self)
17-Jul-2019 16:08:08	+ return wrapper
17-Jul-2019 16:08:08	+ return deco
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+#import casaTestHelper
17-Jul-2019 16:08:08	+#@casaTestHelper.time_execution
17-Jul-2019 16:08:08	+def time_execution(out_dict):
17-Jul-2019 16:08:08	+ def time_decorator(function):
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ Decorator: time execution of test
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ Example:
17-Jul-2019 16:08:08	+ @casaTestHelper.time_execution
17-Jul-2019 16:08:08	+ def test_test(self):
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ @wraps(function)
17-Jul-2019 16:08:08	+ def function_timer(args, *kwargs):
17-Jul-2019 16:08:08	+ t0 = time.time()
17-Jul-2019 16:08:08	+ result = function(args, *kwargs)
17-Jul-2019 16:08:08	+ t1 = time.time()
17-Jul-2019 16:08:08	+ #print ("Total time running %s: %s seconds" % (function.__name__, str(t1-t0)))
17-Jul-2019 16:08:08	+ casalog.post("Total time running {}: {} seconds".format(function.__name__, str(t1-t0)))
17-Jul-2019 16:08:08	+ out_dict[function.__name__]['runtime'] = t1-t0
17-Jul-2019 16:08:08	+ return result
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ return function_timer
17-Jul-2019 16:08:08	+ return time_decorator
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def time_execution_alternative(out_dict):
17-Jul-2019 16:08:08	+ # TODO Ver if this is the better option
17-Jul-2019 16:08:08	+ def time_decorator(function):
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ Decorator: time execution of test
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ Example:
17-Jul-2019 16:08:08	+ @casaTestHelper.time_execution
17-Jul-2019 16:08:08	+ def test_test(self):
17-Jul-2019 16:08:08	+ '''
17-Jul-2019 16:08:08	+ @wraps(function)
17-Jul-2019 16:08:08	+ def function_timer(args, *kwargs):
17-Jul-2019 16:08:08	+ failed = False
17-Jul-2019 16:08:08	+ result = None
17-Jul-2019 16:08:08	+ t0 = time.time()
17-Jul-2019 16:08:08	+ try:
17-Jul-2019 16:08:08	+ result = function(args, *kwargs)
17-Jul-2019 16:08:08	+ except:
17-Jul-2019 16:08:08	+ failed=True
17-Jul-2019 16:08:08	+ t1 = time.time()
17-Jul-2019 16:08:08	+ out_dict[function.__name__]['runtime'] = t1-t0
17-Jul-2019 16:08:08	+ casalog.post("Total time running {}: {} seconds".format(function.__name__, str(t1-t0)))
17-Jul-2019 16:08:08	+ #out_dict[function.__name__]['status'] = False
17-Jul-2019 16:08:08	+ raise
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ t1 = time.time()
17-Jul-2019 16:08:08	+ #print ("Total time running %s: %s seconds" % (function.__name__, str(t1-t0)))
17-Jul-2019 16:08:08	+ casalog.post("Total time running {}: {} seconds".format(function.__name__, str(t1-t0)))
17-Jul-2019 16:08:08	+ print('======================================================')
17-Jul-2019 16:08:08	+ print(function.__name__)
17-Jul-2019 16:08:08	+ out_dict[function.__name__]['runtime'] = t1-t0
17-Jul-2019 16:08:08	+ out_dict[function.__name__]['status'] = True
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ return result
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ return function_timer
17-Jul-2019 16:08:08	+ return time_decorator
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def cpu_usage(out_dict):
17-Jul-2019 16:08:08	+ def cpu_decorator(function):
17-Jul-2019 16:08:08	+ @wraps(function)
17-Jul-2019 16:08:08	+ def function_usage(args, *kwargs):
17-Jul-2019 16:08:08	+ #Temp Fix : CASA 5 Doesnt Have psutil by default
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ try:
17-Jul-2019 16:08:08	+ import psutil
17-Jul-2019 16:08:08	+ use_psutil = True
17-Jul-2019 16:08:08	+ except ImportError:
17-Jul-2019 16:08:08	+ use_psutil = False
17-Jul-2019 16:08:08	+ if use_psutil:
17-Jul-2019 16:08:08	+ process = psutil.Process(os.getpid())
17-Jul-2019 16:08:08	+ snapshot1 = process.memory_info()
17-Jul-2019 16:08:08	+ open_files1 = process.open_files()
17-Jul-2019 16:08:08	+ num_file_descriptors1 = process.num_fds()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ #print ("Function: {}, {} MBs".format(function.__name__, megs1))
17-Jul-2019 16:08:08	+ #print ("Function: {}, Open Files: {}".format(function.__name__, open_files1))
17-Jul-2019 16:08:08	+ #print ("Function: {}, num_file_descriptors: {}".format(function.__name__, num_file_descriptors1))
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ result = function(args, *kwargs)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ process = psutil.Process(os.getpid())
17-Jul-2019 16:08:08	+ snapshot2 = process.memory_info()
17-Jul-2019 16:08:08	+ open_files2 = process.open_files()
17-Jul-2019 16:08:08	+ num_file_descriptors2 = process.num_fds()
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ #print ("Function: {}, {} MBs".format(function.__name__, megs2))
17-Jul-2019 16:08:08	+ #print ("Function: {}, Open Files: {}".format(function.__name__, open_files2))
17-Jul-2019 16:08:08	+ #print ("Function: {}, num_file_descriptors: {}".format(function.__name__, num_file_descriptors2))
17-Jul-2019 16:08:08	+ #print('{:.2f} MB\n'.format(process.memory_info().rss / 1024 / 1024))
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ #print ("Total Mem Info { }: {:.2f} MB".format(function.__name__,(process.memory_info().rss) / 1024 / 1024 ))
17-Jul-2019 16:08:08	+ out_dict[function.__name__]['cpu_usage'] = { "number of file descriptors opened" : num_file_descriptors2 - num_file_descriptors1,
17-Jul-2019 16:08:08	+ "Open files" : open_files2,
17-Jul-2019 16:08:08	+ "Pre Memory Snapshot (bytes)" : snapshot1,
17-Jul-2019 16:08:08	+ "Post Memory Snapshot (bytes)" : snapshot2
17-Jul-2019 16:08:08	+ }
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ #TODO: Add methods to get mem snapshots when psutils is not available
17-Jul-2019 16:08:08	+ result = function(args, *kwargs)
17-Jul-2019 16:08:08	+ out_dict[function.__name__]['cpu_usage'] = { "number of file descriptors opened" : "Unknown",
17-Jul-2019 16:08:08	+ "Open files" : "Unknown",
17-Jul-2019 16:08:08	+ "Pre Memory Snapshot (bytes)" : "Unknown",
17-Jul-2019 16:08:08	+ "Post Memory Snapshot (bytes)" : "Unknown"
17-Jul-2019 16:08:08	+ }
17-Jul-2019 16:08:08	+ return result
17-Jul-2019 16:08:08	+ return function_usage
17-Jul-2019 16:08:08	+ return cpu_decorator
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def peakmem(out_dict):
17-Jul-2019 16:08:08	+ #TODO: https://pytracemalloc.readthedocs.io/examples.html
17-Jul-2019 16:08:08	+ ### NOTE: Only for python3.4+
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ def mem_decorator(function):
17-Jul-2019 16:08:08	+ @wraps(function)
17-Jul-2019 16:08:08	+ def function_mem(args, *kwargs):
17-Jul-2019 16:08:08	+ import sys
17-Jul-2019 16:08:08	+ if (sys.version_info > (3, 3)):
17-Jul-2019 16:08:08	+ import tracemalloc
17-Jul-2019 16:08:08	+ tracemalloc.clear_traces()
17-Jul-2019 16:08:08	+ tracemalloc.start()
17-Jul-2019 16:08:08	+ snapshot1 = tracemalloc.take_snapshot() # Snapshot of traces of memory blocks allocated by Python.
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ result = function(args, *kwargs)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ snapshot2 = tracemalloc.take_snapshot()
17-Jul-2019 16:08:08	+ peakmem = ("{} MiB".format(tracemalloc.get_traced_memory()[1] / 1024 /1024)) #Get the current size and peak size of memory blocks traced by the tracemalloc module as a tuple: (current: int, peak: int)
17-Jul-2019 16:08:08	+ tracemalloc.stop()
17-Jul-2019 16:08:08	+ top_stats = snapshot2.compare_to(snapshot1, 'lineno') # Compute the differences with an old snapshot.
17-Jul-2019 16:08:08	+ out_dict[function.__name__]['peakmem'] = peakmem
17-Jul-2019 16:08:08	+ out_dict[function.__name__]['memleaks'] = top_stats[:10] #
17-Jul-2019 16:08:08	+ else:
17-Jul-2019 16:08:08	+ result = function(args, *kwargs)
17-Jul-2019 16:08:08	+ out_dict[function.__name__]['peakmem'] = "Unknown"
17-Jul-2019 16:08:08	+ out_dict[function.__name__]['memleaks'] = "Unknown" #
17-Jul-2019 16:08:08	+ return result
17-Jul-2019 16:08:08	+ return function_mem
17-Jul-2019 16:08:08	+ return mem_decorator
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def mem_use_deco(out_dict):
17-Jul-2019 16:08:08	+ def mem_decorator(function):
17-Jul-2019 16:08:08	+ @wraps(function)
17-Jul-2019 16:08:08	+ def function_mem(args, *kwargs):
17-Jul-2019 16:08:08	+ out = subprocess.Popen(['ps','v','-p', str(os.getpid())], stdout=subprocess.PIPE).communicate()[0].split(b'\n')
17-Jul-2019 16:08:08	+ vsz_index = out[0].split().index(b'RSS')
17-Jul-2019 16:08:08	+ out_start = float(out[1].split()[vsz_index]) / 1024
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ result = function(args, *kwargs)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ out = subprocess.Popen(['ps','v','-p', str(os.getpid())], stdout=subprocess.PIPE).communicate()[0].split(b'\n')
17-Jul-2019 16:08:08	+ vsz_index = out[0].split().index(b'RSS')
17-Jul-2019 16:08:08	+ out_end = float(out[1].split()[vsz_index]) / 1024
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ out_dict[function.__name__]['Mem Use'] = "{} MiB".format(out_end-out_start)
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+ return result
17-Jul-2019 16:08:08	+ return function_mem
17-Jul-2019 16:08:08	+ return mem_decorator
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+def stats_dict(out_dict):
17-Jul-2019 16:08:08	+ def stats_decorator(function):
17-Jul-2019 16:08:08	+ @time_execution_alternative(out_dict)
17-Jul-2019 16:08:08	+ #@cpu_usage(out_dict)
17-Jul-2019 16:08:08	+ #@peakmem(out_dict)
17-Jul-2019 16:08:08	+ @mem_use_deco(out_dict)
17-Jul-2019 16:08:08	+ @wraps(function)
17-Jul-2019 16:08:08	+ def all_wrapped(args, *kwargs):
17-Jul-2019 16:08:08	+ return function(args, *kwargs)
17-Jul-2019 16:08:08	+ return all_wrapped
17-Jul-2019 16:08:08	+ return stats_decorator
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:08	+
17-Jul-2019 16:08:09	Argument values:
17-Jul-2019 16:08:09	casatasks
17-Jul-2019 16:08:09	/export/home/cbt-el7-6/bamboohome/xml-data/build-dir/CASA-CTB-JOB1/casatasks
17-Jul-2019 16:08:09	master
17-Jul-2019 16:08:09	None
17-Jul-2019 16:08:09	Last Tag: 2019.116
17-Jul-2019 16:08:09	New Tag: 2019.118
17-Jul-2019 16:08:09	Creating a new tag
17-Jul-2019 16:08:09
17-Jul-2019 16:08:09	None
17-Jul-2019 16:08:09	Push enabled. Pushing new tag upstream.
17-Jul-2019 16:08:09	To ssh://git@open-bitbucket.nrao.edu:7999/casa/casatasks.git
17-Jul-2019 16:08:09	* [new tag] 2019.118 -> 2019.118
17-Jul-2019 16:08:09
17-Jul-2019 16:08:09	None
17-Jul-2019 16:08:09	Finished task 'Create wheel' with result: Success
17-Jul-2019 16:08:09	Running post build plugin 'Docker Container Cleanup'
17-Jul-2019 16:08:09	Beginning to execute external process for build 'CASA - Casa6 Tasks Wheel Build - Build and Package RHEL6 #121 (CASA-CTB-JOB1-121)' ... running command line: /bin/docker rm -f task-wheel-container ... in: /export/home/cbt-el7-6/bamboohome/xml-data/build-dir/CASA-CTB-JOB1
17-Jul-2019 16:08:10	task-wheel-container
17-Jul-2019 16:08:10	Running post build plugin 'NCover Results Collector'
17-Jul-2019 16:08:10	Running post build plugin 'Clover Results Collector'
17-Jul-2019 16:08:10	Running post build plugin 'npm Cache Cleanup'
17-Jul-2019 16:08:10	Running post build plugin 'Artifact Copier'
17-Jul-2019 16:08:10	Publishing an artifact: EL6 wheel
17-Jul-2019 16:08:10	Finished publishing of artifact Non required shared artifact: [EL6 wheel], pattern: [*-any.whl] anchored at: [wheeldirectory/] in 104.3 ms
17-Jul-2019 16:08:10	Finalising the build...
17-Jul-2019 16:08:10	Stopping timer.
17-Jul-2019 16:08:10	Build CASA-CTB-JOB1-121 completed.
17-Jul-2019 16:08:10	Running on server: post build plugin 'NCover Results Collector'
17-Jul-2019 16:08:10	Running on server: post build plugin 'Build Hanging Detection Configuration'
17-Jul-2019 16:08:10	Running on server: post build plugin 'Clover Delta Calculator'
17-Jul-2019 16:08:10	Running on server: post build plugin 'Maven Dependencies Postprocessor'
17-Jul-2019 16:08:10	All post build plugins have finished
17-Jul-2019 16:08:10	Generating build results summary...
17-Jul-2019 16:08:10	Saving build results to disk...
17-Jul-2019 16:08:10	Logging substituted variables...
17-Jul-2019 16:08:10	Indexing build results...
17-Jul-2019 16:08:10	Finished building CASA-CTB-JOB1-121.

Build: #121 was successful

Job: Build and Package RHEL6 was successful

Stages & jobs

Default Stage

Build log

Build #121

Build: #121 was successful

Job: Build and Package RHEL6 was successful

Stages & jobs

Default Stage

Build log