X-GEF Test/Analysis Report X-GEF-007 Dan Dewey, 7/26/95 - 8/28/95 file: ~dd/text/XGEF/X-GEF-007_SW_update.txt S/W Update: FITS data files, etc. --------------------------------- Summary ------- This memo summarizes the changes in the X-GEF s/w for grating data analysis made in the past months to accomodate data taken in FITS format. The main changes are: - "automatic" analysis - data in FITS format can be read (quickly) - a "useful" filename can be used to access the FITS data files - Summary information on the test procedure is created - analysis of multiple grating regions and illumination diagram - automatic generation of pspc_anal_driver.txt files - a "line fit" structure is used to unify the data analysis - header-only PSPC files can be created to save disk space yet allowing for meta-analysis - SMD data is analyzed - reference grating and SMD measurements are used to calibrate the test measurements - average first-order efficiency at 1 keV is estimated for each grating tested - test results from many tests can be assembled and displayed - tilt test analysis and plots In spite of these changes the basic PSPC analysis outlined in X-GEF-006 is still at the heart of the analysis. Looking towards the addition of optic measurements and SSD measurements there is still further evolution in the analysis software planned. In particular the s/w will access a database containing information on the gratings containing, e.g., their orientation or SML-measured gold thickness. "Automatic" Analysis -------------------- A relatively automated analysis is being used for analyzing data taken using test procedure HEG2,3 on tests run in July-September '95. Operator instructions for the analysis are in ~/irt/xgef_test/analysis_instructions.txt . The top-level IDL routine is ~/irt/xgef_test/heg3_test_analysis.pro . Perusing these will give a sense of how everything is put together to get efficiencies from the FITS data files. The descriptions below are "targeted" to particular new features and are intended as a reminder rather than a tutorial. FITS Data --------- A major change in X-GEF is the use of data in FITS format. This allows the analysis s/w to know a lot about the data without being told it explicitly. A key to using the FITS (binary table) event data from the PSPC is John Davis's C routines that convert the fits file (xxx.FITS) to a binary format that IDL can read in very quickly (xxx.idlb). [Routines mitrdfits.pro and mitcnvfits.pro in ~/wave/useful and John's C source.] The file ~dd/text/XGEF/XGEF_fits.txt gives the FITS format with some history and detailed comments. The basic X-GEF FITS routines are in ~/wave/fits. The parameters in the FITS header can be read in and used to setup the modeling parameters of the analysis s/w by the following: X-GEF> fits_fill_ps, 'X-GEF_FITS_file_name' Analysis Directories -------------------- Typically data is collected overnight. A test procedure is executed consisting of a series of test items generating the data. A single test tests two gratings. The data from the test is loaded into a directory with the date code of the first test item, e.g., the date the test was initiated. For example, the last F001M test is date code 950629. To analyze this data the IDL variable pspc_directory is set equal to this directory: X-GEF> pspc_directory = '950629' The variable pspc_above_dir is set to the full path above this analysis directory, e.g., (for a remotely mounted volumes) X-GEF> pspc_above_dir = '/nfs/bodicea/d2/F001M' Setting these two variable [which are saved in xrtf_common] allows the analysis routines to find the data. "Useful" filenames ------------------ One disadvantage of the FITS format is our naming convention, e.g., file: 950630083412.PSPC.FITS. In order to allow a more useful interface to the file names the procedure [~dd/wave/fits/] list_fits is run in the analysis directory and fits_common is filled with arrays of the real file name (ufn_real_file) and a "useful" file name (ufn_useful_file). For example, the file: 950712025128.PSPC.FITS is mapped to the useful file: PSPC[323]HEG#131.Al.HA2012 where the useful file now indicates the serial test count value, [321]; the name of the test procedure, HEG; the test item number, #131; the source anode, Al; and the grating in the beam, HA2012. The analysis routines can be given short versions of the useful file name. For example "PSPC#131" can be typed to indicate the above file. The procedure ~dd/wave/fits/convert_to_real_file.pro contains the conversion logic and is used by the lowest level file reading routines so that the useful file names can be used by higher level routines. To run list_fits and create a mapping for the current directory: X-GEF> list_fits The vales of various FITS keywords can also be displayed by runnning list_fits by including an array of keywords on the command line,e.g., X-GEF> list_fits, ['SRC_CURR', 'DET_HV'] For a default list of keywords use: X-GEF> list_fits, /SLIT_INFO Summary Information ------------------- The routines test_summary.pro and test_summary_plots.pro [in ~dd/wave/fits] are used to read the FITS headers of the data files and report/plot some useful information. The output from test_summary looks like: X-GEF Test Summary ------------------ Directory = 950610 (/spectra/d2/F001M) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grating A = MA1006 Grating B = MA1012 Test code = MEG Test count = [265] 1-135 Anodes used: FE C CU TI MGO W SSD time = 0.00 hours PSPC time = 8.31 hours Ave PSPC rate = 1071.9 c/s From: 21:55:10 10/06/95 To: 13:02:43 11/06/95 Operator: KATHY FLANAGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The output plots from test_summary_plots show the Temperature, Vacuum levels, ambient pressure, and source current as a function of the test item during the test. Grating Regions --------------- The test procedure takes data at 5 "horizontal" locations on the grating. Each of these regions is analyzed separatly as independent gratings. Sub- directories are created in the main directory for each region, for example, 950629 has the sub-directories: HX220Ref MA1013m4 MA1013m8 MA1013p0 MA1013p0tilt MA1013p4 MA1013p8 MA1013p8tilt MA1024m4 MA1024m8 MA1024p0 MA1024p0tilt MA1024p4 MA1024p8 MA1024p8tilt MX078Ref The (sub-)directories are named by the grating ID and then a letter/number code: "p4" means "plus 4 mm from center of grating", "m8" is "minus 8 mm from center of grating". The region illuminated on the grating for a given test item can be displayed by reading in the FITS info on the test item and then calling the routine facet_illum with the gratings orientation: X-GEF> fits_fill_ps, 'PSPC#6' X-GEF> facet_illum, 'B' (MA1013 has orientation B) The analysis to be performed in each region is controlled by an anal driver file (described in X-GEF-006.) Note that in addition to the 5 regions per grating, sub-directories are made for the reference gratings and the tilt-test analysis. Automatic anal_driver.txt generation ------------------------------------- The analysis of the PSPC data is driven by the pspc_anal_driver.txt file. A different anal driver is needed to direct analysis for each of the sub- directories. The manual creation of the anal driver files can be tedious. Because the FITS files are self-documented it is reasonable to have a procedure that will create the anal driver files automatically. The procedure ~irt/xgef_test/anal_driver_gen.pro does just this and embodies some heuristics for the anal driver parameters. Other parameters (the number and names of source lines used) can be easily changed by editing the body of anal_driver_gen.pro. The created anal drivers are kept in the ~/irt/xgef_test directory and copied to the analysis directories by the analysis procedure. Thus creation of the anal drivers only needs to be done when a new test procedure is created. Line-Fit data structure ----------------------- One addition to the analysis s/w that helps a great deal (and will be used more in future) is the definition of a line-fit measurement structure and routines to use it. Basically the key measurement that we extract from either a histogram (SSD or SMD) or from the PSPC events is the intensity, say counts per second, of a line in the image or histogram. Thus an SMD histogram can distilled down to the phrase "there are 1025 counts/sec in the Mg K line". Or a PSPC exposure of diffracted X-rays can be summarized by: "there are 500 counts per second per PSPC slot in the +1st order Mg K line, 510 counts per ... in the -1st order Mg K line, ..., etc.". The line-fit structure is defined in ~/wave/fits/lnf_structure_defn.pro: ; Include file to define the LiNe Fit structure ; Line_fit_variable = {Line_fit, File:'', Eexact:0.0, Order:0, $ Slots:[0,0,0,0,0,0], $ Emeas:0.0, FWHM:0.0, Cps:0.0, CpsErr:0.0, Cont:0.0, Chi2:0.0, Time:0.0} ; where: ; File is the filename of the original data ; Eexact is the known line energy ; Order is the diffracted order (..., -2, -1, 0, 1, 2, ... and ; 99 = direct beam) ; Slots is PSPC slot/anodes used ; Emeas is the fitted energy centroid ; FWHM is the fitted FWHM (2.35 * gaussian sigma) ; Cps is the counts per second per slot in the line ; CpsErr is the fit returned error on Cps ; Cont is the continuum scale factor ; Chi2 is the reduced chi^2 of the fit ; Time is the livetime of the detector This definition can be used for PSPC, SMD, and SSD line-fits measurements. Line fit data is written to files in ASCII. The function lnf_from_file('filename') returns an array of line-fit values from a file in line-fit format. Creating line-fit results is essentially what is done by pspc_analyze as documented in X-GEF-006. Given the pspc_anal_driver.txt analysis input file and the pspc_results.txt output file there is all the information to create PSPC line-fit data items. The routine lnf_from_file will also read in "pspc_results.txt" into a line-fit array and will use the pspc_anal_driver.txt file in the same directory to fill the line-fit array. The new procedure smd_analyze creates a file smd_results.txt that can be read into a line-fit structure, also by lnf_from_file. While the line-fit structure is designed for fitting results, it is often desirable to operate on the measured intensities to create ratios etc. Routines lnf_mult and lnf_div allow to line-fit values to be multiplied or divided and errors propogated accordingly. This use of the line-fit structure is capitalized on in the Reference Grating Analysis carried out by lnf_mk_effics, described below. The line-fit routines are generally in ~dd/wave/fits and start with the lnf_ prefix. Header-only PSPC files ---------------------- Once the PSPC FITS files have been analyzed the pspc_results.txt files contain the essential information from the FITS files. To keep the essence of the data available on-line and yet reduce the disk space, a routine is used to create data-less PSPC fits files by not including the binary data in the output FITS file. The command X-GEF> write_fits_header_only, /ALL_PSPC_FILES will make copies of all PSPC FITS files, prefixing the data-less copies name with an "h" (for header only). SMD analysis ------------ The intensity of the line(s), counts/sec., in the SMD exposures in the directory (pspc_directory) are calculated and placed in a data file (smd_results.txt) by the procedure smd_analyze. This procedure repeatedly calls sxd_line_fit which does the fitting. These SMD line fits can then be used for normalizing PSPC and SSD exposures. The procedure lnf_from_file will read smd_results.txt into a line-fit structure for easy access by the software. SMD files can be examined interactively by executing sxd_line_fit for a given SMD file on the command line: X-GEF> sxd_line_fit, 'SMD#100' The SMD data can be compared to the predicted model with the command: X-GEF> sxd_compare, 'SMD#100' [The "x" in sxd_line_fit and sxd_compare indicates the routines will work on either SMD or SSD data files.] Reference Grating analysis -------------------------- Reference grating analysis allows the measurements of the grating under test and the reference grating to be combined with the known reference grating efficiency to assign an efficiency to the grating under test that is relatively insensitive to systematic errors in the test setup (e.g., presence of contaminating lines, calibration errors, poor fitting techniques.) The key equation for using the reference gratings is: I_PSPC_GUT(E,m) I_SMD_Ref(E) Effic_GUT(E,m) = --------------- --------------- Effic_Ref(E,m) I_SMD_GUT(E) I_PSPC_Ref(E,m) where I_PSPC is the measured diffracted-beam rate, I_SMD is the corresponding SMD count rate, Ref indicates reference grating, GUT indicates grating under test, and Effic_Ref is the known reference efficiency at energy E and order m. The calculation is carried out by the routine lnf_mk_effics.pro. All of the data are available in line-fit format including the reference grating known efficiencies (~irt/xgef_test/HX220_Effics.lnf and 'MX078_Effics.lnf). The resulting efficiencies for the grating under test are written to the line-fit format file pspc_effics.lnf. As in the past, the routine pspc_mk_ratios will read in the efficiencies and create an output file of order ratios to be used by a grating model fitting code. Some heuristics are built into pspc_mk_ratios to select measurements that are valid to be fit. For example, the Zero-order measurements, because they are not dispersed, are generally beyond hope even with reference grating analysis as so are not generally fit (though they are plotted for comparison and often agree well with the model.) The output of pspc_mk_ratios is pspc_effics.ratios . The routine pspc_fit_model is used to find the best-fit rectangular bar grating model to the measured data as a sanity check on the data. Grating Specification Efficiencies ---------------------------------- For CEI spec verification purposes it is useful to estimate the combined first-order efficiency at 1 keV for MEG gratings and at 8 keV for HEGs. The routine pspc_fit_model can be run with the option /SPEC_FIT to cause only limited data to be fit by the model. In the case of MEGs the 1st order measurements at 0.93 keV (Cu L) and 1.254 keV (Mg K) are used. The resulting best-fit parameters can then be used to evaluate the model at 1 keV producing an effective interpolation of the data. For HEGs the 4.511 keV (Ti Ka) and 6.40 keV (Fe Ka) lines are extrapolated by the same technique to 8 keV. Test Results assembly --------------------- The analysis process of X-GEF-006 ends with best-fit model parameters assigned to the grating region being tested. The current analysis has extended this to five regions on each of the two gratings being tested. In a first atempt to go one step further in 'meta-analysis' the function test_dirs.pro returns a list of directories (all assumed in the same "pspc_above_directory") and several procedures have been written that then operate on all grating regions in all the directories. For example, creating plots of efficiency or model parameters over a set of gratings. The routines are in a very exploratory stage and so the general ideas are more relevant than the specific implementations. Some examples (all in ~/irt/xgef_test) are: - batch_process_newfits.pro Re-fit the efficiency data to a model starting with previous model values. - batch_first_orders.pro Genreate list of efficiencies at MEG eneries. Interpolate 1 keV efficiency and put it in files. - tstr_fill_common.pro Put information from all gratings into common variables. - tstr_plots.pro Create plots over all gratings/regions from data in common. - tstr_scatter.pro Make a scatter plot of the grating model thickness and line-width for all samples. Tilt Tests ---------- Analysis of tilt test data is the same as other PSPC analysis: a pspc_anal_driver file is created and directs the extraction of line-order fits to the data. The results are in pspc_results.txt . The procedure tilt_analysis.pro is used to read-in in line-fit format the tilt results and then go to the corresponding FITS headers to get the tilt angle corresponding to each measurement. The tilt assymetry is also corrected by the reference grating measurement (which should have no assymetry.) Plots of 1st order efficiency vs. angle and assymetry vs angle are produced. This routine should be modified in future to output a file of assymetry vs. tilt angle for modeling input. Next steps ---------- - PSPC line-fit analysis without pspc_anal_driver files The present anal driver scheme groups the PSPC data files before they are analyzed for their line-fit contents. It is also a hold-over from pre-FITS analysis. Idealy, each PSPC file could be analyzed in and of itself and the results saved. Grouping of the files would then happen after analysis. This will speed analysis (as some files are common to each grating region - the direct beam files and are multiply analyzed) and allow more flexible assembly of data into grating regions. - Database of grating parameters and analysis results The analysis of each file should use the know fabrication parameters of the gratings (polyimide thickness, bar parameters, orientation) as the modeling starting point. This will require that this data be available to the IDL routines. Probably the NE80 spreadsheet will be the source of this data to minimize duplications. Analysis results will need to be stored and organized too. - Analyze each location-slot region independantly An X-GEF test produces measurements in ~ 20 regions over the gratings active area. This is sufficient to allow crude contour plots of parameter variation, etc. Where is the S/W ---------------- The analysis s/w is spread throughout several directories. I use the directory ~dd/irt/develop to run the software and develop new routines without interfering with others who might be using the software. Consequently the file wave_start.pro in this directory sets IDL paths to all the s/w needed and can be used as a guide to the s/w locations. Currently the following additions (beyond the IDL libraries) to the path are used (note that the whole lib_astro is not needed): !path='~dd/irt/cash:'+!path !path='~dd/wave/useful:'+!path !path='~dd/irt/useful:'+!path !path='~dd/wave/labxray:'+!path !path='~dd/irt/xrtf:'+!path !path='~dd/wave/fits:'+!path !path='~dd/irt/xgef_test:'+!path !path=!path+':/apr/idl3.6/lib_astro/pro/fits' !path=!path+':/apr/idl3.6/lib_astro/pro/fits_bintable' !path=!path+':/apr/idl3.6/lib_astro/pro/database' !path=!path+':/apr/idl3.6/lib_astro/pro/misc' !path=!path+':/apr/idl3.6/lib_astro/pro/sdas' and these startup routines are executed: .run dave_fit .rnew irt .run fits_start A snap-shot of the s/w as of 7/19/95 was made and copied along with the F001M results to DAT tape and to /bodicea/d2.