The RTC can save a wide variety of AO telemetry types. There are several different ways to command the RTC to get this data.
If you want just a single measurement type, there are several functions in the ultimate/ directory. These all take an average over a short period and return one data type. In particular use:
phase = ult_get_phase() centroids = ult_get_cents() intensity = ult_get_intensity() references = ult_get_refs() rot_and_focus = ult_get_rotfoc()
The master function here is rtc_retrieve_dumpdata. This function takes a large number of arguments for the data types. It lets you specify several options for the data as a functionin time, including and average of all measurements, only the last measurement, data at the full frame rate, and data decimated from the full frame rate. See the documentation at the link above for all the types.
res = rtc_retrieve_dumpdata(centroids=centroids, /grablast) ;; most recent measurement res = rtc_retrieve_dumpdata(centroids=centroids, /fullbuffer, interval=3) ;; 3 seconds, all measurements
The function ult_telem is the top-level code that gets all the data that you want. Call this simply as
This does several things
Every telem set is identified in file name with a “When” tag that marks the time is was taken. THis format is _When_YYYY.MM.DD_HH.MM.SS, where any fields with a single digit are not padded with a leading 0, e.g. _When_2015.130_23.9.21.
To have access to the data, you several options
To generically load in the telemetry files, we only recommend using the function load_telem as
The gpilib code is set up to assume that the AO telemetry is in a specific location. This does not necessarily match with what is on Dropbox. Pay attention below to the directories and locations!
AO telemetry on the Gemini VMs used to be stored in
This was actually a link to a separate file system. All standard AO telemetry files (i.e. what you get with ult_telem) are now stored in
AO telemetry used to be stored in
All standard AO telemetry files (i.e. what you get with ult_telem) are now stored in
All Reduced data products will be stored in
If you are using the manual sync option (not Dropbox), Lisa has a perl script that will reorganize all your data. Email her to get a copy.
AO telemetry used to be stored in
The new folders are
If you are using dropbox to sync your data it’s your responsibilty to make sure that the synced folders map correctly, that is that Dropbox/GPI_Telemetry/Raw/ is matched to your local gpilib/aotelem/Raw/.
All access to the file paths for loading and saving AO telemetry is now handled by a new function ultimate/ult_get_telem_path.pro. This function takes a _when_2015. tag and returns the path to either the raw or reduced data. If you are saving data (either when taking fresh telemetry at Gemini or for a reduced data product) this function will automatically create the YYYYMMDD directory for you if needed.
To get the path to load in a Raw file use the following. This returns everything except the suffix, e.g. phase.fits since that varies.
path = ult_get_telem_path('_When_2015.11.6_4.43.46')
To get the path to load in a Reduced file use the following.
path = ult_get_telem_path('_When_2015.11.6_4.43.46', /red)
To get the path to save a Reduced file, creating the directory if necessary, do
path = ult_get_telem_path('_When_2015.11.6_4.43.46', /red, /query)
The When tag is supplemented in AORAW _info files with the WHENSTR, which is of the format YYYYMMDDHHMMSS. This does have zero-padding, e.g. 20150130230921 This keyword is either in the header already or autogenerated by the DB ingester.
Webthingie’s DB has five new tables.
info on each telem “set” overall (i.e. _info)
stores all the files that comprise the sets (i.e. _phase)
info on every unique reduced product (e.g. aored_When_...lapeb.fits)
many to many mapping of AORAW to AORED
many to many mapping of AORAW to RawDataProducts
These can be queried directly by table name, e.g. SELECT whenstr from AORAW.
Let’s say that you want to get all the names of telem files from a specific When tag. There are two ways to do this.
First, you can just query on the datafile name with the tag, e.g.
SELECT datafile from AORAW_FILES where datafile like "%_When_2015.1.30_23.9.21%"
Second, you can use the linking UID/RAWID combo with a join.
SELECT datafile from AORAW_FILES left join AORAW on AORAW.UID = AORAW_FILES.RAWID where whenstr like "20150130230921"
To get a list of all AORAW sets, sorted by date, do
SELECT AORAW_FILES.datafile from AORAW left join AORAW_FILES on AORAW.UID = AORAW_FILES.RAWID where datafile like "%_phase" order by WHENSTR
If you want to get a list of all of the telem sets have not yet been analyzed by the error budget code, filter on there not being an entry in the AORAW2RED table. (This query style will break as new reduced types are added into the table. LAP needs to improve the logic). You need to filter on the object name. (THe object name will be null or zenith when we are not observing a star. Otherwise it is the target name.) You can optionally add filter for a certain date range by using whenstr.
SELECT AORAW_FILES.datafile, objname from AORAW left join AORAW_FILES on AORAW.UID = AORAW_FILES.RAWID where AORAW.UID not in (SELECT RAWID from AORAW2RED) and objname not like 'Zenith' and datafile like "%_phase" and cast(WHENSTR as unsigned) > 20150129120000 order by WHENSTR
You can explore relationships between reduced and raw AO data, and between AO data and IFS data easily with joins.
THe ingestion script automatically copies any keywords from AORAW_FILES into the linked AORAW entry, so you don’t need a join for that. However, none of those keywords are stored for the AORAW_FILES. So to get the SAMPLES keyword for a specific telemety file in AORAW_FILES, do the following:
SELECT samples from AORAW left join AORAW_FILES on AORAW.UID = AORAW_FILES.RAWID where datafile like "%_When_2015.1.30_23.9.21_phase"
The AORAW and AORED are linked when the AORED is ingested by examining specific keywords in the header of the AORED file. To get information from the AORAW file that’s relevant to the AORED file, use the linker table. For ezample, to retrieve a listing of all lap_eb AO reduced with a list of star names (OBJNAME), do the following:
SELECT AORED.datafile, objname from AORAW inner join AORAW2RED on AORAW.UID = AORAW2RED.RAWID inner join AORED on AORAW2RED.REDID = AORED.UID where AORED.datafile like "%lap_eb"
The AORAW and IFS images can be linked (see section below on how this is calculated). This is queried using the AO2IFS table. For example, to get a list of all lap_eb reduced AO products that have a not BAD IFS image associated with them, do the following:
SELECT AORAW.whenstr, RawDataProducts.datafile, OBSNOTES.markedbad from RawDataProducts left join OBSNOTES on RawDataProducts.UID = OBSNOTES.UID inner join AO2IFS on RawDataProducts.UID = AO2IFS.IFSID inner join AORAW on AO2IFS.AOID = AORAW.UID where OBSNOTES.markedbad is NULL
The AORED and IFS images can be linked (see section below on how this is calculated). This is queried using the AO2IFS table. For example, to get a list of all lap_eb reduced AO products that have a IFS image associated with them (which could be good or bad), do the following:
SELECT AORED.datafile, RawDataProducts.datafile from RawDataProducts inner join AO2IFS on RawDataProducts.UID = AO2IFS.IFSID inner join AORAW on AO2IFS.AOID = AORAW.UID inner join AORAW2RED on AORAW.UID = AORAW2RED.RAWID inner join AORED on AORAW2RED.REDID = AORED.UID where AORED.datafile like "%lap_eb"
To generate a list of previous performance on a specific target, using parameters from the error budget code, do
SELECT ReducedDataProducts.CONTR025, whenstr, AORED.WFE_BAND from ReducedDataProducts inner join Raw2Reduced on ReducedDataProducts.UID = Raw2Reduced.REDID inner join RawDataProducts on Raw2Reduced.RAWID = RawDataProducts.UID left join OBSNOTES on RawDataProducts.UID = OBSNOTES.UID inner join AO2IFS on RawDataProducts.UID = AO2IFS.IFSID inner join AORAW on AO2IFS.AOID = AORAW.UID inner join AORAW2RED on AORAW.UID = AORAW2RED.RAWID inner join AORED on AORAW2RED.REDID = AORED.UID left join AORAW_FILES on AORAW.UID = AORAW_FILES.RAWID where AORED.datafile like "%lap_eb" and AORAW_FILES.datafile like "%phase" and ReducedDataProducts.datafile like "%spdc_distorcorr.fits" and ReducedDataProducts.datapath like "%Spec%" and OBSNOTES.markedbad is NULL and AORAW.objname like "%c Eri%" order by whenstr
To generate the data for a plot of IFS contrast at 250 mas with AO bandwidth error, ignoring measurements that have been marked bad, do the following. Since there may be multiple reduced version, choose the most recent one from the data cruncher by the directory name
SELECT ReducedDataProducts.CONTR025, ReducedDataProducts.datafile, AORAW_FILES.datafile, AORED.WFE_BAND from ReducedDataProducts inner join Raw2Reduced on ReducedDataProducts.UID = Raw2Reduced.REDID inner join RawDataProducts on Raw2Reduced.RAWID = RawDataProducts.UID left join OBSNOTES on RawDataProducts.UID = OBSNOTES.UID inner join AO2IFS on RawDataProducts.UID = AO2IFS.IFSID inner join AORAW on AO2IFS.AOID = AORAW.UID inner join AORAW2RED on AORAW.UID = AORAW2RED.RAWID inner join AORED on AORAW2RED.REDID = AORED.UID left join AORAW_FILES on AORAW.UID = AORAW_FILES.RAWID where AORED.datafile like "%lap_eb" and AORAW_FILES.datafile like "%phase" and ReducedDataProducts.datafile like "%spdc_distorcorr.fits" and ReducedDataProducts.datapath like "%Spec%" and OBSNOTES.markedbad is NULL
An AO reduced data product from a particular piece of code will be a single FITS file. This file has the following requirements:
To facilitate linking to original AORAW data files, the header also includes a keyword for each file used as FILE_1, FILE_2, etc for each AORAW file. The value is a string which is the filename (no file paths) without the .fits suffix. E.g. the header looks like:
FILE_1 = 'ugp_When_2015.1.30_23.9.21_ttdata' /AORaw file used in this analysis
We also recommend that the header contain information on how the products were generated. Approved keywords for that are:
CODE = 'webthingie_eb.pro' /name of IDL code used to reduce the data A_UTDATE= '2015-11-19' /Analysis of data - date conducted (UT) A_UTTIME= '21:42:23.000' /Analysis of data - time conducted (UT) AUTHOR = 'Lisa Poyneer' /Person who wrote this code
There is no need to copy any keywords from the AORAW into the AORED file. The DB will take care of all of that.
Right now the error budget analysis has set the keywords in the AORED schema. If you do a new type of analysis that produces new single-element parameters, you can add in new keywords to the schema. This will require that you email Dmitry a list of the parameters, in the following format:
CTPERSUB DOUBLE COMMENT 'Counts (DN) per subaperture on WFS CCD per frame' NOLQG INT COMMENT 'if 1, not using LQG ' CGAINADJ DOUBLE COMMENT 'Multiplier to get right centroid gain (provision' S_UTDATE DATE COMMENT 'Start of telemetry - date (UT) '
Get the gpi codebase from the seti SVN repo as gpi/database:
svn checkout https://repos.seti.org/gpi/database --username username1
For your python installation with macports do selfupdate, then grab sudo port install py27-mysql and any other packages that you don’t have.
A simple query looks like
import numpy as np import datetime from dateutil import parser import gpifilesdb # in a terminal you have opened a tuinnel already with the command # ssh email@example.com -L 3306:localhost:3306 db = gpifilesdb.GPIFilesDB(server='127.0.0.1', username='username') aoraw_tag = "_When_2015.1.30_23.9.21" aoraw_datafile = "ugp"+aoraw_tag+"_info" aorawquery1 = "SELECT DATEOBS from AORAW where datafile like '%s'"%(aoraw_datafile) savedatera = db._do_query(aorawquery1) savedate = savedatera aorawquery2 = "SELECT UTSTART from AORAW where datafile like '%s'"%(aoraw_datafile) savetimera = db._do_query(aorawquery2) savetime = savetimera uttime = parser.parse(savedate + ' ' + savetime) aorawquery4 = "SELECT SAMPLES from AORAW where datafile like '%s'"%(aoraw_datafile) samplesra = db._do_query(aorawquery4) samples = samplesra
DONE [Lisa] Adapt local eb.pro process the data and save the new specified products with new headers into the correct location.
DONE [Lisa] Write stand-alone version of code that takes an AO telemetry file and queries the DB to find a match for an IFS image. FInal goal is to have a code snippet in python that will be plugged into existing routines on palan.
DONE [Lisa] write script to retrofit all the telem dumps with the _info.fits file.
DONE [Lisa] edit ult_telem to save this new _info.fits file with proper new keywords in the header!
Done [Lisa] write perl script to move al of the files to new Raw and Red directories!
DONE [Lisa] edit ult_get_headervals have a new keyword in the default header of:
WHENSTR char 14 Repackaged when string in format YYYYMMDDHHMMSS
Done [Lisa] Change ult_telem and associated codes to save raw data products to the new location.
Done [Lisa] Change load_telem and associated codes load these raw files from correct location.
When everything is working, switch over to saving everything to this location by default.