gpilib Overview, Installation and Setup

gpilib is used to interact with GPI from IDL. It is intended to command GPI actions, quickly get data from various GPI subsystems, and to run automation that is not included in the TLC or Instrument Sequencer (IS). gpilib always attempts to call the highest level function available, but certain interactions occur at the assembly level, or are sent to the specific subsystem directly.

Code directories

gpilib is split into the following directories:

Dependencies

gpilib dependencies are a strict superset of the GPI pipeline dependencies, which include the IDL Astronomy User’s Library (http://idlastro.gsfc.nasa.gov/), the Coyote Library (http://www.idlcoyote.com/documents/programs.php), the IDL Curve Fitting and Function Optimization library (http://www.physics.wisc.edu/~craigm/idl/fitting.html) and multiple standalone functions from many different sources.

In addition to this, gpilib uses the SERTS Image Display Routines. These are no longer officially supported, but are available in snapshot form from multiple online repositories. This library requires the addition of several non-standard IDL environment variables, which is done by routines imagelib.pro and devicelib.pro. These routines should be included in your IDL startup file (see below).

All of these libraries have been modified to work with the current IDL version, and to remove overloaded functions and other incompatibilities. They are available in the SETI SVN hosting gpilib and the pipeline under the external branch.

Warning

It is recommended that you DO NOT download these libraries from their original sources. The versions included in the repository have been modified to work together and extensively tested against. If you already have versions of these libraries on your system, you should change your IDL path so that the external branch takes precedence.

Installation

There are two ways of using gpilib: if you are deploying the library on a new machine in order to control the instrument, then you need to make sure that you have access to the GPI computers, various binary functions and require certain environment variables to be set. However, if you are only using gpilib for offline data analysis, the process is much simpler.

In either case, the first step is to obtain the library and dependencies (typically from the SETI SVN). Once downloaded, create a directory called private immediately under the gpilib directory. This will be the location of data generated by the library (or where you place data products for analysis. Add both gpilib and the dependencies to your IDL path (see http://www.exelisvis.com/docs/Managing_IDL_Paths.html for information on the IDL path). Be sure to add all subdirectories (use ‘+’ in the path definition).

If you are using gpilib for data analysis only, you are now done. Simply start IDL and run simaoc. This will completely configure your IDL session, populate the common block (see Common Blocks) and allow you to use any of the gpilib analysis and visualization codes.

If you are going be using gpilib for GPI control, you will need to set up an idl startup script, configure the filesystem, and install the TLC binaries (see TLC Binaries).

IDL Startup Script

In order to pre-load required components for gpilib every IDL session should run a script called .idl_startup. This script is typically kept in the home directory and is identified to IDL via environment variable $IDL_STRATUP. The startup file loads two libraries: imagelib and devicelib which are required to use the imaging program exptv. It also sets the default color table to Standard Gamma II. To change to a linear greyscale table, run the command:

IDL> loadct, 0

For more information about default color tables, see http://www.exelisvis.com/docs/LoadingDefaultColorTables.html

The startup script also creates the necessary common blocks (see Common Blocks) for gpilib operation by running IDL program setup_globals and compiles the library of functions required to communicate directly via the AOC socket (see Socket or RPC).

Here is an example .idl_startup:

1
2
3
4
5
6
7
8
device, decompose=0
device, retain=2
loadct, 5
imagelib
devicelib
!edit_input = 128

setup_globals

Note

If you are using gpilib for analysis only, you do not need to create or modify your startup script. Running simaoc at the beginning of the IDL session does everything necessary to set up the environment.

The File System

gpilib requires direct access to the AOC config and data directories, and the IFS and CAL data directories. At Gemini, these are all mounted on VM cpogpi01 (see Virtual Machines).

The AOC is mounted at /gemsoft/opt/gpi/aoc. Environment variable $AOC_ROOT is set to /gemsoft/opt/gpi/aoc/gpi_root/aoc and must be equivalent to $AOC_ROOT on the AOC. This is the directory containing the data directory (which contains all data dumped by the AOC, and the AOC error logs) and the config directory (which contains the AOC parameters file parms.txt and other various configuration files).

The CAL is mounted at /gemsoft/opt/gpi/cal, corresponding to environment variable $CAL_ROOT. This contains the CAL data and config directories.

There are no crossmounts to the TLC, but $TLC_ROOT is defined locally in order to locate the TLC binaries (see TLC Binaries).

Finally, there are two data mounts: /gemsoft/dhs/perm, corresponding to environment variable $GPI_RAW_DATA_DIR, where IFS raw frames are written, and /gemsoft/var/data/gpidata, which hosts the IFS reduced data directory and CAL, AOC, and IDL data archives. The environment variable $GPI_REDUCED_DATA_DIR points at /gemsoft/var/data/gpidata/ifs//Reduced/.

Note

All of these environment variables ($AOC_ROOT, $CAL_ROOT, $TLC_ROOT, $GPI_RAW_DATA_DIR, and $GPI_REDUCED_DATA_DIR) must be set for gpilib to properly communicate with the instrument.

Using IDL

IDL is a mature, high-level scripting language with tons of features and a large amount of available documentation. To get started, see the official doc pages at Exelis (http://www.exelisvis.com/docs/), and for more tips and tricks see Coyote’s Guide to IDL Programming (http://www.idlcoyote.com/). In general, here are a few things to watch out for:

  • IDL indexing is 0 based (like C/C++ and unlike MATLAB).
  • As in MATLAB/Python(numpy), IDL arrays can be indexed with other arrays producing arrays of the same dimension as the indexing array.
  • IDL differentiates between three types of interpreted commands: functions, procedures and directives. Functions and procedures are user-scriptable and differ only in that functions must have explicit return values whereas procedures do not have any returns (akin to C void typing). Procedures are called as program, input1, input2,... while functions are called as res = function(input1, input2,...). Both functions and procedures support optional keyword inputs (key=value pairs or boolean /key settings) that can be added to program/function calls in any order (after all named inputs). Directives are internal IDL commands, prefaced with a period (i.e., .compile) that mostly deal with execution and runtime compilation of IDL programs and functions and take no inputs.
  • Internal IDL environment variables are prefaced with exclamation points (i.e., !PATH). The shell escape is the dollar sign ($).
  • Error handling in program/function execution differs greatly from most other scripting languages (i.e., MATLAB/Python). When errors occur (if there is not an explicitly defined error handler in the code being executed), code execution will stop within the execution scope (i.e., the interpreter frame defined by the specific code in which the error occurs). This is hugely useful in debugging (because you can interact with the exact variables defined and manipulated in this scope) but can be confusing to users because you no longer have direct access to variables defined in the main scope. At any point, you can check the scope you are operating in via IDL function scope_level (1 is the MAIN scope). You can navigate within an interpreter frame with directives .step n (step n lines in code), .skip n (skip n lines of code), .continue (continue code execution), .out (run until return of current interpreter frame). Additionally, running command retall will return you to the MAIN scope. You can create a stop condition arbitrarily in your own code by adding stop to the code (or using breakpoints).
  • As with Python, IDL routines are compiled at run-time. If you make changes to code that is already compiled, you must either run it in a new IDL session, reset the current IDL session (directive .reset) or recompile the code (directive .comp name_of_file_you_want_to_recompile).
  • IDL graphics are hugely complex, carrying both a legacy windowing system as well as a more modern one that is very similar to MATLAB/Python(MATPLOTLIB). See Coyote’s Guide for classical IDL graphics and the official docs for the new stuff.

Common Blocks

A “Common block” is IDL terminology for a set of global variables that are shared between many procedures and functions. gpilib relies heavily on the common block gpi_globals. This common block is defined by program gpilib/config/setup_globals.pro, which is executed automatically on IDL startup. In order to add or remove entries from the common block, you must edit set_globals.pro. In order to access the common block from the IDL command line, issue the command

IDL> common gpi_globals

Note

If you define a variable from the common block prior to running the common command, you will get a conflict error when attempting to get the common block: % varname is already defined with a conflicting definition. To fix this, remove any variables by running IDL> delvar, 'varname'

A second common block aocIF is used to store information about the AOC socket (variable unit) when direct socket communication is begin employed. This common block is defined in gpilib/rtc/aoc_socket_functions.pro, which is compiled at the beginning of every IDL session with a call from .idl_startup. In normal operations it should never be necessary to access this common block since:

  1. By default, we do not access the socket directly.
  2. When direct socket communication is used, all read/writes to the socket are handled by the aoc socket functions and no direct interaction with the file unit representing the socket is required.

For more information on AOC communication, see Socket or RPC.

Log Files

There is a log file of all RPC commands sent to each subsystem on the subsystem’s host computer. For the AOC and CAL, this is stored in directory $TLC_ROOT/log; on the IFS, it is currently in /home/gpi/cooldowns/daemonLogs/.

Additionally, the AOC stores its own internal logs in $AOC_ROOT/data/logs (see The File System for the equivalent local path). The last (newest) files in these directories will be the current log files (to list files in reverse time order use ls -lrt). Alternatively, you can find the current log file on any system by running ps -ef | grep -i debug and looking for the process gpUtDebugService.

To display a log in real time, use:

> tail -n 100 -f filename

Documentation

Updating This Documentation

In the documentation directory under gpilib there is a script called gen_all_doc_pages.sh. Running this script from the command line will generate all of the program listings and compile the HTML documentation.

Note

The environment variable $GPI_CODE_ROOT must be set to the parent directory of gpilib in order for this script to work. Program listings are generated by a Perl script: gen_doc_pages2.pl, located in the documentation directory.

The GPI Data Pipeline Documentation developer’s guide contains some more general information in how to use the Sphinx documentation tool used to author these pages.

Using IDL Code Blocks in Documentation

This documentation contains numerous IDL code blocks. For clarity, these include the IDL prompt and (occasionally) outputs.

You can toggle these on and off in order to cut and paste directly to an IDL session by clicking the IDL> button in the upper right-hand corner of each block, as in the example below:

IDL> an idl command
some IDL output
IDL> another idl command
some more IDL output