Fellow CNOC'ers--

With all CNOC2 data processed, I've taken a moment to summarize the
existing CNOC2 pipeline.  A copy of this file can be found at
http://www.astro.utoronto.ca/~hall/cnoc/pipeline/README


Existing CNOC2 pipeline
-----------------------

1) basic data reduction
cnocobspkg -- Greg Wirth's IRAF package for flatfielding etc. at the telescope
http://www.astro.utoronto.ca/~hall/cnocobspkg/	(contact PBH for a tar file)

2) creation of axf files (using PPP and possibly software from Huan Lin)
http://www.astro.utoronto.ca/~hall/cnoc/pipeline/howto_axf
Copies of all files mentioned here can be found at
http://www.astro.utoronto.ca/~hall/cnoc/pipeline/

3) de-cosmic-raying and extraction
cnoc97b -- Greg Wirth's IRAF package for decosmicraying & spectral reductions
NOTE that this package will probably be renamed ("cnoc99") for future work
to avoid any possible confusion; it may also be a good idea to rename this 
package if you install it and plan to make changes of your own to it.
http://www.astro.utoronto.ca/~hall/cnoc97b/	(contact PBH for a tar file)
http://www.astro.utoronto.ca/~hall/cnoc97b/de-CR.html
http://www.astro.utoronto.ca/~hall/cnoc97b/extract.html
http://www.astro.utoronto.ca/~hall/cnoc/pipeline/howto_xcor

The only notable change I have made to this package is the creation of
a *_nointerp.ms.fits output file.  Previously, the output included a final
sky-interpolated spectrum and an unweighted, non-interpolated spectrum.
In cases where sky interpolation cut out emission lines, it was necessary
to cross-correlate on the unweighted spectrum.  Obviously it is better
to use the *_nointerp.ms.fits spectrum instead, which is the same as
*_final.ms.fits except it lacks any sky, 0 order, or overlap interpolation.

4) cross-correlation (using software from Erica Ellingson & Huan Lin)
http://www.astro.utoronto.ca/~hall/cnoc/pipeline/howto_xcor
Copies of all files mentioned here can be found at
http://www.astro.utoronto.ca/~hall/cnoc/pipeline/
Of general interest may be Huan's xcdisp5.cl program, which displays
the coadded de-cr'd spectrum, the two original spectra, and postage
stamps of the object in all 5 filters, plus another postage stamp 
with the slit overlaid and all objects in the axf file marked.


Future suggestions
------------------

I think I've helped with every stage of the pipeline except mask creation.
I've found it to be pretty well optimized, but one possible area where there's
room for improvement concerns the visual inspection of the 2-D spectral image.
Currently this occurs:
   --during creation of the axf file
   --during extraction
   --during cross-correlation
This is obviously time-consuming.  In general, I think that inspection 
during extraction could be curtailed.  Currently we note emission lines,
interpolated regions, etc. during extraction.  However, the direct and
spectral images are inspected simultaneously to look for these same things
during cross-correlation using Huan's xcdisp5 program.  I find this to be
a more useful inspection, which could supplant much of the inspection
during extraction with very little impact.

Another possible way to cut down on the time involved
is for the same person to do all 3 steps for a given field -- it becomes
familiar and the inspection quicker.  But obviously this can introduce 
field-to-field variations arising from person-to-person variations.
Perhaps the same person could do just the last two steps above.

I can think of two other ways to improve:

A) For survey purposes, you can cut way down on the inspection during 
   axf file creation and just trust the automated process.  This will
   throw away some information on close pairs and serendipitous objects,
   but it's the obvious choice for projects where the ratio of 
   objects to available person-hours is high.

B) To squeeze the maximum return out of the data, inspection during axf
   file creation is arguably the most important inspection stage.  
   Currently, potentially interesting serendipitous objects (e.g single
   emission line objects) missed here but noticed later can be recovered
   only with time-consuming re-extractions.  Perhaps xcdisp5 could be
   adapted to run at this stage to improve the inspection.

Personally, I would suggest a more careful inspection during axf creation,
minimal inspection during extraction, and inspection using xcdisp5 during
cross-correlation just as it's currently done.  

Comments welcome,

Pat Hall
hall@astro.utoronto.ca