RIC MET-2134 FCa : Chopping/nodding

[astronomyk]

https://jira.eso.org/browse/MET-2134

"Whether separate nod positions are stored as individual FITS files or as extension units of a single FITS file remains to be determined."
How are NEAR/VISIR dealing with this currently ?

@oczoske @hugobuddel @roy - do any of you know anything about the VISIR raw data output? Additionally, have we decided on a strategy regarding how chop-nod images will be outputted?

[hugobuddel]

It would be better if we would not have kept a "remains to be determined" in the document (See #103). As stated in he next line we have "a preference for small FITS files containing minimal calibratable subsets of DITs". However, Roy announced an improved n-band scheme quite late (see #144 and #122), so I wasn't actually sure what the 'minimal calibratable subset of DITs' is, so I left the sentence in.

If I understand correctly, we need at least one nod-cycle for a minimal calibratable subset of DITs, so at least two nod-positions. But not sure what we'd need if we do multiple nod cycles. Can we process each nod cycle independently or do we have to combine them?

I'll ask Roy.

[hugobuddel]

Sent to Roy:

We got a RIC about the chopping/nodding, MET-2134, could you perhaps
help us out with this?

The RIX is quite technical, so we can answer it, but your assistance
would be appreciated.

The RIC is by Faustine Cantalloube, about DRLD Sect. 3.3 (p.25), and states:

"Whether separate nod positions are stored as individual FITS files or as extension units of a single FITS file remains to be determined."
How are NEAR/VISIR dealing with this currently ?

https://jira.eso.org/browse/MET-2134
#264

There are two points to the RIC. The first is what to store in a
single file. The next line in that section states that our preference
is to have "small FITS files containing minimal calibratable subsets
of DITs". However, it is not clear to me what would be this 'minimal
calibratable subset of DITs', can you help us with that?

I see three options:

• A single nod position. I suppose that is not enough and that we need
  both nod positions at least once.
• One nod cycle, so visiting both nod positions once. I think that should work.
• Multiple or all nod cycles, so going back and forth between the two
  nod positions multiple times.

Which of those three (or another) options would be the 'minimal
calibratable subset of DITs'?

The second point is the actual question about how NEAR/VISIR are
dealing with this. Would you perhaps know in what 'granularity'
NEAR/VISIR store their raw data? I suppose we can figure this out
ourselves though, but maybe you know from experience.

[Rumpelstil]

This is a tricky issue. METIS sensitivity will be limited by systematics in the background. The pipeline aims to reduce/remove the systematics as much as possible. The current "default" chop-nod processing is to reduce each chop-nod cycle individually, and then to combine the outputs. Experience with NACO in L-band for simple dithered ("nod") observations (no chopping) shows that one can reduce systematics by a factor of 4 (1.5 mag) by fitting a time-variable background (as is implemented in ESO's Eclipse/jitter routine). We currently have no evidence, though, that fitting a time-variable background yields any advantage when processing chop-nod sequences.

[hugobuddel]

Copying email conversations about this RIX

Email from Roy on 2023/10/31:

I would say the shortest unit to store in a single file is a nod half-cycle. I.e. containing all the chops (very likely a 3-point pattern of “chop + counter-chop” exposures, for the N-band/GeoSnap with additional dither offsets). This is in my opinion the minimum calibrate-able unit of data that makes sense. It assumes that the counter-chop technique proves to be effective for METIS, too (it worked quite well on VISIR).

I am not sure about VISIR/NEAR, see the message I just sent to Ralf. Maybe he will still reply in time.

Email from Roy to Ralf Siebenmorgen

We are discussing how to group the data, which should be the “minimum scientifically useful / calibrate-able” unit. In my opinion this is one nod half-cycle (which, assuming that the “chop and counter-chop” technique works similarly well as it did on VISIR, is the minimum data unit from which we can extract a decent background-subtracted image).

The specific question from one of the reviewers is how VISIR/NEAR handled this. Did they store each frame in a separate file or was this also somehow grouped?

I remember you worked on the data-organisation of this, making the data available in a format that is small enough in terms of Gb to be practical for the community.

Reply from Ralf Siebenmorgen

as far I can remember this was done with NEAR as you suggested:
we stored not eachindividual DIT but the averaged DIT of a chopper half cycle where we also removed NDITskip frames
at the beginning so that we did not get spoiled by the M2 movements (=our chopper).

I am unsure if NDITskip > 0 is required for the METIS chopper.

Reply from Roy

However, my suggestion was to store NOD half-cycles, i.e. containing a number of chop cycles. A single chop half-cycle can in principle not result in a “minimum scientifically useful” unit of data, because it will have (potentially strong) chopping residuals. (moreover, if we do the chop + counter-chop scheme, a “half cycle” is not quite the right nomenclature, because a chop cycle will contain 4 positions, where the 1st and 3rd are the same, i.e. something like chop position 0,+1,0,-1, repeat. Where 0 is the central position and +/-1 are the respective offset positions).

[hugobuddel]

Answered with

We are in contact with the NEAR/VISIR team to compare our approaches in storing the raw data.

Our current baseline is to store a nod half-cycle in a single file. That is, containing all the chops (very likely a 3-point pattern of “chop + counter-chop” exposures, for the N-band/GeoSnap with additional dither offsets). We anticipate this to be the minimum calibrate-able unit of data that makes sense. It assumes that the counter-chop technique proves to be effective for METIS, too (it worked quite well on VISIR).