User Tools

Site Tools


data:data_analysis_manual

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
data:data_analysis_manual [2022/03/31 23:53]
Peter Young [Data products]
data:data_analysis_manual [2024/01/23 19:27] (current)
eric buchlin [Burn-in Effects] corrected files published in data release 4.0
Line 11: Line 11:
 SPICE papers: SPICE papers:
   * Instrument paper: //Astron. Astrophys.//,​ **642**, A14, 2020. [[https://​doi.org/​10.1051/​0004-6361/​201935574|DOI]],​ [[https://​ui.adsabs.harvard.edu/​abs/​2020A%26A...642A..14S|ADS]]   * Instrument paper: //Astron. Astrophys.//,​ **642**, A14, 2020. [[https://​doi.org/​10.1051/​0004-6361/​201935574|DOI]],​ [[https://​ui.adsabs.harvard.edu/​abs/​2020A%26A...642A..14S|ADS]]
-  * First results: ​Submitted+  * First results: ​//Astron. Astrophys.//,​ **656**, A38, 2021. [[https://​doi.org/​10.1051/​0004-6361/​202141221|DOI]],​ [[https://​ui.adsabs.harvard.edu/​abs/​2021A%26A...656A..38F/​abstract|ADS]]
   * In-flight calibration:​ Not available yet   * In-flight calibration:​ Not available yet
   * Other papers can be found in the [[https://​ui.adsabs.harvard.edu/​public-libraries/​rIi8Zx4zTg2stP5nTojx2Q|SPICE ADS library]]   * Other papers can be found in the [[https://​ui.adsabs.harvard.edu/​public-libraries/​rIi8Zx4zTg2stP5nTojx2Q|SPICE ADS library]]
Line 22: Line 22:
  
 The data levels, FITS files, and headers are described in The data levels, FITS files, and headers are described in
-  * The [[https://​spice.ias.u-psud.fr/​spice-data/documents/SPICE-UIO-DPDD-0002-1.4-Data_Product_Description_Document.pdf|Data Products Description Document]] (DPDD)+  * The [[https://​spice-wiki.ias.u-psud.fr/​lib/exe/fetch.php/​public:​spice-uio-dpdd-0002-1.8-data_product_description_document_spicefits_.pdf|Data Products Description Document]] (DPDD)
   * The LLDPDD for the Low-Latency (LL) files   * The LLDPDD for the Low-Latency (LL) files
  
-The full SPICE data set (to be cited in papers) ​has a DOI: [[https://​doi.org/​10.5270/​esa-lbmdy7c|doi:​10.5270/​esa-lbmdy7c]]. DOIs will also be attributed to each data release.+The full SPICE data set has a DOI: [[https://​doi.org/​10.5270/​esa-lbmdy7c|doi:​10.5270/​esa-lbmdy7c]]. DOIs are also attributed to [[https://​spice.osups.universite-paris-saclay.fr/​spice-data/​|each data release]].
  
 +A typical Level 2 SPICE filename has the form ''​solo\_L2\_spice-w-exp\_20230604T073743\_V02\_184550205-019.fits'',​ where "​w-exp"​ is the code, "​184550205"​ is the unique observation ID (''​SPIOBSID''​),​ and "​019"​ is the raster number (''​RASTERNUM''​) within the observation. ​ (More information is given in the [[https://​spice-wiki.ias.u-psud.fr/​lib/​exe/​fetch.php/​public:​spice-uio-dpdd-0002-1.8-data_product_description_document_spicefits_.pdf|DPDD]].) ​ When browsing the SPICE catalog and archive you will see the following codes ("file descriptors"​ in Solar Orbiter terminology):​
 +  * **n-ras**: a raster scan with one of the narrow slits (2", 4" or 6")\\
 +  * **w-ras**: a raster scan with the wide slit (30"​)\\
 +  * **n-sit**: a sit-and-stare study with one of the narrow slits\\
 +  * **w-sit**: a sit-and-stare study with the wide slit\\
 +  * **n-exp**: a single exposure with a narrow slit that yields the entire spectrum\\
 +  * **w-exp**: a single exposure with the wide slit that yields the entire spectrum
  
-When browsing ​the SPICE catalog and archive you will see the following codes:+For the "​ras"​ data, the FITS file contains a single raster scan (with ''​RASTERNUM=000''​). If the raster is repeated, then each repeat goes in a new file (with an incremented ''​RASTERNUM''​).
  
-**n-ras**: a raster scan with one of the narrow slits (2", ​4" or 6")\\ +Full spectrum rasters can be performed in the "exp" modebut then each exposure will end up in a different FITS file, with a single ''​SPIOBSID''​ and incrementing ''​RASTERNUM''​. 
-**w-ras**a raster scan with the wide slit (30"​)\\ + 
-**n-sit**a sit-and-stare study with one of the narrow slits\\ +==== Software ==== 
-**w-sit**: a sit-and-stare study with the wide slit\\ + 
-**n-exp**a single exposure with a narrow slit that yields the entire spectrum\\ +Different pieces of software are mentioned below. Here is an overview: 
-**w-exp**: a single exposure with the wide slit that yields the entire spectrum+  ​For IDL/SSW
 +    [[https://​github.com/​ITA-Solar/​solo-spice-ql#​spice-quicklook-and-data-analysis-software|SPICE Quicklook ​and Data Analysis Software]] 
 +  For Python
 +    [[https://​sospice.readthedocs.io/​en/​stable/​|sospice]]:​ instrument-specific functionalities required for day-to-day SPICE data analysis 
 +    ​[[https://​docs.sunpy.org/​projects/​sunraster/​en/​latest/​|sunraster]]:​ tools to read and analyze spectrogram data
  
-For the "​ras"​ data, the FITS file contains a single raster scan. If the raster is repeated, then each repeat goes in a new file. 
  
-Rasters can be performed in the "​exp"​ mode, but each exposure will end up in a different FITS file. 
 ===== Find data ===== ===== Find data =====
  
-UiO catalogue:​ +SPICE files catalogue:​ 
-  * Can be generated with IDL/​SolarSoft (SSW) from a local data tree:+  * The catalogue file generated from all SPICE files (latest versions) is accessible [[http://​astro-sdc-db.uio.no/​vol/​spice/​fits/​spice_catalog.csv|through the UiO website]] (restricted to SPICE team) 
 +  * This catalogue can be generated with IDL/​SolarSoft (SSW) from a local data tree:
     - Make sure ''​$SPICE\_DATA''​ points to the data tree you want to have catalogued     - Make sure ''​$SPICE\_DATA''​ points to the data tree you want to have catalogued
     - Run SPICE\_GEN\_CAT. The catalogue file will be written to the data tree root directory (you need to have write access)     - Run SPICE\_GEN\_CAT. The catalogue file will be written to the data tree root directory (you need to have write access)
     - Run SPICE\_CAT     - Run SPICE\_CAT
-  * The catalogue ​file generated from all SPICE files is accessible ​[[http://astro-sdc-db.uio.no/vol/spice/fits/spice_catalog.txt|through the UiO website]] (restricted to SPICE team+  * The catalogue ​of all files in a specific data release ​is included with the data release (see below). This version of the catalogue also include additional columns for the list of windows, the file size, and a reference to the dark used for on-ground dark subtraction. 
-  * How to use it in [[data:​data_analysis_manual:​read_catalog_python|Python]] +  * How to use it in Python: a [[data:​data_analysis_manual:​read_catalog_python|simple script]] can be used to read it as a Python pandas DataFrame object. The same can be done using the ''​Catalog''​ object from the [[https://sospice.readthedocs.io/en/​stable/​|''​sospice''​ Python module]] (see examples in [[https://​git.ias.u-psud.fr/spice/data-analysis-club/-/​blob/​main/​20230627-sospice/​sospice-demo.ipynb|this tutorial]]). 
-  * CSV export ​of the part of the catalogue corresponding to each release is provided with the release.+  * How to use it in IDL: [[data:​data_analysis_manual#​the_spice_catalog|see below]] 
 + 
 + 
 +Data releases: 
 + 
 +  * All L1 and L2 files with their latest versions (at a given date) are collected in a data release 
 +  * The releases are provided under https://​spice.osups.universite-paris-saclay.fr/​spice-data/​ 
 +  * Released files are also sent to the SOAR (see below) 
 +  * The latest release is updated with new data as they arrive (and after a superficial validation step). 
 +  * Each release includes release notes, and a CSV catalog ​of the files in the release
 +  * The SPICE data used in a publication have to be cited properly. How to do this is explained in the release ​notes.
  
-SOAR (and institute mirrors)+SOAR: 
-  * [[https://​soar.esac.esa.int/​|SOAR]] ([[https://​issues.cosmos.esa.int/​solarorbiterwiki/​display/​SOSP/​SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]])+  * [[https://​soar.esac.esa.int/​|SOAR]] ([[https://​issues.cosmos.esa.int/​solarorbiterwiki/​display/​SOSP/​SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]], [[https://​www.cosmos.esa.int/​web/​soar/​home|SOAR help pages]]) 
 +  * SOAR has a TAP interface that can be used for automated queries ([[https://​www.cosmos.esa.int/​web/​soar/​tables-views-and-columns|documentation of available tables]]) 
 +  * The ''​sunpy-soar''​ [[https://​github.com/​sunpy/​sunpy-soar/​|Python module]] uses this TAP interface for queries through SunPy/​Fido 
 +    * Only queries by instrument / time / level / product [[https://​github.com/​sunpy/​sunpy-soar/​issues/​46|at the moment]]
  
 Quicklooks: Quicklooks:
   * Quicklook archive (to be developed)   * Quicklook archive (to be developed)
-  * [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki/​spice_xfiles|Quicklooks in SSW]]+  * [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki/​spice_xfiles|Quicklooks ​procedure ​in SSW]] 
 +  * A quicklook app (in Python) is in development at IAS. 
 +  * [[https://​git.ias.u-psud.fr/​spice/​pre-analysis/​|"​Pre-analysis"​]]:​ already processed quicklook images (per STP)
  
-In the future: VSO. VSO through SSW or sunpy. Sunpy/Fido+VSO: 
 +  * Released SPICE files are ingested from SOAR to the VSO ([[https://​sdac.virtualsolar.org/​cgi/​search|search form]]) 
 +  * Accessible ​through ​IDL/SSW and Python/Fido
  
 ===== Tips and Advice on Analyzing SPICE Data ===== ===== Tips and Advice on Analyzing SPICE Data =====
Line 68: Line 95:
 ==== Burn-in Effects ==== ==== Burn-in Effects ====
  
-The SPICE detectors have microchannel plates (MCPs). Over time, the sensitivity of MCPs decreases at the locations of strong emission lines. This effect is referred to as "​burn-in"​ and results in a flat-top or "​self-reversal"​ to the line profile. This has already been noticed for the C III 977 line. The SPICE team plans to provide ​a correction ​to this effect once it is fully characterized.+The SPICE detectors have microchannel plates (MCPs). Over time, the sensitivity of MCPs decreases at the locations of strong emission lines. This effect is referred to as "​burn-in"​ and results in a flat-top or "​self-reversal"​ to the line profile. This has already been noticed for the C III 977 line. From data release 4.0, level-2 files are corrected for burn-in for six lines. The effects of burn-in continue ​to be monitored, particularly for weaker lines for which a correction is not currently available. 
 + 
 +The lines that are being corrected (as of 2023-11) are: N IV 76.5 nm, Ne VIII 77.0 nm, Ne VIII 78.0 nm, C III 97.7 nm, Lyman-beta 102.6 nm, and O VI 103.2 nm.
  
 ==== Flare lines ==== ==== Flare lines ====
  
-The two key flare lines for EIS are Fe XVIII 974.86 and Fe XX 721.56 that are formed around ​7 MK and 10 MK, respectivelyThe Fe XVIII line will probably ​be seen in active region cores (at least from bright active regions).+The two key flare lines for SPICE are
 +  * Fe XVIII 974.86Å, formed around 7MK. It is located between the stronger H I 972.54Å (Lyman-gamma) ​and C III 977.02Å lines, but measurements are possible in flares, and are probably also possible in bright active region cores. 
 +  * Fe XX 721.56Å, formed around ​10MKIt lies close to Fe VIII 721.26Å. One can estimate the strength of this line by comparing with Mg IX 706.06Å. The latter ​will be about a factor 10 stronger.
  
-The Fe XVIII line is located between ​the stronger H I 972.54 (Lyman-gamma) and C III 977.02, but measurements should be possible.+To determine if a flare is occurring at the time of a SPICE observation,​ please check [[https://​datacenter.stix.i4ds.net/​view/​plot/​lightcurves|the STIX Data Browser]].
  
-The Fe XX line lies close to Fe VIII 721.26. One can estimate the strength ​of this line by comparing with Mg IX 706.06. The latter will be about a factor 10 stronger. +A list of flares observed ​by SPICE from the beginning ​of the mission to around August 2023 is available ​at [[https://spice-wiki.ias.u-psud.fr/doku.php/public:​eventlists:​flares|https:​//​spice-wiki.ias.u-psud.fr/​doku.php/​public:​eventlists:​flares]].
- +
-To determine if a flare is occurring at the time of a SPICE observation,​ check the STIX Data Browser ​at+
- +
-[[https://datacenter.stix.i4ds.net/view/plot/lightcurves +
-]]+
 ===== Access data ===== ===== Access data =====
 +
 +Once you have found data ([[#​find_data|see above]]), data files can generally be accessed from the same place and/or using the same libraries.
  
 SPICE team internal access: SPICE team internal access:
   * [[http://​astro-sdc-db.uio.no/​vol/​spice/​|UiO website]] (restricted).   * [[http://​astro-sdc-db.uio.no/​vol/​spice/​|UiO website]] (restricted).
  
-Once the data will be released: +For released ​data:
-  * [[https://​soar.esac.esa.int/​|SOAR]] (and institute mirrors)+
   * [[https://​spice.ias.u-psud.fr/​data/​archives|List of data releases]].   * [[https://​spice.ias.u-psud.fr/​data/​archives|List of data releases]].
-  * VSO. VSO through SSW or sunpy. Sunpy/Fido+  * [[https://​soar.esac.esa.int/​|SOAR]] (and institute mirrors) 
 +  * [[https://​www.virtualsolar.org/​cgi-bin/​search|VSO]]. VSO through SSW or SunPy/Fido 
 +  * [[https://​sospice.readthedocs.io/​en/​stable/​|sospice]] provides access to data files either from a release, from SOAR, or from any directory tree containing SPICE files (see this [[https://​git.ias.u-psud.fr/​spice/​data-analysis-club/​-/​blob/​main/​20230627-sospice/​sospice-demo.ipynb|tutorial]]).
  
  
 ===== Read and display data (IDL) ===== ===== Read and display data (IDL) =====
  
-External links: +Detailed information about the IDL data analysis software is available at [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki|https://​github.com/ITA-Solar/​solo-spice-ql/wiki]].
-  * Oslo's quicklook tools [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki|spice_data, spice_object...]] +
-  * Peter Young'​s [[https://pyoung.org/spice/analysis_guide.html|SPICE Analysis Guide]] +
  
 +Some additional information is given below.
 ==== Environment variables ==== ==== Environment variables ====
  
Line 132: Line 159:
 The first true science observations were obtained during 18 to 22 November 2020. For example, an active region can be seen in the raster beginning 19:57 UT on 18-Nov-2020. The first true science observations were obtained during 18 to 22 November 2020. For example, an active region can be seen in the raster beginning 19:57 UT on 18-Nov-2020.
  
 +An M-class flare was captured with SPICE during observations on 2 April 2022 [[https://​ui.adsabs.harvard.edu/​abs/​2023arXiv230702396J/​abstract|(Janvier et al., 2023, A&A)]]. A sequence of 15-min cadence large rasters was run for most of the day and the flare can be seen in the raster beginning 13:15.The same active region complex produced a number of other, smaller flares during 1-4 April.
 ==== Ingesting downloaded data ==== ==== Ingesting downloaded data ====
  
Line 148: Line 176:
 A file can be read into an IDL object with: A file can be read into an IDL object with:
  
-''​IDL>​ d=spice\_object(file)''​+''​IDL>​ d=spice\_data(file)''​
  
 ==== Extracting information from the data object ==== ==== Extracting information from the data object ====
Line 190: Line 218:
 Use a 3-button mouse to browse the images and spectra: the middle button allows you to select a new pixel, the left button zooms in and the right button zooms out. Use a 3-button mouse to browse the images and spectra: the middle button allows you to select a new pixel, the left button zooms in and the right button zooms out.
  
 +
 +===== Ephemeris information =====
 +
 +You can access ephemeris information from the SPICE headers. For example, the keywords:
 +  * ''​DSUN\_AU'':​ distance of the spacecraft from the Sun in AU\\
 +  * ''​EAR\_TDEL'':​ light travel time difference between Sun centre to Spacecraft and Sun centre to Earth (seconds)
 +
 +The full list of keywords is given in the [[https://​spice-wiki.ias.u-psud.fr/​lib/​exe/​fetch.php/​public:​spice-uio-dpdd-0002-1.8-data_product_description_document_spicefits_.pdf|DPDD]] - search for "Solar ephemeris keywords"​.
 +
 +Users should make sure to correct the SPICE observing times for EAR\_TDEL in order to compare with observations from Earth-orbiting spacecraft. This is especially important for highly-dynamic structures such as flares or CMEs.
 +
 +Coordinate frames and data axes are described using standard WCS keywords. In Python, they can be managed using the astropy/WCS and SunPy framework, and the corresponding metadata are loaded automatically when reading the SPICE files using ''​sunraster''​.
 ===== Read and display data (Python) ===== ===== Read and display data (Python) =====
  
 Links: Links:
-  * [[:​data:​data_analysis_manual:​sunraster|with sunraster]].+  * Example with [[:​data:​data_analysis_manual:​sunraster|with sunraster]]. 
 +  * Tutorial about [[https://​git.ias.u-psud.fr/​spice/​data-analysis-club/​-/​blob/​main/​20230516-coordinates/​coordinates.ipynb|plotting and coordinates transforms]]
  
  
Line 202: Line 243:
 Calibration reports (links) Calibration reports (links)
  
-Data calibration applied to L2 data (links) ​+Data calibration ​steps applied to L2 data and known instrumental artefacts are listed in the release notes of [[https://​spice.ias.u-psud.fr/​data/​archives|each release]].
  
-Known instrumental artefacts: see release notes.+==== Change of radiometric calibration with time ====
  
 +The SPICE instrument sensitivity has decreased by a factor 2-3 in the first two years of the mission and has remained approximately constant since then. The SPICE team have characterized these changes and, as of 8-Nov-2023, the level-2 SPICE files incorporate the sensitivity degradation. ​
 +
 +==== Units and uncertainties ====
 +
 +The data cube (returned by the SSW object method ''​get\_window\_data''​ or by the Python ''​astropy''​ or ''​sunraster''​ libraries) of the SPICE level-2 files contains spectral radiances in units of W/​m²/​sr/​nm. The wavelength vector has units of nm. To convert a radiance in W/m²/sr to erg/​cm²/​s/​sr,​ it is necessary to multiply by 1000.
 +
 +Uncertainties on the spectral radiances can be obtained with:
 +  * SSW: the routine ''​spice\_getwindata''​. This returns an IDL structure with ''​int''​ and ''​err''​ tags. The former is the irradiance and the latter is the irradiance uncertainty.
 +  * Python: the ''​sospice''​ function [[https://​sospice.readthedocs.io/​en/​stable/​calibration.html#​error-computation|spice_error]]
 +
 +The method for computing the uncertainties is described in  Appendix B of [[https://​ui.adsabs.harvard.edu/​abs/​2023A%26A...673A..82H/​abstract|Huang et al. (2023, A&A, 673, A82)]].
 ==== Line fitting ==== ==== Line fitting ====
  
-TBW+=== Using EIS fitting software for SPICE === 
 + 
 +The routine spice\_getwindata returns a "​windata"​ structure in a form compatible with the eis\_auto\_fit suite of routines for Hinode/EIS. Further information on these routines is given in [[https://​doi.org/​10.5281/​zenodo.6339584 
 +|EIS Software Note 15]]. The mask spectra fitting routines (Case 2 in the document) do not currently work with SPICE, however. 
 + 
 +The EIS software enable a number of operations to be performed on windata structures, as described in [[https://​sohoftp.nascom.nasa.gov/​solarsoft/​hinode/​eis/​doc/​eis_notes/​21_WINDATA/​eis_swnote_21.pdf|EIS Software Note 21]]. For example, eis\_bin\_windata allows spatial binning of the rasters. 
 + 
 +If you have problems running the EIS software on SPICE data, please contact [[https://​science.gsfc.nasa.gov/​sed/​bio/​peter.r.young|Dr. Peter Young]]. 
 + 
 +=== Line window width and spectral tilt === 
 + 
 +The SPICE slits are tilted with respect to the detector axes, an effect that is corrected in the level-2 SPICE files. That is, the emission line is aligned to the detector y-axis in the level-2 data, but the image of the window is seen to be tilted (see images below). The maximum window width for unbinned data is 32 wavelength pixels. The spectrum tilt is about 10 pixels from bottom-to-top for the SW data and about 5 pixels for the LW data. In the corrected data, the wavelength window is tilted to the right (going from bottom to top) for the LW channel, and to the left for the SW channel. 
 + 
 +{{ :​data:​plot_tilt_example.png?​500 |}} 
 + 
 +A consequence of the tilt is that the amount of continuum on the sides of an isolated emission line varies along the slit. For, e.g., Ne VIII 770 in the SW channel, there is relatively little continuum on the right side of the profile at the top of the slit, and relatively little on the left side at the bottom of the slit. If you use an automatic fitting routine for a SPICE raster, you may find bad fits at the top and bottom of the slit because of this effect. The routine will struggle to find a good continuum measurement. Forcing the continuum to be uniform can be a good solution.
  
 +A further problem is when a window contains more than one line, which makes  it more difficult to estimate a continuum level. For example, N IV 765.15 has a weak blend on the short-wavelength side (N III 764.35). At the top of the slit there may not be enough continuum for a good fit, while at the bottom of the slit there may not be enough pixels to accurately fit the blending line.
data/data_analysis_manual.1648763626.txt.gz · Last modified: 2022/03/31 23:53 by Peter Young