Differences

This shows you the differences between two versions of the page.

--- data:data_analysis_manual [2021/06/08 14:21] – More details, links... eric buchlin
+++ data:data_analysis_manual [2026/08/24 14:48] (current) – [Find data] added event list links Peter Young
@@ Line 2: / Line 2: @@
 <WRAP center round important 70%>
-This manual is very preliminary, please be patient or help adding what is missing.
+This manual is preliminary, please be patient or help adding what is missing.
 </WRAP>
@@ Line 11: / Line 11: @@
 SPICE papers:
   * Instrument paper: //Astron. Astrophys.//, **642**, A14, 2020. [[https://doi.org/10.1051/0004-6361/201935574|DOI]], [[https://ui.adsabs.harvard.edu/abs/2020A%26A...642A..14S|ADS]]
-  * First results: Submitted
+  * First results: //Astron. Astrophys.//, **656**, A38, 2021. [[https://doi.org/10.1051/0004-6361/202141221|DOI]], [[https://ui.adsabs.harvard.edu/abs/2021A%26A...656A..38F/abstract|ADS]]
   * In-flight calibration: Not available yet
   * Other papers can be found in the [[https://ui.adsabs.harvard.edu/public-libraries/rIi8Zx4zTg2stP5nTojx2Q|SPICE ADS library]]
@@ Line 22: / Line 22: @@
 The data levels, FITS files, and headers are described in
-  * The [[https://spice.ias.u-psud.fr/spice-data/documents/SPICE-UIO-DPDD-0002-1.4-Data_Product_Description_Document.pdf|Data Products Description Document]] (DPDD)
+  * The {{ :public:spice-uio-dpdd-0002-2.2-data_product_description_document_spicefits_all_accepted.pdf |Data Product Description Document (DPDD) version 2.2}}
   * The LLDPDD for the Low-Latency (LL) files
-The full SPICE data set (to be cited in papers) has a DOI: [[https://doi.org/10.5270/esa-lbmdy7c|doi:10.5270/esa-lbmdy7c]]. DOIs will also be attributed to each data release.
+The full SPICE data set has a DOI: [[https://doi.org/10.5270/esa-lbmdy7c|doi:10.5270/esa-lbmdy7c]]. DOIs are also attributed to [[https://spice.osups.universite-paris-saclay.fr/spice-data/|each data release]].
+A typical Level 2 SPICE filename has the form ''solo\_L2\_spice-w-exp\_20230604T073743\_V02\_184550205-019.fits'', where "w-exp" is the code, "184550205" is the unique observation ID (''SPIOBSID''), and "019" is the raster number (''RASTERNUM'') within the observation.  (More information is given in the [[https://spice-wiki.ias.u-psud.fr/lib/exe/fetch.php/public:spice-uio-dpdd-0002-1.8-data_product_description_document_spicefits_.pdf|DPDD]].)  When browsing the SPICE catalog and archive you will see the following codes ("file descriptors" in Solar Orbiter terminology):
+  * **n-ras**: a raster scan with one of the narrow slits (2", 4" or 6")\\
+  * **w-ras**: a raster scan with the wide slit (30")\\
+  * **n-sit**: a sit-and-stare study with one of the narrow slits\\
+  * **w-sit**: a sit-and-stare study with the wide slit\\
+  * **n-exp**: a single exposure with a narrow slit that yields the entire spectrum\\
+  * **w-exp**: a single exposure with the wide slit that yields the entire spectrum
+For the "ras" data, the FITS file contains a single raster scan (with ''RASTERNUM=000''). If the raster is repeated, then each repeat goes in a new file (with an incremented ''RASTERNUM'').
+Full spectrum rasters can be performed in the "exp" mode, but then each exposure will end up in a different FITS file, with a single ''SPIOBSID'' and incrementing ''RASTERNUM''.
+==== Software ====
+Different pieces of software are mentioned below. Here is an overview:
+  * For IDL/SSW:
+    * [[https://github.com/ITA-Solar/solo-spice-ql#spice-quicklook-and-data-analysis-software|SPICE Quicklook and Data Analysis Software]]
+  * For Python:
+    * [[https://sospice.readthedocs.io/en/stable/|sospice]]: instrument-specific functionalities required for day-to-day SPICE data analysis
+    * [[https://docs.sunpy.org/projects/sunraster/en/latest/|sunraster]]: tools to read and analyze spectrogram data
 ===== Find data =====
-UiO catalogue:
+SPICE files catalogue:
-  * Can be generated with IDL/SolarSoft (SSW) from a local data tree:
+  * The catalogue file generated from all SPICE files (latest versions) is accessible [[http://astro-sdc-db.uio.no/vol/spice/fits/spice_catalog.csv|through the UiO website]] (restricted to SPICE team)
-    - Make sure $SPICE_DATA points to the data tree you want to have catalogued
+  * This catalogue can be generated with IDL/SolarSoft (SSW) from a local data tree:
-    - Run SPICE_GEN_CAT. The catalogue file will be written to the data tree root directory (you need to have write access)
+    - Make sure ''$SPICE\_DATA'' points to the data tree you want to have catalogued
-    - Run SPICE_CAT
+    - Run SPICE\_GEN\_CAT. The catalogue file will be written to the data tree root directory (you need to have write access)
-  * It is accessible [[http://astro-sdc-db.uio.no/vol/spice/fits/spice_catalog.txt|through the UiO website]] (restricted to SPICE team)
+    - Run SPICE\_CAT
-  * How to use it in [[spice_internal:data:read_catalog_python|Python]]
+  * The catalogue of all files in a specific data release is included with the data release (see below). This version of the catalogue also include additional columns for the list of windows, the file size, and a reference to the dark used for on-ground dark subtraction.
-  * A CSV export of the part of the catalogue corresponding to each release is provided with the release.
+  * How to use it in Python: the catalogue can be read as a ''Catalog'' object from the [[https://sospice.readthedocs.io/en/stable/|''sospice'' Python module]], and then used to find observations with specific characteristics (see examples in [[https://git.ias.u-psud.fr/spice/data-analysis-club/-/blob/main/20230627-sospice/sospice-demo.ipynb|this tutorial]]).
+  * How to use it in IDL: [[data:data_analysis_manual#the_spice_catalog|see below]]
-SOAR (and institute mirrors):
-  * [[https://soar.esac.esa.int/|SOAR]] ([[https://issues.cosmos.esa.int/solarorbiterwiki/display/SOSP/SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]])
+Data releases:
+  * All L1 and L2 files with their latest versions (at a given date) are collected in a data release
+  * The releases are provided under https://spice.osups.universite-paris-saclay.fr/spice-data/, in subdirectories named ''release-{release_tag}/''
+  * The latest release tag can be obtained from the content of [[https://spice.osups.universite-paris-saclay.fr/spice-data/metadata/latest-release.txt|latest-release.txt]]
+  * Released files are also sent to the SOAR (see below)
+  * The latest release is updated with new data as they arrive (and after a superficial validation step).
+  * Each release includes release notes, and a CSV catalog of the files in the release.
+  * The SPICE data used in a publication have to be cited properly. How to do this is explained in the release notes.
+SOAR:
+  * [[https://soar.esac.esa.int/|SOAR]] ([[https://issues.cosmos.esa.int/solarorbiterwiki/display/SOSP/SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]], [[https://www.cosmos.esa.int/web/soar/home|SOAR help pages]], [[https://www.cosmos.esa.int/web/soar/inventory-plots|SOAR data coverage]])
+  * SOAR has a TAP interface that can be used for automated queries ([[https://www.cosmos.esa.int/web/soar/tables-views-and-columns|documentation of available tables]])
+  * The ''sunpy-soar'' [[https://github.com/sunpy/sunpy-soar/|Python module]] uses this TAP interface for queries through SunPy/Fido
+    * Only queries by instrument / time / level / product [[https://github.com/sunpy/sunpy-soar/issues/46|at the moment]]
 Quicklooks:
   * Quicklook archive (to be developed)
-  * [[https://github.com/ITA-Solar/solo-spice-ql/wiki/spice_xfiles|Quicklooks in SSW]]
+  * [[https://github.com/ITA-Solar/solo-spice-ql/wiki/spice_xfiles|Quicklooks procedure in SSW]]
+  * A [[https://git.ias.u-psud.fr/spice/data_quicklook|quicklook app]] (in Python) is in development at IAS.
+  * [[https://git.ias.u-psud.fr/spice/pre-analysis/|"Pre-analysis"]]: already processed quicklook images (per STP)
-In the future: VSO. VSO through SSW or sunpy. Sunpy/Fido
+VSO:
+  * Released SPICE files are ingested from SOAR to the VSO ([[https://sdac.virtualsolar.org/cgi/search|search form]])
+  * Accessible through IDL/SSW and Python/Fido
+  * FIXME As of 2024-03-29, the SPICE files on VSO are old files from release 3.0 only
+Event lists:
+  * [[public:eventlists:flares|Flares]]
+  * [[public:eventlists:prominences|Prominences]]
+===== Tips and Advice on Analyzing SPICE Data =====
+==== Doppler Maps ====
+WARNING: Doppler maps created from SPICE lines show unusual patterns that we believe are due to the point spread function of the instrument. Work is underway on a deconvolution procedure that will correct for this effect. If you need advice on features you see in a Doppler map, please contact the SPICE PI at frederic.auchere -at- universite-paris-saclay.fr or another member of the SPICE team.
+==== Burn-in Effects ====
+The SPICE detectors have microchannel plates (MCPs). Over time, the sensitivity of MCPs decreases at the locations of strong emission lines. This effect is referred to as "burn-in" and results in a flat-top or "self-reversal" to the line profile. This has already been noticed for the C III 977 line. From data release 4.0, level-2 files are corrected for burn-in for six lines. The effects of burn-in continue to be monitored, particularly for weaker lines for which a correction is not currently available.
+The lines that are being corrected (as of 2023-11) are: N IV 76.5 nm, Ne VIII 77.0 nm, Ne VIII 78.0 nm, C III 97.7 nm, Lyman-beta 102.6 nm, and O VI 103.2 nm.
+==== Flare lines ====
+The two key flare lines for SPICE are:
+  * Fe XVIII 974.86Å, formed around 7MK. It is located between the stronger H I 972.54Å (Lyman-gamma) and C III 977.02Å lines, but measurements are possible in flares, and are probably also possible in bright active region cores.
+  * Fe XX 721.56Å, formed around 10MK. It lies close to Fe VIII 721.26Å. One can estimate the strength of this line by comparing with Mg IX 706.06Å. The latter will be about a factor 10 stronger.
+To determine if a flare is occurring at the time of a SPICE observation, please check [[https://datacenter.stix.i4ds.net/view/plot/lightcurves|the STIX Data Browser]].
+A list of flares observed by SPICE from the beginning of the mission to around August 2023 is available at [[https://spice-wiki.ias.u-psud.fr/doku.php/public:eventlists:flares|https://spice-wiki.ias.u-psud.fr/doku.php/public:eventlists:flares]].
 ===== Access data =====
+Once you have found data ([[#find_data|see above]]), data files can generally be accessed from the same place and/or using the same libraries.
 SPICE team internal access:
   * [[http://astro-sdc-db.uio.no/vol/spice/|UiO website]] (restricted).
-Once the data will be released:
+For released data:
-  * [[https://soar.esac.esa.int/|SOAR]] (and institute mirrors)
   * [[https://spice.ias.u-psud.fr/data/archives|List of data releases]].
-  * VSO. VSO through SSW or sunpy. Sunpy/Fido
+  * [[https://soar.esac.esa.int/|SOAR]] (and institute mirrors)
+  * [[https://www.virtualsolar.org/cgi-bin/search|VSO]]. VSO through SSW or SunPy/Fido
+  * [[https://sospice.readthedocs.io/en/stable/|sospice]] provides access to data files either from a release, from SOAR, or from any directory tree containing SPICE files (see this [[https://git.ias.u-psud.fr/spice/data-analysis-club/-/blob/main/20230627-sospice/sospice-demo.ipynb|tutorial]]).
-===== Read an display data =====
+===== Read and display data (IDL) =====
-IDL
+Detailed information about the IDL data analysis software is available at [[https://github.com/ITA-Solar/solo-spice-ql/wiki|https://github.com/ITA-Solar/solo-spice-ql/wiki]].
-  * Oslo's quicklook tools [[https://github.com/ITA-Solar/solo-spice-ql/wiki|spice_data, spice_object...]]
-Python
+Some additional information is given below.
-  * [[https://docs.sunpy.org/projects/sunraster/|sunraster]].
+==== Environment variables ====
+Choose a location to store your SPICE data (e.g., '/my_data/spice') and then point the environment variable ''$SPICE\_DATA'' to it:
+''setenv,'SPICE\_DATA=/my\_data/spice'
+''
+This line should be added to your IDL\_STARTUP file.
+Data are organized under ''$SPICE\_DATA'' with a year/month/day subdirectory structure. See the "Ingesting downloaded data" section below.
+==== The SPICE catalog ====
+The SPICE catalog can be accessed by doing:
+''IDL> spice\_cat''
+Use the "SPICE\_GEN\_CAT" button to make sure you have the most up-to-date list.
+==== Reference data-sets ====
+Perhaps the most useful data from the early commissioning phase (before July 2020) are the raster scans on 28-May-2020.
+:05, 16:50 - disk center rasters with 20s exposures \\
+:50, 18:35 - north limb rasters with 20s exposures\\
+:35, 20:20 - south limb rasters with 20s exposures\\
+:20, 22:05 - west limb rasters with 20s exposures\\
+:05, 23:47 - east limb rasters with 20s exposures
+The first true science observations were obtained during 18 to 22 November 2020. For example, an active region can be seen in the raster beginning 19:57 UT on 18-Nov-2020.
+An M-class flare was captured with SPICE during observations on 2 April 2022 [[https://ui.adsabs.harvard.edu/abs/2023arXiv230702396J/abstract|(Janvier et al., 2023, A&A)]]. A sequence of 15-min cadence large rasters was run for most of the day and the flare can be seen in the raster beginning 13:15.The same active region complex produced a number of other, smaller flares during 1-4 April.
+==== Ingesting downloaded data ====
+After you have downloaded some SPICE FITS files, you can ingest them into your data directory with spice\_ingest:
+''IDL> spice\_ingest, files''
+This routine automatically creates the sub-directory structure (year/month/day) within $SPICE\_DATA for the files.
+==== Finding and reading a FITS file ====
+Once a file has been ingested, then you can find it with spice\_find\_file using the observation time:
+''IDL> file=spice\_find\_file('28-may-2020 16:05')''
+A file can be read into an IDL object with:
+''IDL> d=spice\_data(file)''
+==== Extracting information from the data object ====
+The table below gives some methods for extracting information out of the data object. Where "i" is given, it means the index of a wavelength window should be specified (indices begin at 0).
+You can get a list of all methods by doing:
+IDL> d->help
+^ Command                        ^ Function                               ^
+| d->get\_number\_windows()      | No. of spectral windows                |
+| d->get\_window\_data(i,/load)  | Extract a data window                  |
+| d->get\_lambda\_vector(i)      | Get wavelength vector for the window   |
+| d->get\_time\_vector(i)        | Get the mid-time for each exposure     |
+| d->get\_header(i)              | Extract a data window header           |
+| d->get\_window\_id(i)          | Get the data window ID                 |
+| d->get\_start\_time()          | Start time of observation              |
+| d->get\_end\_time()            | End time of observation                |
+| d->get\_sit\_and\_stare()      | Set to 1 if sit-and-stare observation  |
+| d->get\_number\_exposures()    | Number of exposures in raster          |
+| d->get\_xcen(i)                | Get the X-center for the window        |
+| d->get\_ycen(i)                | Get the Y-center for the window        |
+==== Quicklook tools ====
+A set of five widget-based tools are available for browsing SPICE data, and these can be accessed through spice\_xfiles:
+'' IDL> spice\_xfiles''
+This allows you to select a FITS file from your SPICE data directory. A new widget appears from which you can then select one of the five quicklook tools: Detector, Raster Browser, Raster, Whisker and Intensity map. These tools mimic software that were available for EIS and IRIS.
+For further details, please visit the [[https://github.com/ITA-Solar/solo-spice-ql/wiki|SPICE Quicklook and Data Analysis Software Page]].
+=== Raster Browser ===
+This is useful for browsing the 3D data cubes from SPICE rasters. In addition to being called from spice\_xfiles (see above), it can also be called directly from the command line:
+''IDL> spice\_raster\_browser,file''
+Use a 3-button mouse to browse the images and spectra: the middle button allows you to select a new pixel, the left button zooms in and the right button zooms out.
+===== Ephemeris information =====
+You can access ephemeris information from the SPICE headers. For example, the keywords:
+  * ''DSUN\_AU'': distance of the spacecraft from the Sun in AU\\
+  * ''EAR\_TDEL'': light travel time difference between Sun centre to Spacecraft and Sun centre to Earth (seconds)
+The full list of keywords is given in the [[https://spice-wiki.ias.u-psud.fr/lib/exe/fetch.php/public:spice-uio-dpdd-0002-1.8-data_product_description_document_spicefits_.pdf|DPDD]] - search for "Solar ephemeris keywords".
+Users should make sure to correct the SPICE observing times for EAR\_TDEL in order to compare with observations from Earth-orbiting spacecraft. This is especially important for highly-dynamic structures such as flares or CMEs.
+Coordinate frames and data axes are described using standard WCS keywords. In Python, they can be managed using the astropy/WCS and SunPy framework, and the corresponding metadata are loaded automatically when reading the SPICE files using ''sunraster''.
+===== Read and display data (Python) =====
+Links:
+  * Python quicklook tool: [[https://pypi.org/project/spiceitup/|SPICE-IT-UP]].
+  * Example with [[:data:data_analysis_manual:sunraster|with sunraster]].
+  * Tutorial about [[https://git.ias.u-psud.fr/spice/data-analysis-club/-/blob/main/20230516-coordinates/coordinates.ipynb|plotting and coordinates transforms]]
@@ Line 73: / Line 251: @@
 Calibration reports (links)
-Data calibration applied to L2 data (links)
+Data calibration steps applied to L2 data and known instrumental artefacts are listed in the release notes of [[https://spice.ias.u-psud.fr/data/archives|each release]].
-Known instrumental artefacts: see release notes.
+==== Change of radiometric calibration with time ====
+The SPICE instrument sensitivity has decreased by a factor 2-3 in the first two years of the mission and has remained approximately constant since then. The SPICE team have characterized these changes and, as of 8-Nov-2023, the level-2 SPICE files incorporate the sensitivity degradation.
+==== Units and uncertainties ====
+The data cube (returned by the SSW object method ''get\_window\_data'' or by the Python ''astropy'' or ''sunraster'' libraries) of the SPICE level-2 files contains spectral radiances in units of W/m²/sr/nm. The wavelength vector has units of nm. To convert a radiance in W/m²/sr to erg/cm²/s/sr, it is necessary to multiply by 1000.
+Uncertainties on the spectral radiances can be obtained with:
+  * SSW: the routine ''spice\_getwindata''. This returns an IDL structure with ''int'' and ''err'' tags. The former is the irradiance and the latter is the irradiance uncertainty.
+  * Python: the ''sospice'' function [[https://sospice.readthedocs.io/en/stable/calibration.html#error-computation|spice_error]]
+The method for computing the uncertainties is described in  Appendix B of [[https://ui.adsabs.harvard.edu/abs/2023A%26A...673A..82H/abstract|Huang et al. (2023, A&A, 673, A82)]].
 ==== Line fitting ====
-TBW
+For IDL Gaussian fitting software, please check: **[[data:line_fitting_idl|IDL software for emission line fitting]]**.
+Fitting in Python can be done with any fitting library, including [[https://docs.astropy.org/en/stable/modeling/|astropy.modelling]]. From astropy 7.0, there is a user-friendly way of doing the fitting in parallel ([[https://github.com/aperiosoftware/parallel-modeling-examples/blob/main/spice/Final_Tutorial.ipynb|tutorial]] by Stuart Mumford).
+This is used by the [[https://pypi.org/project/pycfit/|PyCFIT]] library, which provides a GUI for fitting the different components.
+=== Line window width and spectral tilt ===
+The SPICE slits are tilted with respect to the detector axes, an effect that is corrected in the level-2 SPICE files. That is, the emission line is aligned to the detector y-axis in the level-2 data, but the image of the window is seen to be tilted (see images below). The maximum window width for unbinned data is 32 wavelength pixels. The spectrum tilt is about 10 pixels from bottom-to-top for the SW data and about 5 pixels for the LW data. In the corrected data, the wavelength window is tilted to the right (going from bottom to top) for the LW channel, and to the left for the SW channel.
+{{ :data:plot_tilt_example.png?500 |}}
+A consequence of the tilt is that the amount of continuum on the sides of an isolated emission line varies along the slit. For, e.g., Ne VIII 770 in the SW channel, there is relatively little continuum on the right side of the profile at the top of the slit, and relatively little on the left side at the bottom of the slit. If you use an automatic fitting routine for a SPICE raster, you may find bad fits at the top and bottom of the slit because of this effect. The routine will struggle to find a good continuum measurement. Forcing the continuum to be uniform can be a good solution.
+A further problem is when a window contains more than one line, which makes  it more difficult to estimate a continuum level. For example, N IV 765.15 has a weak blend on the short-wavelength side (N III 764.35). At the top of the slit there may not be enough continuum for a good fit, while at the bottom of the slit there may not be enough pixels to accurately fit the blending line.

User Tools

Site Tools

Differences

Page Tools