data:data_analysis_manual
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
data:data_analysis_manual [2023/10/17 10:59] eric buchlin [Ephemeris information] |
data:data_analysis_manual [2024/03/29 14:21] (current) eric buchlin [Find data] format warning |
||
|---|---|---|---|
| Line 2: | Line 2: | ||
| <WRAP center round important 70%> | <WRAP center round important 70%> | ||
| - | This manual is very preliminary, please be patient or help adding what is missing. | + | This manual is preliminary, please be patient or help adding what is missing. |
| </WRAP> | </WRAP> | ||
| Line 22: | Line 22: | ||
| The data levels, FITS files, and headers are described in | The data levels, FITS files, and headers are described in | ||
| - | * The [[https://spice-wiki.ias.u-psud.fr/lib/exe/fetch.php/documents_public:spice-uio-dpdd-0002-1.7-data_product_description_document_spicefits_.pdf|Data Products Description Document]] (DPDD) | + | * The [[https://spice-wiki.ias.u-psud.fr/lib/exe/fetch.php/public:spice-uio-dpdd-0002-1.8-data_product_description_document_spicefits_.pdf|Data Products Description Document]] (DPDD) |
| * The LLDPDD for the Low-Latency (LL) files | * The LLDPDD for the Low-Latency (LL) files | ||
| The full SPICE data set has a DOI: [[https://doi.org/10.5270/esa-lbmdy7c|doi:10.5270/esa-lbmdy7c]]. DOIs are also attributed to [[https://spice.osups.universite-paris-saclay.fr/spice-data/|each data release]]. | The full SPICE data set has a DOI: [[https://doi.org/10.5270/esa-lbmdy7c|doi:10.5270/esa-lbmdy7c]]. DOIs are also attributed to [[https://spice.osups.universite-paris-saclay.fr/spice-data/|each data release]]. | ||
| - | + | A typical Level 2 SPICE filename has the form ''solo\_L2\_spice-w-exp\_20230604T073743\_V02\_184550205-019.fits'', where "w-exp" is the code, "184550205" is the unique observation ID (''SPIOBSID''), and "019" is the raster number (''RASTERNUM'') within the observation. (More information is given in the [[https://spice-wiki.ias.u-psud.fr/lib/exe/fetch.php/public:spice-uio-dpdd-0002-1.8-data_product_description_document_spicefits_.pdf|DPDD]].) When browsing the SPICE catalog and archive you will see the following codes ("file descriptors" in Solar Orbiter terminology): | |
| - | When browsing the SPICE catalog and archive you will see the following codes ("file descriptors" in Solar Orbiter terminology): | + | |
| * **n-ras**: a raster scan with one of the narrow slits (2", 4" or 6")\\ | * **n-ras**: a raster scan with one of the narrow slits (2", 4" or 6")\\ | ||
| * **w-ras**: a raster scan with the wide slit (30")\\ | * **w-ras**: a raster scan with the wide slit (30")\\ | ||
| Line 38: | Line 37: | ||
| For the "ras" data, the FITS file contains a single raster scan (with ''RASTERNUM=000''). If the raster is repeated, then each repeat goes in a new file (with an incremented ''RASTERNUM''). | For the "ras" data, the FITS file contains a single raster scan (with ''RASTERNUM=000''). If the raster is repeated, then each repeat goes in a new file (with an incremented ''RASTERNUM''). | ||
| - | Full spectrum rasters can be performed in the "exp" mode, but then each exposure will end up in a different FITS file. | + | Full spectrum rasters can be performed in the "exp" mode, but then each exposure will end up in a different FITS file, with a single ''SPIOBSID'' and incrementing ''RASTERNUM''. |
| ==== Software ==== | ==== Software ==== | ||
| Line 59: | Line 58: | ||
| - Run SPICE\_CAT | - Run SPICE\_CAT | ||
| * The catalogue of all files in a specific data release is included with the data release (see below). This version of the catalogue also include additional columns for the list of windows, the file size, and a reference to the dark used for on-ground dark subtraction. | * The catalogue of all files in a specific data release is included with the data release (see below). This version of the catalogue also include additional columns for the list of windows, the file size, and a reference to the dark used for on-ground dark subtraction. | ||
| - | * How to use it in Python: a [[data:data_analysis_manual:read_catalog_python|simple script]] can be used to read it as a Python pandas DataFrame object. The same can be done using the ''Catalog'' object from the [[https://sospice.readthedocs.io/en/stable/|''sospice'' Python module]] (see examples in [[https://git.ias.u-psud.fr/spice/data-analysis-club/-/blob/main/20230627-sospice/sospice-demo.ipynb|this tutorial]]). | + | * How to use it in Python: the catalogue can be read as a ''Catalog'' object from the [[https://sospice.readthedocs.io/en/stable/|''sospice'' Python module]], and then used to find observations with specific characteristics (see examples in [[https://git.ias.u-psud.fr/spice/data-analysis-club/-/blob/main/20230627-sospice/sospice-demo.ipynb|this tutorial]]). |
| * How to use it in IDL: [[data:data_analysis_manual#the_spice_catalog|see below]] | * How to use it in IDL: [[data:data_analysis_manual#the_spice_catalog|see below]] | ||
| Line 66: | Line 65: | ||
| * All L1 and L2 files with their latest versions (at a given date) are collected in a data release | * All L1 and L2 files with their latest versions (at a given date) are collected in a data release | ||
| - | * The releases are provided under https://spice.osups.universite-paris-saclay.fr/spice-data/ | + | * The releases are provided under https://spice.osups.universite-paris-saclay.fr/spice-data/, in subdirectories named ''release-{release_tag}/'' |
| + | * The latest release tag can be obtained from the content of [[https://spice.osups.universite-paris-saclay.fr/spice-data/metadata/latest-release.txt|latest-release.txt]] | ||
| * Released files are also sent to the SOAR (see below) | * Released files are also sent to the SOAR (see below) | ||
| * The latest release is updated with new data as they arrive (and after a superficial validation step). | * The latest release is updated with new data as they arrive (and after a superficial validation step). | ||
| Line 73: | Line 73: | ||
| SOAR: | SOAR: | ||
| - | * [[https://soar.esac.esa.int/|SOAR]] ([[https://issues.cosmos.esa.int/solarorbiterwiki/display/SOSP/SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]], [[https://www.cosmos.esa.int/web/soar/home|SOAR help pages]]) | + | * [[https://soar.esac.esa.int/|SOAR]] ([[https://issues.cosmos.esa.int/solarorbiterwiki/display/SOSP/SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]], [[https://www.cosmos.esa.int/web/soar/home|SOAR help pages]], [[https://www.cosmos.esa.int/web/soar/inventory-plots|SOAR data coverage]]) |
| * SOAR has a TAP interface that can be used for automated queries ([[https://www.cosmos.esa.int/web/soar/tables-views-and-columns|documentation of available tables]]) | * SOAR has a TAP interface that can be used for automated queries ([[https://www.cosmos.esa.int/web/soar/tables-views-and-columns|documentation of available tables]]) | ||
| * The ''sunpy-soar'' [[https://github.com/sunpy/sunpy-soar/|Python module]] uses this TAP interface for queries through SunPy/Fido | * The ''sunpy-soar'' [[https://github.com/sunpy/sunpy-soar/|Python module]] uses this TAP interface for queries through SunPy/Fido | ||
| Line 81: | Line 81: | ||
| * Quicklook archive (to be developed) | * Quicklook archive (to be developed) | ||
| * [[https://github.com/ITA-Solar/solo-spice-ql/wiki/spice_xfiles|Quicklooks procedure in SSW]] | * [[https://github.com/ITA-Solar/solo-spice-ql/wiki/spice_xfiles|Quicklooks procedure in SSW]] | ||
| - | * A quicklook app (in Python) is in development at IAS. | + | * A [[https://git.ias.u-psud.fr/spice/data_quicklook|quicklook app]] (in Python) is in development at IAS. |
| * [[https://git.ias.u-psud.fr/spice/pre-analysis/|"Pre-analysis"]]: already processed quicklook images (per STP) | * [[https://git.ias.u-psud.fr/spice/pre-analysis/|"Pre-analysis"]]: already processed quicklook images (per STP) | ||
| Line 87: | Line 87: | ||
| * Released SPICE files are ingested from SOAR to the VSO ([[https://sdac.virtualsolar.org/cgi/search|search form]]) | * Released SPICE files are ingested from SOAR to the VSO ([[https://sdac.virtualsolar.org/cgi/search|search form]]) | ||
| * Accessible through IDL/SSW and Python/Fido | * Accessible through IDL/SSW and Python/Fido | ||
| + | * FIXME As of 2024-03-29, the SPICE files on VSO are old files from release 3.0 only | ||
| ===== Tips and Advice on Analyzing SPICE Data ===== | ===== Tips and Advice on Analyzing SPICE Data ===== | ||
| Line 96: | Line 97: | ||
| ==== Burn-in Effects ==== | ==== Burn-in Effects ==== | ||
| - | The SPICE detectors have microchannel plates (MCPs). Over time, the sensitivity of MCPs decreases at the locations of strong emission lines. This effect is referred to as "burn-in" and results in a flat-top or "self-reversal" to the line profile. This has already been noticed for the C III 977 line. The SPICE team plans to provide a correction to this effect once it is fully characterized. | + | The SPICE detectors have microchannel plates (MCPs). Over time, the sensitivity of MCPs decreases at the locations of strong emission lines. This effect is referred to as "burn-in" and results in a flat-top or "self-reversal" to the line profile. This has already been noticed for the C III 977 line. From data release 4.0, level-2 files are corrected for burn-in for six lines. The effects of burn-in continue to be monitored, particularly for weaker lines for which a correction is not currently available. |
| + | |||
| + | The lines that are being corrected (as of 2023-11) are: N IV 76.5 nm, Ne VIII 77.0 nm, Ne VIII 78.0 nm, C III 97.7 nm, Lyman-beta 102.6 nm, and O VI 103.2 nm. | ||
| ==== Flare lines ==== | ==== Flare lines ==== | ||
| Line 105: | Line 108: | ||
| To determine if a flare is occurring at the time of a SPICE observation, please check [[https://datacenter.stix.i4ds.net/view/plot/lightcurves|the STIX Data Browser]]. | To determine if a flare is occurring at the time of a SPICE observation, please check [[https://datacenter.stix.i4ds.net/view/plot/lightcurves|the STIX Data Browser]]. | ||
| + | |||
| + | A list of flares observed by SPICE from the beginning of the mission to around August 2023 is available at [[https://spice-wiki.ias.u-psud.fr/doku.php/public:eventlists:flares|https://spice-wiki.ias.u-psud.fr/doku.php/public:eventlists:flares]]. | ||
| ===== Access data ===== | ===== Access data ===== | ||
| Line 222: | Line 227: | ||
| * ''EAR\_TDEL'': light travel time difference between Sun centre to Spacecraft and Sun centre to Earth (seconds) | * ''EAR\_TDEL'': light travel time difference between Sun centre to Spacecraft and Sun centre to Earth (seconds) | ||
| - | The full list of keywords is given in the [[https://spice.ias.u-psud.fr/spice-data/documents/SPICE-UIO-DPDD-0002-1.4-Data_Product_Description_Document.pdf|DPDD]] - search for "Solar ephemeris keywords". | + | The full list of keywords is given in the [[https://spice-wiki.ias.u-psud.fr/lib/exe/fetch.php/public:spice-uio-dpdd-0002-1.8-data_product_description_document_spicefits_.pdf|DPDD]] - search for "Solar ephemeris keywords". |
| Users should make sure to correct the SPICE observing times for EAR\_TDEL in order to compare with observations from Earth-orbiting spacecraft. This is especially important for highly-dynamic structures such as flares or CMEs. | Users should make sure to correct the SPICE observing times for EAR\_TDEL in order to compare with observations from Earth-orbiting spacecraft. This is especially important for highly-dynamic structures such as flares or CMEs. | ||
| Line 244: | Line 249: | ||
| ==== Change of radiometric calibration with time ==== | ==== Change of radiometric calibration with time ==== | ||
| - | The SPICE calibration pipeline applies the same radiometric calibration to all of the SPICE data, irrespective of the time it was taken. However, the SPICE team has found that the instrument sensitivity decreased by a factor 2-3 in the first two years of the mission and has remained constant since then. The SPICE team is characterizing these changes and they will be incorporated into the SPICE pipeline. Until then, users should be careful about comparing emission line radiances at different times during the mission. | + | The SPICE instrument sensitivity has decreased by a factor 2-3 in the first two years of the mission and has remained approximately constant since then. The SPICE team have characterized these changes and, as of 8-Nov-2023, the level-2 SPICE files incorporate the sensitivity degradation. |
| ==== Units and uncertainties ==== | ==== Units and uncertainties ==== | ||
| - | The data cube (returned by the SSW object method ''get\_window\_data'' or by the Python ''astropy'' or ''sunraster'' libraries) of the SPICE level-2 files contains spectral radiances in units of W/m²/s/sr/nm. The wavelength vector has units of nm. To convert a radiance in W/m²/s/sr to erg/cm²/s/sr, it is necessary to multiply by 1000. | + | The data cube (returned by the SSW object method ''get\_window\_data'' or by the Python ''astropy'' or ''sunraster'' libraries) of the SPICE level-2 files contains spectral radiances in units of W/m²/sr/nm. The wavelength vector has units of nm. To convert a radiance in W/m²/sr to erg/cm²/s/sr, it is necessary to multiply by 1000. |
| Uncertainties on the spectral radiances can be obtained with: | Uncertainties on the spectral radiances can be obtained with: | ||
data/data_analysis_manual.1697533173.txt.gz · Last modified: 2023/10/17 10:59 by eric buchlin
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
