This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
data:data_analysis_manual [2021/06/03 10:03] frederic auchere |
data:data_analysis_manual [2022/08/27 10:04] eric buchlin [Find data] link to using the catalog in IDL |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== Data analysis user's manual ====== | ====== Data analysis user's manual ====== | ||
- | ===== Introduction ===== | + | <WRAP center round important 70%> |
This manual is very preliminary, please be patient or help adding what is missing. | This manual is very preliminary, please be patient or help adding what is missing. | ||
+ | </WRAP> | ||
+ | |||
+ | ===== Introduction ===== | ||
==== SPICE instrument and capabilities ==== | ==== SPICE instrument and capabilities ==== | ||
Line 9: | Line 11: | ||
SPICE papers: | SPICE papers: | ||
* Instrument paper: //Astron. Astrophys.//, **642**, A14, 2020. [[https://doi.org/10.1051/0004-6361/201935574|DOI]], [[https://ui.adsabs.harvard.edu/abs/2020A%26A...642A..14S|ADS]] | * Instrument paper: //Astron. Astrophys.//, **642**, A14, 2020. [[https://doi.org/10.1051/0004-6361/201935574|DOI]], [[https://ui.adsabs.harvard.edu/abs/2020A%26A...642A..14S|ADS]] | ||
- | * First results: Submitted | + | * First results: //Astron. Astrophys.//, **656**, A38, 2021. [[https://doi.org/10.1051/0004-6361/202141221|DOI]], [[https://ui.adsabs.harvard.edu/abs/2021A%26A...656A..38F/abstract|ADS]] |
* In-flight calibration: Not available yet | * In-flight calibration: Not available yet | ||
* Other papers can be found in the [[https://ui.adsabs.harvard.edu/public-libraries/rIi8Zx4zTg2stP5nTojx2Q|SPICE ADS library]] | * Other papers can be found in the [[https://ui.adsabs.harvard.edu/public-libraries/rIi8Zx4zTg2stP5nTojx2Q|SPICE ADS library]] | ||
Line 18: | Line 20: | ||
==== Data products ==== | ==== Data products ==== | ||
- | |||
- | DOI for the SPICE data set (to be cited in papers): [[https://doi.org/10.5270/esa-lbmdy7c|doi:10.5270/esa-lbmdy7c]]. | ||
The data levels, FITS files, and headers are described in | The data levels, FITS files, and headers are described in | ||
Line 25: | Line 25: | ||
* The LLDPDD for the Low-Latency (LL) files | * The LLDPDD for the Low-Latency (LL) files | ||
+ | The full SPICE data set (to be cited in papers) has a DOI: [[https://doi.org/10.5270/esa-lbmdy7c|doi:10.5270/esa-lbmdy7c]]. DOIs will also be attributed to each data release. | ||
+ | |||
+ | When browsing the SPICE catalog and archive you will see the following codes: | ||
+ | |||
+ | **n-ras**: a raster scan with one of the narrow slits (2", 4" or 6")\\ | ||
+ | **w-ras**: a raster scan with the wide slit (30")\\ | ||
+ | **n-sit**: a sit-and-stare study with one of the narrow slits\\ | ||
+ | **w-sit**: a sit-and-stare study with the wide slit\\ | ||
+ | **n-exp**: a single exposure with a narrow slit that yields the entire spectrum\\ | ||
+ | **w-exp**: a single exposure with the wide slit that yields the entire spectrum | ||
+ | |||
+ | For the "ras" data, the FITS file contains a single raster scan. If the raster is repeated, then each repeat goes in a new file. | ||
+ | |||
+ | Rasters can be performed in the "exp" mode, but each exposure will end up in a different FITS file. | ||
===== Find data ===== | ===== Find data ===== | ||
UiO catalogue: | UiO catalogue: | ||
- | * can be generated with SSW | + | * Can be generated with IDL/SolarSoft (SSW) from a local data tree: |
- | * accessible [[http://astro-sdc-db.uio.no/vol/spice/fits/spice_catalog.txt|through the UiO website]] | + | - Make sure ''$SPICE\_DATA'' points to the data tree you want to have catalogued |
- | * how to use it in SSW: see [[https://pyoung.org/spice/analysis_guide.html|Peter Young's analysis guide]] and [[https://github.com/ITA-Solar/solo-spice-ql/wiki#file-management-utilities|UiO's wiki]] | + | - Run SPICE\_GEN\_CAT. The catalogue file will be written to the data tree root directory (you need to have write access) |
- | * how to use it in [[spice_internal:data:read_catalog_python|Python]] | + | - Run SPICE\_CAT |
+ | * The catalogue file generated from all SPICE files is accessible [[http://astro-sdc-db.uio.no/vol/spice/fits/spice_catalog.txt|through the UiO website]] (restricted to SPICE team) | ||
+ | * How to use it in [[data:data_analysis_manual:read_catalog_python|Python]] | ||
+ | * How to use it in IDL: [[data:data_analysis_manual#the_spice_catalog|see below]] | ||
+ | * A CSV export of the part of the catalogue corresponding to each release is provided with the release. | ||
SOAR (and institute mirrors): | SOAR (and institute mirrors): | ||
* [[https://soar.esac.esa.int/|SOAR]] ([[https://issues.cosmos.esa.int/solarorbiterwiki/display/SOSP/SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]]) | * [[https://soar.esac.esa.int/|SOAR]] ([[https://issues.cosmos.esa.int/solarorbiterwiki/display/SOSP/SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]]) | ||
- | |||
Quicklooks: | Quicklooks: | ||
- | * Quicklook archive | + | * Quicklook archive (to be developed) |
* [[https://github.com/ITA-Solar/solo-spice-ql/wiki/spice_xfiles|Quicklooks in SSW]] | * [[https://github.com/ITA-Solar/solo-spice-ql/wiki/spice_xfiles|Quicklooks in SSW]] | ||
+ | In the future: VSO. VSO through SSW or sunpy. Sunpy/Fido | ||
- | VSO. VSO through SSW or sunpy. Sunpy/Fido | + | ===== Tips and Advice on Analyzing SPICE Data ===== |
- | Internal (IAS) (link) | + | ==== Doppler Maps ==== |
+ | WARNING: Doppler maps created from SPICE lines show unusual patterns that we believe are due to the point spread function of the instrument. Work is underway on a deconvolution procedure that will correct for this effect. If you need advice on features you see in a Doppler map, please contact the SPICE PI at frederic.auchere -at- universite-paris-saclay.fr or another member of the SPICE team. | ||
+ | |||
+ | ==== Burn-in Effects ==== | ||
+ | |||
+ | The SPICE detectors have microchannel plates (MCPs). Over time, the sensitivity of MCPs decreases at the locations of strong emission lines. This effect is referred to as "burn-in" and results in a flat-top or "self-reversal" to the line profile. This has already been noticed for the C III 977 line. The SPICE team plans to provide a correction to this effect once it is fully characterized. | ||
+ | |||
+ | ==== Flare lines ==== | ||
+ | |||
+ | The two key flare lines for EIS are Fe XVIII 974.86 and Fe XX 721.56 that are formed around 7 MK and 10 MK, respectively. The Fe XVIII line will probably be seen in active region cores (at least from bright active regions). | ||
+ | |||
+ | The Fe XVIII line is located between the stronger H I 972.54 (Lyman-gamma) and C III 977.02, but measurements should be possible. | ||
+ | |||
+ | The Fe XX line lies close to Fe VIII 721.26. One can estimate the strength of this line by comparing with Mg IX 706.06. The latter will be about a factor 10 stronger. | ||
+ | |||
+ | To determine if a flare is occurring at the time of a SPICE observation, check the STIX Data Browser at: | ||
+ | |||
+ | [[https://datacenter.stix.i4ds.net/view/plot/lightcurves | ||
+ | ]] | ||
===== Access data ===== | ===== Access data ===== | ||
SPICE team internal access: | SPICE team internal access: | ||
- | * UiO website (SPICE internal use). | + | * [[http://astro-sdc-db.uio.no/vol/spice/|UiO website]] (restricted). |
Once the data will be released: | Once the data will be released: | ||
- | * SOAR (and institute mirrors), once the data will be released | + | * [[https://soar.esac.esa.int/|SOAR]] (and institute mirrors) |
+ | * [[https://spice.ias.u-psud.fr/data/archives|List of data releases]]. | ||
* VSO. VSO through SSW or sunpy. Sunpy/Fido | * VSO. VSO through SSW or sunpy. Sunpy/Fido | ||
- | ===== Read an display data ===== | + | ===== Read and display data (IDL) ===== |
+ | |||
+ | External links: | ||
+ | * Oslo's quicklook tools [[https://github.com/ITA-Solar/solo-spice-ql/wiki|spice_data, spice_object...]] | ||
+ | * Peter Young's [[https://pyoung.org/spice/analysis_guide.html|SPICE Analysis Guide]] | ||
+ | |||
+ | |||
+ | ==== Environment variables ==== | ||
+ | |||
+ | Choose a location to store your SPICE data (e.g., '/my_data/spice') and then point the environment variable ''$SPICE\_DATA'' to it: | ||
+ | |||
+ | ''setenv,'SPICE\_DATA=/my\_data/spice' | ||
+ | '' | ||
+ | |||
+ | This line should be added to your IDL\_STARTUP file. | ||
+ | |||
+ | Data are organized under ''$SPICE\_DATA'' with a year/month/day subdirectory structure. See the "Ingesting downloaded data" section below. | ||
+ | |||
+ | |||
+ | ==== The SPICE catalog ==== | ||
+ | |||
+ | The SPICE catalog can be accessed by doing: | ||
+ | |||
+ | ''IDL> spice\_cat'' | ||
+ | |||
+ | Use the "SPICE\_GEN\_CAT" button to make sure you have the most up-to-date list. | ||
+ | |||
+ | ==== Reference data-sets ==== | ||
+ | |||
+ | Perhaps the most useful data from the early commissioning phase (before July 2020) are the raster scans on 28-May-2020. | ||
+ | |||
+ | 16:05, 16:50 - disk center rasters with 20s exposures \\ | ||
+ | 17:50, 18:35 - north limb rasters with 20s exposures\\ | ||
+ | 19:35, 20:20 - south limb rasters with 20s exposures\\ | ||
+ | 21:20, 22:05 - west limb rasters with 20s exposures\\ | ||
+ | 23:05, 23:47 - east limb rasters with 20s exposures | ||
+ | |||
+ | The first true science observations were obtained during 18 to 22 November 2020. For example, an active region can be seen in the raster beginning 19:57 UT on 18-Nov-2020. | ||
+ | |||
+ | ==== Ingesting downloaded data ==== | ||
+ | |||
+ | After you have downloaded some SPICE FITS files, you can ingest them into your data directory with spice\_ingest: | ||
+ | |||
+ | ''IDL> spice\_ingest, files'' | ||
+ | |||
+ | This routine automatically creates the sub-directory structure (year/month/day) within $SPICE\_DATA for the files. | ||
+ | |||
+ | ==== Finding and reading a FITS file ==== | ||
+ | |||
+ | Once a file has been ingested, then you can find it with spice\_find\_file using the observation time: | ||
+ | |||
+ | ''IDL> file=spice\_find\_file('28-may-2020 16:05')'' | ||
+ | |||
+ | A file can be read into an IDL object with: | ||
+ | |||
+ | ''IDL> d=spice\_object(file)'' | ||
+ | |||
+ | ==== Extracting information from the data object ==== | ||
+ | |||
+ | The table below gives some methods for extracting information out of the data object. Where "i" is given, it means the index of a wavelength window should be specified (indices begin at 0). | ||
+ | |||
+ | You can get a list of all methods by doing: | ||
+ | |||
+ | IDL> d->help | ||
+ | |||
+ | ^ Command ^ Function ^ | ||
+ | | d->get\_number\_windows() | No. of spectral windows | | ||
+ | | d->get\_window\_data(i,/load) | Extract a data window | | ||
+ | | d->get\_lambda\_vector(i) | Get wavelength vector for the window | | ||
+ | | d->get\_header(i) | Extract a data window header | | ||
+ | | d->get\_window\_id(i) | Get the data window ID | | ||
+ | | d->get\_start\_time() | Start time of observation | | ||
+ | | d->get\_end\_time() | End time of observation | | ||
+ | | d->get\_sit\_and\_stare() | Set to 1 if sit-and-stare observation | | ||
+ | | d->get\_number\_exposures() | Number of exposures in raster | | ||
+ | | d->get\_xcen(i) | Get the X-center for the window | | ||
+ | | d->get\_ycen(i) | Get the Y-center for the window | | ||
+ | |||
+ | |||
+ | ==== Quicklook tools ==== | ||
+ | |||
+ | A set of five widget-based tools are available for browsing SPICE data, and these can be accessed through spice\_xfiles: | ||
+ | |||
+ | '' IDL> spice\_xfiles'' | ||
+ | |||
+ | This allows you to select a FITS file from your SPICE data directory. A new widget appears from which you can then select one of the five quicklook tools: Detector, Raster Browser, Raster, Whisker and Intensity map. These tools mimic software that were available for EIS and IRIS. | ||
+ | |||
+ | For further details, please visit the [[https://github.com/ITA-Solar/solo-spice-ql/wiki|SPICE Quicklook and Data Analysis Software Page]]. | ||
+ | |||
+ | === Raster Browser === | ||
+ | |||
+ | This is useful for browsing the 3D data cubes from SPICE rasters. In addition to being called from spice\_xfiles (see above), it can also be called directly from the command line: | ||
+ | |||
+ | ''IDL> spice\_raster\_browser,file'' | ||
+ | |||
+ | Use a 3-button mouse to browse the images and spectra: the middle button allows you to select a new pixel, the left button zooms in and the right button zooms out. | ||
+ | |||
+ | |||
+ | ===== Ephemeris information ===== | ||
+ | |||
+ | You can access ephemeris information from the SPICE headers. For example, the keywords: | ||
+ | |||
+ | DSUN\_AU - distance of the spacecraft from the Sun in AU\\ | ||
+ | EAR\_TDEL - light travel time from the spacecraft to Earth (seconds) | ||
+ | |||
+ | The full list of keywords is given in the [[https://spice.ias.u-psud.fr/spice-data/documents/SPICE-UIO-DPDD-0002-1.4-Data_Product_Description_Document.pdf|DPDD]] - search for "Solar ephemeris keywords". | ||
+ | |||
+ | Users should make sure to correct the SPICE observing times for EAR\_TDEL in order to compare with observations from Earth-orbiting spacecraft. This is especially important for highly-dynamic structures such as flares or CMEs. | ||
- | IDL | + | ===== Read and display data (Python) ===== |
- | * [[https://github.com/ITA-Solar/solo-spice-ql/wiki|spice_data, spice_object...]] | + | |
- | Python | + | Links: |
- | * [[https://docs.sunpy.org/projects/sunraster/|sunraster]]. | + | * [[:data:data_analysis_manual:sunraster|with sunraster]]. |