User Tools

Site Tools


data:data_analysis_manual

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
data:data_analysis_manual [2021/06/03 10:03]
frederic auchere
data:data_analysis_manual [2022/08/27 10:45] (current)
eric buchlin [Find data] Releases, SOAR TAP and sunpy-soar, pre-analysis
Line 1: Line 1:
 ====== Data analysis user's manual ====== ====== Data analysis user's manual ======
  
-===== Introduction ===== +<WRAP center round important 70%>
 This manual is very preliminary,​ please be patient or help adding what is missing. This manual is very preliminary,​ please be patient or help adding what is missing.
 +</​WRAP>​
 +
 +===== Introduction =====
  
 ==== SPICE instrument and capabilities ==== ==== SPICE instrument and capabilities ====
Line 9: Line 11:
 SPICE papers: SPICE papers:
   * Instrument paper: //Astron. Astrophys.//,​ **642**, A14, 2020. [[https://​doi.org/​10.1051/​0004-6361/​201935574|DOI]],​ [[https://​ui.adsabs.harvard.edu/​abs/​2020A%26A...642A..14S|ADS]]   * Instrument paper: //Astron. Astrophys.//,​ **642**, A14, 2020. [[https://​doi.org/​10.1051/​0004-6361/​201935574|DOI]],​ [[https://​ui.adsabs.harvard.edu/​abs/​2020A%26A...642A..14S|ADS]]
-  * First results: ​Submitted+  * First results: ​//Astron. Astrophys.//,​ **656**, A38, 2021. [[https://​doi.org/​10.1051/​0004-6361/​202141221|DOI]],​ [[https://​ui.adsabs.harvard.edu/​abs/​2021A%26A...656A..38F/​abstract|ADS]]
   * In-flight calibration:​ Not available yet   * In-flight calibration:​ Not available yet
   * Other papers can be found in the [[https://​ui.adsabs.harvard.edu/​public-libraries/​rIi8Zx4zTg2stP5nTojx2Q|SPICE ADS library]]   * Other papers can be found in the [[https://​ui.adsabs.harvard.edu/​public-libraries/​rIi8Zx4zTg2stP5nTojx2Q|SPICE ADS library]]
Line 18: Line 20:
  
 ==== Data products ==== ==== Data products ====
- 
-DOI for the SPICE data set (to be cited in papers): [[https://​doi.org/​10.5270/​esa-lbmdy7c|doi:​10.5270/​esa-lbmdy7c]]. 
  
 The data levels, FITS files, and headers are described in The data levels, FITS files, and headers are described in
Line 25: Line 25:
   * The LLDPDD for the Low-Latency (LL) files   * The LLDPDD for the Low-Latency (LL) files
  
 +The full SPICE data set (to be cited in papers) has a DOI: [[https://​doi.org/​10.5270/​esa-lbmdy7c|doi:​10.5270/​esa-lbmdy7c]]. DOIs will also be attributed to each data release.
  
 +
 +When browsing the SPICE catalog and archive you will see the following codes:
 +
 +**n-ras**: a raster scan with one of the narrow slits (2", 4" or 6")\\
 +**w-ras**: a raster scan with the wide slit (30"​)\\
 +**n-sit**: a sit-and-stare study with one of the narrow slits\\
 +**w-sit**: a sit-and-stare study with the wide slit\\
 +**n-exp**: a single exposure with a narrow slit that yields the entire spectrum\\
 +**w-exp**: a single exposure with the wide slit that yields the entire spectrum
 +
 +For the "​ras"​ data, the FITS file contains a single raster scan. If the raster is repeated, then each repeat goes in a new file.
 +
 +Rasters can be performed in the "​exp"​ mode, but each exposure will end up in a different FITS file.
 ===== Find data ===== ===== Find data =====
  
 UiO catalogue: UiO catalogue:
-  * can be generated with SSW +  * Can be generated with IDL/​SolarSoft (SSW) from a local data tree: 
-  * accessible [[http://​astro-sdc-db.uio.no/​vol/​spice/​fits/​spice_catalog.txt|through the UiO website]] +    - Make sure ''​$SPICE\_DATA''​ points to the data tree you want to have catalogued 
-  * how to use it in SSW: see [[https://​pyoung.org/​spice/​analysis_guide.html|Peter Young'​s analysis guide]] and [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki#​file-management-utilities|UiO's wiki]] +    - Run SPICE\_GEN\_CAT. The catalogue file will be written to the data tree root directory (you need to have write access) 
-  * how to use it in [[spice_internal:​data:read_catalog_python|Python]]+    - Run SPICE\_CAT 
 +  * The catalogue file generated from all SPICE files is accessible [[http://​astro-sdc-db.uio.no/​vol/​spice/​fits/​spice_catalog.csv|through the UiO website]] ​(restricted to SPICE team) 
 +  * How to use it in [[data:data_analysis_manual:read_catalog_python|Python]] 
 +  * How to use it in IDL: [[data:data_analysis_manual#​the_spice_catalog|see below]] 
 +  * A CSV export of the part of the catalogue corresponding to each release is provided with the data releases.
  
-SOAR (and institute mirrors): +Data releases:
-  * [[https://​soar.esac.esa.int/​|SOAR]] ([[https://​issues.cosmos.esa.int/​solarorbiterwiki/​display/​SOSP/​SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]])+
  
 +  * All L1+ files with their latest versions (at a given date) are collected in a data release
 +  * The releases are provided under https://​spice.osups.universite-paris-saclay.fr/​spice-data/​
 +  * The latest release is updated with new data as they arrive (and after a superficial validation step).
 +  * Each release includes release notes, and a CSV catalog of the files in the release.
 +  * The SPICE data used in a publication have to be cited properly. How to do this is explained in the release notes.
 +
 +SOAR:
 +  * [[https://​soar.esac.esa.int/​|SOAR]] ([[https://​issues.cosmos.esa.int/​solarorbiterwiki/​display/​SOSP/​SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]])
 +  * SOAR has a TAP interface that can be used for automated queries
 +  * The ''​sunpy-soar''​ Python module uses this TAP interface for queries through SunPy/Fido
 +    * Only queries by instrument / time / level / product [[https://​github.com/​sunpy/​sunpy-soar/​issues/​46|at the moment]]
  
 Quicklooks: Quicklooks:
-  * Quicklook archive+  * Quicklook archive ​(to be developed)
   * [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki/​spice_xfiles|Quicklooks in SSW]]   * [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki/​spice_xfiles|Quicklooks in SSW]]
 +  * [[https://​git.ias.u-psud.fr/​spice/​pre-analysis/​|Pre-analysis]] (per STP)
  
 +VSO:
 +  * (in the near future)
 +  * Accessible through IDL/SSW and Python/Fido
  
-VSO. VSO through SSW or sunpy. Sunpy/Fido+===== Tips and Advice on Analyzing SPICE Data =====
  
-Internal (IAS) (link)+==== Doppler Maps ====
  
 +WARNING: Doppler maps created from SPICE lines show unusual patterns that we believe are due to the point spread function of the instrument. Work is underway on a deconvolution procedure that will correct for this effect. If you need advice on features you see in a Doppler map, please contact the SPICE PI at frederic.auchere -at- universite-paris-saclay.fr or another member of the SPICE team.
 +
 +==== Burn-in Effects ====
 +
 +The SPICE detectors have microchannel plates (MCPs). Over time, the sensitivity of MCPs decreases at the locations of strong emission lines. This effect is referred to as "​burn-in"​ and results in a flat-top or "​self-reversal"​ to the line profile. This has already been noticed for the C III 977 line. The SPICE team plans to provide a correction to this effect once it is fully characterized.
 +
 +==== Flare lines ====
 +
 +The two key flare lines for EIS are Fe XVIII 974.86 and Fe XX 721.56 that are formed around 7 MK and 10 MK, respectively. The Fe XVIII line will probably be seen in active region cores (at least from bright active regions).
 +
 +The Fe XVIII line is located between the stronger H I 972.54 (Lyman-gamma) and C III 977.02, but measurements should be possible.
 +
 +The Fe XX line lies close to Fe VIII 721.26. One can estimate the strength of this line by comparing with Mg IX 706.06. The latter will be about a factor 10 stronger.
 +
 +To determine if a flare is occurring at the time of a SPICE observation,​ check the STIX Data Browser at:
 +
 +[[https://​datacenter.stix.i4ds.net/​view/​plot/​lightcurves
 +]]
 ===== Access data ===== ===== Access data =====
  
 SPICE team internal access: SPICE team internal access:
-  * UiO website (SPICE internal use).+  * [[http://​astro-sdc-db.uio.no/​vol/​spice/​|UiO website]] (restricted).
  
 Once the data will be released: Once the data will be released:
-  * SOAR (and institute mirrors), once the data will be released+  * [[https://​soar.esac.esa.int/​|SOAR]] (and institute mirrors) 
 +  * [[https://​spice.ias.u-psud.fr/​data/​archives|List of data releases]].
   * VSO. VSO through SSW or sunpy. Sunpy/Fido   * VSO. VSO through SSW or sunpy. Sunpy/Fido
  
  
-===== Read an display data =====+===== Read and display data (IDL) =====
  
-IDL+External links:
   * Oslo's quicklook tools [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki|spice_data,​ spice_object...]]   * Oslo's quicklook tools [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki|spice_data,​ spice_object...]]
 +  * Peter Young'​s [[https://​pyoung.org/​spice/​analysis_guide.html|SPICE Analysis Guide]]
 +
 +
 +==== Environment variables ====
 +
 +Choose a location to store your SPICE data (e.g., '/​my_data/​spice'​) and then point the environment variable ''​$SPICE\_DATA''​ to it:
 +
 +''​setenv,'​SPICE\_DATA=/​my\_data/​spice'​
 +''​
 +        ​
 +This line should be added to your IDL\_STARTUP file.
 +
 +Data are organized under ''​$SPICE\_DATA''​ with a year/​month/​day subdirectory structure. See the "​Ingesting downloaded data" section below.
 +
 +
 +==== The SPICE catalog ====
 +
 +The SPICE catalog can be accessed by doing:
 +
 +''​IDL>​ spice\_cat''​
 +
 +Use the "​SPICE\_GEN\_CAT"​ button to make sure you have the most up-to-date list.
 +
 +==== Reference data-sets ====
 +
 +Perhaps the most useful data from the early commissioning phase (before July 2020) are the raster scans on 28-May-2020.
 +
 +16:05, 16:50 - disk center rasters with 20s exposures \\
 +17:50, 18:35 - north limb rasters with 20s exposures\\
 +19:35, 20:20 - south limb rasters with 20s exposures\\
 +21:20, 22:05 - west limb rasters with 20s exposures\\
 +23:05, 23:47 - east limb rasters with 20s exposures
 +
 +The first true science observations were obtained during 18 to 22 November 2020. For example, an active region can be seen in the raster beginning 19:57 UT on 18-Nov-2020.
 +
 +==== Ingesting downloaded data ====
 +
 +After you have downloaded some SPICE FITS files, you can ingest them into your data directory with spice\_ingest:​
 +
 +''​IDL>​ spice\_ingest,​ files''​
 +
 +This routine automatically creates the sub-directory structure (year/​month/​day) within $SPICE\_DATA for the files.
 +
 +==== Finding and reading a FITS file ====
 +
 +Once a file has been ingested, then you can find it with spice\_find\_file using the observation time:
 +
 +''​IDL>​ file=spice\_find\_file('​28-may-2020 16:​05'​)''​
 +
 +A file can be read into an IDL object with:
 +
 +''​IDL>​ d=spice\_object(file)''​
 +
 +==== Extracting information from the data object ====
 +
 +The table below gives some methods for extracting information out of the data object. Where "​i"​ is given, it means the index of a wavelength window should be specified (indices begin at 0).
 +
 +You can get a list of all methods by doing:
 +
 +IDL> d->help
 +
 +^ Command ​                       ^ Function ​                              ^
 +| d->​get\_number\_windows() ​     | No. of spectral windows ​               |
 +| d->​get\_window\_data(i,/​load) ​ | Extract a data window ​                 |
 +| d->​get\_lambda\_vector(i) ​     | Get wavelength vector for the window ​                 |
 +| d->​get\_header(i) ​             | Extract a data window header ​          |
 +| d->​get\_window\_id(i) ​         | Get the data window ID                 |
 +| d->​get\_start\_time() ​         | Start time of observation ​             |
 +| d->​get\_end\_time() ​           | End time of observation ​               |
 +| d->​get\_sit\_and\_stare() ​     | Set to 1 if sit-and-stare observation ​ |
 +| d->​get\_number\_exposures() ​   | Number of exposures in raster ​         |
 +| d->​get\_xcen(i) ​               | Get the X-center for the window ​       |
 +| d->​get\_ycen(i) ​               | Get the Y-center for the window ​       |
 +
 +
 +==== Quicklook tools ====
 +
 +A set of five widget-based tools are available for browsing SPICE data, and these can be accessed through spice\_xfiles:​
 +
 +''​ IDL> spice\_xfiles''​
 +
 +This allows you to select a FITS file from your SPICE data directory. A new widget appears from which you can then select one of the five quicklook tools: Detector, Raster Browser, Raster, Whisker and Intensity map. These tools mimic software that were available for EIS and IRIS.
 +
 +For further details, please visit the [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki|SPICE Quicklook and Data Analysis Software Page]].
 +
 +=== Raster Browser ===
 +
 +This is useful for browsing the 3D data cubes from SPICE rasters. In addition to being called from spice\_xfiles (see above), it can also be called directly from the command line:
 +
 +''​IDL>​ spice\_raster\_browser,​file''​
 +
 +Use a 3-button mouse to browse the images and spectra: the middle button allows you to select a new pixel, the left button zooms in and the right button zooms out.
 +
 +
 +===== Ephemeris information =====
 +
 +You can access ephemeris information from the SPICE headers. For example, the keywords:
 +
 +DSUN\_AU - distance of the spacecraft from the Sun in AU\\
 +EAR\_TDEL - light travel time from the spacecraft to Earth (seconds)
 +
 +The full list of keywords is given in the [[https://​spice.ias.u-psud.fr/​spice-data/​documents/​SPICE-UIO-DPDD-0002-1.4-Data_Product_Description_Document.pdf|DPDD]] - search for "Solar ephemeris keywords"​.
 +
 +Users should make sure to correct the SPICE observing times for EAR\_TDEL in order to compare with observations from Earth-orbiting spacecraft. This is especially important for highly-dynamic structures such as flares or CMEs.
 +
 +
 +===== Read and display data (Python) =====
  
-Python +Links: 
-  * [[https://​docs.sunpy.org/​projects/​sunraster/|sunraster]].+  * [[:data:​data_analysis_manual:​sunraster|with sunraster]].
  
  
data/data_analysis_manual.1622707411.txt.gz · Last modified: 2021/06/03 10:03 by frederic auchere