User Tools

Site Tools


data:data_analysis_manual

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
data:data_analysis_manual [2021/06/30 19:04]
Peter Young [Reference data-sets]
data:data_analysis_manual [2022/08/27 10:45] (current)
eric buchlin [Find data] Releases, SOAR TAP and sunpy-soar, pre-analysis
Line 11: Line 11:
 SPICE papers: SPICE papers:
   * Instrument paper: //Astron. Astrophys.//,​ **642**, A14, 2020. [[https://​doi.org/​10.1051/​0004-6361/​201935574|DOI]],​ [[https://​ui.adsabs.harvard.edu/​abs/​2020A%26A...642A..14S|ADS]]   * Instrument paper: //Astron. Astrophys.//,​ **642**, A14, 2020. [[https://​doi.org/​10.1051/​0004-6361/​201935574|DOI]],​ [[https://​ui.adsabs.harvard.edu/​abs/​2020A%26A...642A..14S|ADS]]
-  * First results: ​Submitted+  * First results: ​//Astron. Astrophys.//,​ **656**, A38, 2021. [[https://​doi.org/​10.1051/​0004-6361/​202141221|DOI]],​ [[https://​ui.adsabs.harvard.edu/​abs/​2021A%26A...656A..38F/​abstract|ADS]]
   * In-flight calibration:​ Not available yet   * In-flight calibration:​ Not available yet
   * Other papers can be found in the [[https://​ui.adsabs.harvard.edu/​public-libraries/​rIi8Zx4zTg2stP5nTojx2Q|SPICE ADS library]]   * Other papers can be found in the [[https://​ui.adsabs.harvard.edu/​public-libraries/​rIi8Zx4zTg2stP5nTojx2Q|SPICE ADS library]]
Line 27: Line 27:
 The full SPICE data set (to be cited in papers) has a DOI: [[https://​doi.org/​10.5270/​esa-lbmdy7c|doi:​10.5270/​esa-lbmdy7c]]. DOIs will also be attributed to each data release. The full SPICE data set (to be cited in papers) has a DOI: [[https://​doi.org/​10.5270/​esa-lbmdy7c|doi:​10.5270/​esa-lbmdy7c]]. DOIs will also be attributed to each data release.
  
 +
 +When browsing the SPICE catalog and archive you will see the following codes:
 +
 +**n-ras**: a raster scan with one of the narrow slits (2", 4" or 6")\\
 +**w-ras**: a raster scan with the wide slit (30"​)\\
 +**n-sit**: a sit-and-stare study with one of the narrow slits\\
 +**w-sit**: a sit-and-stare study with the wide slit\\
 +**n-exp**: a single exposure with a narrow slit that yields the entire spectrum\\
 +**w-exp**: a single exposure with the wide slit that yields the entire spectrum
 +
 +For the "​ras"​ data, the FITS file contains a single raster scan. If the raster is repeated, then each repeat goes in a new file.
 +
 +Rasters can be performed in the "​exp"​ mode, but each exposure will end up in a different FITS file.
 ===== Find data ===== ===== Find data =====
  
 UiO catalogue: UiO catalogue:
   * Can be generated with IDL/​SolarSoft (SSW) from a local data tree:   * Can be generated with IDL/​SolarSoft (SSW) from a local data tree:
-    - Make sure ''​$SPICE_DATA''​ points to the data tree you want to have catalogued +    - Make sure ''​$SPICE\_DATA''​ points to the data tree you want to have catalogued 
-    - Run SPICE_GEN_CAT. The catalogue file will be written to the data tree root directory (you need to have write access) +    - Run SPICE\_GEN\_CAT. The catalogue file will be written to the data tree root directory (you need to have write access) 
-    - Run SPICE_CAT +    - Run SPICE\_CAT 
-  * The catalogue file generated from all SPICE files is accessible [[http://​astro-sdc-db.uio.no/​vol/​spice/​fits/​spice_catalog.txt|through the UiO website]] (restricted to SPICE team)+  * The catalogue file generated from all SPICE files is accessible [[http://​astro-sdc-db.uio.no/​vol/​spice/​fits/​spice_catalog.csv|through the UiO website]] (restricted to SPICE team)
   * How to use it in [[data:​data_analysis_manual:​read_catalog_python|Python]]   * How to use it in [[data:​data_analysis_manual:​read_catalog_python|Python]]
-  * A CSV export of the part of the catalogue corresponding to each release is provided with the release.+  ​* How to use it in IDL: [[data:​data_analysis_manual#​the_spice_catalog|see below]] 
 +  ​* A CSV export of the part of the catalogue corresponding to each release is provided with the data releases. 
 + 
 +Data releases: 
 + 
 +  * All L1+ files with their latest versions (at a given date) are collected in a data release 
 +  * The releases are provided under https://​spice.osups.universite-paris-saclay.fr/​spice-data/​ 
 +  * The latest release is updated with new data as they arrive (and after a superficial validation step). 
 +  * Each release includes release notes, and a CSV catalog of the files in the release. 
 +  * The SPICE data used in a publication have to be cited properly. How to do this is explained in the release ​notes.
  
-SOAR (and institute mirrors):+SOAR:
   * [[https://​soar.esac.esa.int/​|SOAR]] ([[https://​issues.cosmos.esa.int/​solarorbiterwiki/​display/​SOSP/​SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]])   * [[https://​soar.esac.esa.int/​|SOAR]] ([[https://​issues.cosmos.esa.int/​solarorbiterwiki/​display/​SOSP/​SOAR+(Solar+Orbiter+Archive)+-+How+to+Use|how to use it]])
 +  * SOAR has a TAP interface that can be used for automated queries
 +  * The ''​sunpy-soar''​ Python module uses this TAP interface for queries through SunPy/Fido
 +    * Only queries by instrument / time / level / product [[https://​github.com/​sunpy/​sunpy-soar/​issues/​46|at the moment]]
  
 Quicklooks: Quicklooks:
   * Quicklook archive (to be developed)   * Quicklook archive (to be developed)
   * [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki/​spice_xfiles|Quicklooks in SSW]]   * [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki/​spice_xfiles|Quicklooks in SSW]]
 +  * [[https://​git.ias.u-psud.fr/​spice/​pre-analysis/​|Pre-analysis]] (per STP)
  
-In the future: VSO. VSO through SSW or sunpy. Sunpy/Fido+VSO: 
 +  * (in the near future
 +  * Accessible ​through ​IDL/SSW and Python/Fido
  
 +===== Tips and Advice on Analyzing SPICE Data =====
 +
 +==== Doppler Maps ====
 +
 +WARNING: Doppler maps created from SPICE lines show unusual patterns that we believe are due to the point spread function of the instrument. Work is underway on a deconvolution procedure that will correct for this effect. If you need advice on features you see in a Doppler map, please contact the SPICE PI at frederic.auchere -at- universite-paris-saclay.fr or another member of the SPICE team.
 +
 +==== Burn-in Effects ====
 +
 +The SPICE detectors have microchannel plates (MCPs). Over time, the sensitivity of MCPs decreases at the locations of strong emission lines. This effect is referred to as "​burn-in"​ and results in a flat-top or "​self-reversal"​ to the line profile. This has already been noticed for the C III 977 line. The SPICE team plans to provide a correction to this effect once it is fully characterized.
 +
 +==== Flare lines ====
 +
 +The two key flare lines for EIS are Fe XVIII 974.86 and Fe XX 721.56 that are formed around 7 MK and 10 MK, respectively. The Fe XVIII line will probably be seen in active region cores (at least from bright active regions).
 +
 +The Fe XVIII line is located between the stronger H I 972.54 (Lyman-gamma) and C III 977.02, but measurements should be possible.
 +
 +The Fe XX line lies close to Fe VIII 721.26. One can estimate the strength of this line by comparing with Mg IX 706.06. The latter will be about a factor 10 stronger.
 +
 +To determine if a flare is occurring at the time of a SPICE observation,​ check the STIX Data Browser at:
 +
 +[[https://​datacenter.stix.i4ds.net/​view/​plot/​lightcurves
 +]]
 ===== Access data ===== ===== Access data =====
  
Line 67: Line 117:
 ==== Environment variables ==== ==== Environment variables ====
  
-Choose a location to store your SPICE data (e.g., '/​my_data/​spice'​) and then point the environment variable ''​$SPICE_DATA''​ to it:+Choose a location to store your SPICE data (e.g., '/​my_data/​spice'​) and then point the environment variable ''​$SPICE\_DATA''​ to it:
  
-''​setenv,'​SPICE_DATA=/my_data/​spice'​+''​setenv,'​SPICE\_DATA=/my\_data/​spice'​
 ''​ ''​
         ​         ​
-This line should be added to your IDL_STARTUP ​file.+This line should be added to your IDL\_STARTUP ​file.
  
-Data are organized under ''​$SPICE_DATA''​ with a year/​month/​day subdirectory structure. See the "​Ingesting downloaded data" section below.+Data are organized under ''​$SPICE\_DATA''​ with a year/​month/​day subdirectory structure. See the "​Ingesting downloaded data" section below.
  
  
Line 81: Line 131:
 The SPICE catalog can be accessed by doing: The SPICE catalog can be accessed by doing:
  
-''​IDL> ​spice_cat''​+''​IDL> ​spice\_cat''​
  
-Use the "SPICE_GEN_CAT" button to make sure you have the most up-to-date list.+Use the "SPICE\_GEN\_CAT" button to make sure you have the most up-to-date list.
  
 ==== Reference data-sets ==== ==== Reference data-sets ====
Line 99: Line 149:
 ==== Ingesting downloaded data ==== ==== Ingesting downloaded data ====
  
-After you have downloaded some SPICE FITS files, you can ingest them into your data directory with spice_ingest:+After you have downloaded some SPICE FITS files, you can ingest them into your data directory with spice\_ingest:
  
-''​IDL> ​spice_ingest, files''​+''​IDL> ​spice\_ingest, files''​
  
-This routine automatically creates the sub-directory structure (year/​month/​day) within $SPICE_DATA ​for the files.+This routine automatically creates the sub-directory structure (year/​month/​day) within $SPICE\_DATA ​for the files.
  
 ==== Finding and reading a FITS file ==== ==== Finding and reading a FITS file ====
  
-Once a file has been ingested, then you can find it with spice_find_file ​using the observation time:+Once a file has been ingested, then you can find it with spice\_find\_file ​using the observation time:
  
-''​IDL>​ file=spice_find_file('​28-may-2020 16:​05'​)''​+''​IDL>​ file=spice\_find\_file('​28-may-2020 16:​05'​)''​
  
 A file can be read into an IDL object with: A file can be read into an IDL object with:
  
-''​IDL>​ d=spice_object(file)''​+''​IDL>​ d=spice\_object(file)''​ 
 + 
 +==== Extracting information from the data object ==== 
 + 
 +The table below gives some methods for extracting information out of the data object. Where "​i"​ is given, it means the index of a wavelength window should be specified (indices begin at 0). 
 + 
 +You can get a list of all methods by doing: 
 + 
 +IDL> d->​help 
 + 
 +^ Command ​                       ^ Function ​                              ^ 
 +| d->​get\_number\_windows() ​     | No. of spectral windows ​               | 
 +| d->​get\_window\_data(i,/​load) ​ | Extract a data window ​                 | 
 +| d->​get\_lambda\_vector(i) ​     | Get wavelength vector for the window ​                 | 
 +| d->​get\_header(i) ​             | Extract a data window header ​          | 
 +| d->​get\_window\_id(i) ​         | Get the data window ID                 | 
 +| d->​get\_start\_time() ​         | Start time of observation ​             | 
 +| d->​get\_end\_time() ​           | End time of observation ​               | 
 +| d->​get\_sit\_and\_stare() ​     | Set to 1 if sit-and-stare observation ​ | 
 +| d->​get\_number\_exposures() ​   | Number of exposures in raster ​         | 
 +| d->​get\_xcen(i) ​               | Get the X-center for the window ​       | 
 +| d->​get\_ycen(i) ​               | Get the Y-center for the window ​       | 
 + 
 + 
 +==== Quicklook tools ==== 
 + 
 +A set of five widget-based tools are available for browsing SPICE data, and these can be accessed through spice\_xfiles:​ 
 + 
 +''​ IDL> spice\_xfiles''​ 
 + 
 +This allows you to select a FITS file from your SPICE data directory. A new widget appears from which you can then select one of the five quicklook tools: Detector, Raster Browser, Raster, Whisker and Intensity map. These tools mimic software that were available for EIS and IRIS. 
 + 
 +For further details, please visit the [[https://​github.com/​ITA-Solar/​solo-spice-ql/​wiki|SPICE Quicklook and Data Analysis Software Page]]. 
 + 
 +=== Raster Browser === 
 + 
 +This is useful for browsing the 3D data cubes from SPICE rasters. In addition to being called from spice\_xfiles (see above), it can also be called directly from the command line: 
 + 
 +''​IDL>​ spice\_raster\_browser,​file''​ 
 + 
 +Use a 3-button mouse to browse the images and spectra: the middle button allows you to select a new pixel, the left button zooms in and the right button zooms out. 
 + 
 + 
 +===== Ephemeris information ===== 
 + 
 +You can access ephemeris information from the SPICE headers. For example, the keywords: 
 + 
 +DSUN\_AU - distance of the spacecraft from the Sun in AU\\ 
 +EAR\_TDEL - light travel time from the spacecraft to Earth (seconds) 
 + 
 +The full list of keywords is given in the [[https://​spice.ias.u-psud.fr/​spice-data/​documents/​SPICE-UIO-DPDD-0002-1.4-Data_Product_Description_Document.pdf|DPDD]] - search for "Solar ephemeris keywords"​. 
 + 
 +Users should make sure to correct the SPICE observing times for EAR\_TDEL in order to compare with observations from Earth-orbiting spacecraft. This is especially important for highly-dynamic structures such as flares or CMEs. 
 + 
 ===== Read and display data (Python) ===== ===== Read and display data (Python) =====
  
data/data_analysis_manual.1625072656.txt.gz · Last modified: 2021/06/30 19:04 by Peter Young