User Tools

Site Tools


data:data_analysis_manual:read_catalog_python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revision Both sides next revision
data:data_analysis_manual:read_catalog_python [2022/08/27 09:52]
eric buchlin No padding anymore in UiO catalog
data:data_analysis_manual:read_catalog_python [2022/08/30 18:20]
eric buchlin Read catalog included in releases
Line 1: Line 1:
-====== How to read the UiO FITS files catalog in Python ======+====== How to read and use the UiO FITS files catalog in Python ====== 
 + 
 +===== CSV catalog (new, recommended) ​=====
  
 <file python read_uio_cat.py>​ <file python read_uio_cat.py>​
Line 18: Line 20:
 def read_uio_cat():​ def read_uio_cat():​
     """​     """​
-    Read UiO text table SPICE FITS files catalog +    Read UiO SPICE FITS files CSV catalog 
-    http://​astro-sdc-db.uio.no/​vol/​spice/​fits/​spice_catalog.txt+    http://​astro-sdc-db.uio.no/​vol/​spice/​fits/​spice_catalog.csv
  
     Return     Return
Line 25: Line 27:
     pandas.DataFrame     pandas.DataFrame
         Table         Table
 +    """​
 +    cat_file = Path(data_path) / "​fits"​ / "​spice_catalog.txt"​
 +    if not cat_file.exists():​
 +        print(f'​Error:​ Catalog file not available at {cat_file.as_posix()}'​)
 +        sys.exit(1)
 +    date_columns = ['​DATE-BEG','​DATE',​ '​TIMAQUTC'​]
 +    df = pd.read_table(cat_file,​ parse_dates=date_columns,​ date_parser=date_parser)
 +    return df
 +</​file>​
  
-    Example queries that can be done on the result:+The same applies for the catalog included in the data releases (here: release 2.0), which can simply ​be read by:
  
-    * `df[(df.LEVEL == "​L2"​(df["DATE-BEG">"​2020-11-17"​) & (df["​DATE-BEG"] < "2020-11-18") & (df.XPOSURE > 60.)]` +<file python read_release_cat.py>​ 
-    ​`df[(df.LEVEL == "L2"​) ​(df.STUDYDES == "Standard dark for cruise phase")]`+import pandas as pd 
 + 
 +def date_parser(string)
 +    try: 
 +        return pd.Timestamp(string) 
 +    except ValueError:​ 
 +         ​return pd.NaT 
 + 
 +date_columns = ['DATE-BEG','​DATE',​ '​TIMAQUTC'​] 
 +cat pd.read_csv( 
 +    '​https://​spice.osups.universite-paris-saclay.fr/​spice-data/​release-2.0/catalog.csv', 
 +    ​date_parser=date_parser,​ 
 +    parse_dates=date_columns 
 +
 +# TODO interpret the JSON included in columns `proc_steps` and `windows`. 
 +</​file>​ 
 + 
 + 
 +===== Text catalog ===== 
 + 
 +<file python read_uio_cat.py>​ 
 +from pathlib import Path 
 +import pandas as pd 
 + 
 +# SPICE data tree path, to be changed to your SPICE data mirror 
 +data_path ​= "/​archive/​SOLAR-ORBITER/​SPICE" ​     # example for IAS computing servers 
 + 
 + 
 +def date_parser(string)
 +    try: 
 +        return pd.Timestamp(string) 
 +    except ValueError:​ 
 +         ​return pd.NaT 
 + 
 + 
 +def read_uio_cat():​ 
 +    ​""​
 +    Read UiO text table SPICE FITS files catalog 
 +    http://​astro-sdc-db.uio.no/​vol/​spice/​fits/​spice_catalog.txt 
 + 
 +    Return 
 +    ------ 
 +    pandas.DataFrame 
 +        Table
     """​     """​
     cat_file = Path(data_path) / "​fits"​ / "​spice_catalog.txt"​     cat_file = Path(data_path) / "​fits"​ / "​spice_catalog.txt"​
Line 43: Line 97:
 </​file>​ </​file>​
  
 +
 +===== Using the catalog =====
  
 Then we can read the catalog and filter it: Then we can read the catalog and filter it:
data/data_analysis_manual/read_catalog_python.txt · Last modified: 2022/12/19 17:47 by eric buchlin