User Tools

Site Tools


data:data_analysis_manual:read_catalog_python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
data:data_analysis_manual:read_catalog_python [2022/08/27 09:54]
eric buchlin Mention CSV catalog
data:data_analysis_manual:read_catalog_python [2022/09/21 13:57] (current)
gabriel pelouze fix function reading uio csv catalog (it was reading the txt catalog before)
Line 1: Line 1:
-====== How to read the UiO FITS files catalog in Python ======+====== How to read and use the UiO FITS files catalog in Python ======
  
-The following script is for reading the text format catalog. Now UiO also provides a CSV catalog.+===== CSV catalog ​(new, recommended) =====
  
-<file python ​read_uio_cat.py>+<file python ​read_uio_cat_csv.py>
 from pathlib import Path from pathlib import Path
 import pandas as pd import pandas as pd
Line 20: Line 20:
 def read_uio_cat():​ def read_uio_cat():​
     """​     """​
-    Read UiO text table SPICE FITS files catalog +    Read UiO SPICE FITS files CSV catalog 
-    http://​astro-sdc-db.uio.no/​vol/​spice/​fits/​spice_catalog.txt+    http://​astro-sdc-db.uio.no/​vol/​spice/​fits/​spice_catalog.csv
  
     Return     Return
Line 27: Line 27:
     pandas.DataFrame     pandas.DataFrame
         Table         Table
 +    """​
 +    cat_file = Path(data_path) / "​fits"​ / "​spice_catalog.csv"​
 +    if not cat_file.exists():​
 +        print(f'​Error:​ Catalog file not available at {cat_file.as_posix()}'​)
 +        sys.exit(1)
 +    date_columns = ['​DATE-BEG','​DATE',​ '​TIMAQUTC'​]
 +    df = pd.csv(cat_file,​ parse_dates=date_columns,​ date_parser=date_parser)
 +    return df
 +</​file>​
  
-    Example queries that can be done on the result:+The same applies for the catalog included in the data releases (here: release 2.0), which can simply ​be read by:
  
-    * `df[(df.LEVEL == "​L2"​(df["DATE-BEG">"​2020-11-17"​) & (df["​DATE-BEG"] < "2020-11-18") & (df.XPOSURE > 60.)]` +<file python read_release_cat.py>​ 
-    ​`df[(df.LEVEL == "L2"​) ​(df.STUDYDES == "Standard dark for cruise phase")]`+import pandas as pd 
 + 
 +def date_parser(string)
 +    try: 
 +        return pd.Timestamp(string) 
 +    except ValueError:​ 
 +         ​return pd.NaT 
 + 
 +date_columns = ['DATE-BEG','​DATE',​ '​TIMAQUTC'​] 
 +cat pd.read_csv( 
 +    '​https://​spice.osups.universite-paris-saclay.fr/​spice-data/​release-2.0/catalog.csv', 
 +    ​date_parser=date_parser,​ 
 +    parse_dates=date_columns 
 +
 +# TODO interpret the JSON included in columns `proc_steps` and `windows`. 
 +</​file>​ 
 + 
 + 
 +===== Text catalog ===== 
 + 
 +<file python read_uio_cat_txt.py>​ 
 +from pathlib import Path 
 +import pandas as pd 
 + 
 +# SPICE data tree path, to be changed to your SPICE data mirror 
 +data_path ​= "/​archive/​SOLAR-ORBITER/​SPICE" ​     # example for IAS computing servers 
 + 
 + 
 +def date_parser(string)
 +    try: 
 +        return pd.Timestamp(string) 
 +    except ValueError:​ 
 +         ​return pd.NaT 
 + 
 + 
 +def read_uio_cat():​ 
 +    ​""​
 +    Read UiO text table SPICE FITS files catalog 
 +    http://​astro-sdc-db.uio.no/​vol/​spice/​fits/​spice_catalog.txt 
 + 
 +    Return 
 +    ------ 
 +    pandas.DataFrame 
 +        Table
     """​     """​
     cat_file = Path(data_path) / "​fits"​ / "​spice_catalog.txt"​     cat_file = Path(data_path) / "​fits"​ / "​spice_catalog.txt"​
Line 45: Line 97:
 </​file>​ </​file>​
  
 +
 +===== Using the catalog =====
  
 Then we can read the catalog and filter it: Then we can read the catalog and filter it:
data/data_analysis_manual/read_catalog_python.1661586896.txt.gz · Last modified: 2022/08/27 09:54 by eric buchlin