User Tools

Site Tools


data:data_analysis_manual:read_catalog_python

This is an old revision of the document!


How to read the UiO FITS files catalog in Python

The following script is for reading the text format catalog. Now UiO also provides a CSV catalog.

read_uio_cat.py
from pathlib import Path
import pandas as pd
 
# SPICE data tree path, to be changed to your SPICE data mirror
data_path = "/archive/SOLAR-ORBITER/SPICE"      # example for IAS computing servers
 
 
def date_parser(string):
    try:
        return pd.Timestamp(string)
    except ValueError:
         return pd.NaT
 
 
def read_uio_cat():
    """
    Read UiO text table SPICE FITS files catalog
    http://astro-sdc-db.uio.no/vol/spice/fits/spice_catalog.txt
 
    Return
    ------
    pandas.DataFrame
        Table
 
    Example queries that can be done on the result:
 
    * `df[(df.LEVEL == "L2") & (df["DATE-BEG"] >= "2020-11-17") & (df["DATE-BEG"] < "2020-11-18") & (df.XPOSURE > 60.)]`
    * `df[(df.LEVEL == "L2") & (df.STUDYDES == "Standard dark for cruise phase")]`
    """
    cat_file = Path(data_path) / "fits" / "spice_catalog.txt"
    if not cat_file.exists():
        print(f'Error: Catalog file not available at {cat_file.as_posix()}')
        sys.exit(1)
    columns = list(pd.read_csv(cat_file, nrows=0).keys())
    date_columns = ['DATE-BEG','DATE', 'TIMAQUTC']
    df = pd.read_table(cat_file, skiprows=1, names=columns,
                    parse_dates=date_columns, date_parser=date_parser,
                    low_memory=False)
    return df

Then we can read the catalog and filter it:

filter_cat.py
cat = read_uio_cat()
filtered_cat = cat[(cat['DATE-BEG'] > '2021-11-05') & (cat.LEVEL == 'L2')]

`cat` then contains the full catalogue (as a `pandas` dataframe) and `filtered_cat` contains a catalogue in which rows have been filtered (in this particular case) by observation date and file level.

The catalogue or filtered catalogue can be exported, e.g. to a CSV table by `cat.to_csv()`.

data/data_analysis_manual/read_catalog_python.1661586896.txt.gz · Last modified: by eric buchlin