data:data_analysis_manual:read_catalog_python
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| data:data_analysis_manual:read_catalog_python [2022/08/27 07:54] – Mention CSV catalog eric buchlin | data:data_analysis_manual:read_catalog_python [2024/03/29 13:11] (current) – Page is deprecated eric buchlin | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== How to read the UiO FITS files catalog in Python ====== | + | ====== How to read and use the UiO FITS files catalog in Python ====== |
| - | The following script is for reading | + | <note warning> |
| - | <file python | + | ===== CSV catalog (new, recommended) ===== |
| + | |||
| + | <file python | ||
| from pathlib import Path | from pathlib import Path | ||
| import pandas as pd | import pandas as pd | ||
| Line 20: | Line 22: | ||
| def read_uio_cat(): | def read_uio_cat(): | ||
| """ | """ | ||
| - | Read UiO text table SPICE FITS files catalog | + | Read UiO SPICE FITS files CSV catalog |
| - | http:// | + | http:// |
| Return | Return | ||
| Line 27: | Line 29: | ||
| pandas.DataFrame | pandas.DataFrame | ||
| Table | Table | ||
| + | """ | ||
| + | cat_file = Path(data_path) / " | ||
| + | if not cat_file.exists(): | ||
| + | print(f' | ||
| + | sys.exit(1) | ||
| + | date_columns = [' | ||
| + | df = pd.read_csv(cat_file, | ||
| + | return df | ||
| + | </ | ||
| - | Example queries that can be done on the result: | + | The same applies for the catalog included in the data releases (here: release 2.0), which can simply |
| - | * `df[(df.LEVEL == " | + | <file python read_release_cat.py> |
| - | | + | import pandas as pd |
| + | |||
| + | def date_parser(string): | ||
| + | try: | ||
| + | return pd.Timestamp(string) | ||
| + | except ValueError: | ||
| + | | ||
| + | |||
| + | date_columns = ['DATE-BEG',' | ||
| + | cat = pd.read_csv( | ||
| + | ' | ||
| + | | ||
| + | parse_dates=date_columns | ||
| + | ) | ||
| + | # TODO interpret the JSON included in columns `proc_steps` and `windows`. | ||
| + | </ | ||
| + | |||
| + | |||
| + | ===== Text catalog ===== | ||
| + | |||
| + | <file python read_uio_cat_txt.py> | ||
| + | from pathlib import Path | ||
| + | import pandas as pd | ||
| + | |||
| + | # SPICE data tree path, to be changed to your SPICE data mirror | ||
| + | data_path | ||
| + | |||
| + | |||
| + | def date_parser(string): | ||
| + | try: | ||
| + | return pd.Timestamp(string) | ||
| + | except ValueError: | ||
| + | | ||
| + | |||
| + | |||
| + | def read_uio_cat(): | ||
| + | | ||
| + | Read UiO text table SPICE FITS files catalog | ||
| + | http:// | ||
| + | |||
| + | Return | ||
| + | ------ | ||
| + | pandas.DataFrame | ||
| + | Table | ||
| """ | """ | ||
| cat_file = Path(data_path) / " | cat_file = Path(data_path) / " | ||
| Line 45: | Line 99: | ||
| </ | </ | ||
| + | |||
| + | ===== Using the catalog ===== | ||
| Then we can read the catalog and filter it: | Then we can read the catalog and filter it: | ||
data/data_analysis_manual/read_catalog_python.1661586896.txt.gz · Last modified: by eric buchlin
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
