Typical use case for physical parameters

This example notebook shows a run on the COSMOS2020 (Weaver et al. 2022) data set in order to estimate the physical parameters.

In this notebook we follow what has been done for photo-z. We will be looking at the same COSMOS data set but only use the ugrizy bands. The main difference is the use of BC03 templates to compute physical parameters, and we set the redshift to its spectrosocpic value.

The notebook can be downloaded here.

[1]:

import lephare as lp
from astropy.table import Table
import numpy as np
import os
from matplotlib import pylab as plt
import time

%matplotlib inline
%load_ext wurlitzer

LEPHAREDIR is being set to the default cache directory:
/home/docs/.cache/lephare/data
More than 1Gb may be written there.
LEPHAREWORK is being set to the default cache directory:
/home/docs/.cache/lephare/work
Default work cache is already linked.
This is linked to the run directory:
/home/docs/.cache/lephare/runs/20260304T132506

Update the config

We will start with the COSMOS configuration as a basis. We will update the various keywords needed for this example. We use the default which is shipped with lephare. You could also download the example text file config from here or write it completely from scratch.

[2]:

config = lp.default_cosmos_config.copy()

# You could also load this from a local text file:
# config=lp.keymap_to_string_dict(lp.read_config("your_own.para"))
# An example can be downloaded with curl
# !curl -s -o ./COSMOS.para https://raw.githubusercontent.com/lephare-photoz/lephare-data/refs/heads/main/examples/COSMOS.para
# config = lp.read_config("./COSMOS.para")

config.update(
    {
        # For a quick demonstration we use a very sparse redshift grid. DO NOT USE FOR SCIENCE!
        # Comment out the following line to improve results.
        "Z_STEP": "0.1,0.,3.",
        # SED
        # In order to get the physical parameters you need to use
        # Composite Stellar Population synthesis models. Here Bruzual & Charlot (2003).
        # This can be done only for galaxies.
        "GAL_SED": "$LEPHAREDIR/sed/GAL/BC03_CHAB/BC03COMB_MOD.list",
        # Limit the number of ages
        "SEL_AGE": "$LEPHAREDIR/sed/GAL/BC03_CHAB/AGE_BC03COMB.dat",
        "MOD_EXTINC": "0,12,0,12",
        "EXTINC_LAW": "SB_calzetti.dat,SMC_prevot.dat",
        "EM_LINES": "PHYS",
        "EM_DISPERSION": "1.",
        # FILTERS
        # A reduced list of filters:
        "FILTER_LIST": "cosmos/u_new.pb,hsc/gHSC.pb,hsc/rHSC.pb,hsc/iHSC.pb,hsc/zHSC.pb,hsc/yHSC.pb",
        "FILTER_CALIB": "0",
        "FILTER_FILE": "filter_test",
        # FIT
        # We set the redshift to the spec-z value
        "ZFIX": "YES",
        "ERR_SCALE": "0.02",
        "ERR_FACTOR": "1.5",
        "SPEC_OUT": "NO",  # We would like to see the output
        "VERBOSE": "NO",
    }
)

Download the missing data

One does not need to use this functionality if already cloned the full auxiliary data.

[3]:

lp.data_retrieval.get_auxiliary_data(
    keymap=config,
    # The additional extinction laws for galaxies are not in the principle config
    # so we must add them to be downloaded:
    additional_files=[
        # We also want the example cosmos catalogue to experiment with
        "examples/COSMOS.in",
        "ext/SMC_prevot.dat",
        "ext/SB_calzetti.dat",
        "sed/GAL/BC03_CHAB/AGE_BC03COMB.dat",
    ],
)

Registry file downloaded and saved as data_registry.txt.

Downloading file 'sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau01_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau01_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB/BC03COMB_MOD.list' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB/BC03COMB_MOD.list' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau03_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau03_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau1_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau1_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau3_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau3_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB/bc2003_lr_m52_chab_tau5_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB/bc2003_lr_m52_chab_tau5_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau5_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau5_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.

Created directory: /home/docs/.cache/lephare/data/sed/GAL/BC03_CHAB_DELAYED
Created directory: /home/docs/.cache/lephare/data/sed/GAL/BC03_CHAB
Checking/downloading 400 files...

Downloading file 'sed/GAL/BC03_CHAB/bc2003_lr_m52_chab_tau30_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB/bc2003_lr_m52_chab_tau30_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau30_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB/bc2003_lr_m62_chab_tau30_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB_DELAYED/bc2003_lr_m52_chab_delayed1_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB_DELAYED/bc2003_lr_m52_chab_delayed1_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB_DELAYED/bc2003_lr_m52_chab_delayed3_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB_DELAYED/bc2003_lr_m52_chab_delayed3_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB_DELAYED/bc2003_lr_m62_chab_delayed1_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB_DELAYED/bc2003_lr_m62_chab_delayed1_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB_DELAYED/bc2003_lr_m62_chab_delayed3_dust00.ised_ASCII' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB_DELAYED/bc2003_lr_m62_chab_delayed3_dust00.ised_ASCII' to '/home/docs/.cache/lephare/data'.
Downloading file 'sed/GAL/BC03_CHAB/AGE_BC03COMB.dat' from 'https://raw.githubusercontent.com/lephare-photoz/lephare-data/main/sed/GAL/BC03_CHAB/AGE_BC03COMB.dat' to '/home/docs/.cache/lephare/data'.

400 completed.
All files downloaded successfully and are non-empty.
Checking/downloading 4 files...
4 completed.
All files downloaded successfully and are non-empty.

Run prepare

These are the key preparatory stages that calculate the filters in the LePHARE format, calculate the library of SEDs and finally calculate the library of magnitudes for all the models. The prepare method runs filter, sedtolib, and mag_gal that would be run independently at the command line. These are all explained in detail in the documentation.

[4]:

lp.prepare(
    config,
)

Creating the input table

We need to make an astropy table as input. This can be done using the standard column order: id, flux0, err0, flux1, err1,…, context, zspec, arbitrary_string. A simple example table with two filters might look like this: | id | flux_filt1 | fluxerr_filt1 | flux_filt2 | fluxerr_filt2 | context | zspec | string_data | |—|---|—|---|—|---|—|—| | 0 | 1.e-31 | 1.e-32 | 1.e-31 | 2.e-32 | 3 | NaN | “This is just a note” | | 1 | 2.e-31 | 1.e-32 | 1.e-31 | 2.e-32 |3 | 1. | “This has a specz” | | 2 | 2.e-31 | 1.e-32 | 2.e-31 | 2.e-32 | 2 | NaN| “This context only uses the second filter” |

The context detemermines which bands are used but can be -99 or a numpy.nan. We do not need to have units on the flux columns but LePHARE assumes they are in erg /s /cm**2 / Hz if we are using fluxes. The number of columns must be two times the number of filters plus the four additional columns.

This input table must use the standard column ordering to determine column meaning. This odering depends on the filter order in the config FILTER_LIST value.

[5]:

# Load the full cosmos example we downloaded at the start
cosmos_full = Table.read(f"{lp.LEPHAREDIR}/examples/COSMOS.in", format="ascii")
# Lets just look at the first 1000 specz between 0 and 3 to be fast and have a small sample to test
specz_colname = cosmos_full.colnames[-2]
mask = cosmos_full[specz_colname] > 0

mask &= cosmos_full[specz_colname] < 3
cosmos_full = cosmos_full[mask][:1000]

[6]:

input_table = Table()
# The id is in the first column
input_table["id"] = cosmos_full[cosmos_full.colnames[0]]
# Loop over the filters we want to keep to get the number of the filter, n, and the name, b,
filter_names = config["FILTER_LIST"].split(",")
for n, filter_name in enumerate(filter_names):
    # The ugrizy fluxes and errors are in cols 3 to 14
    f_col = cosmos_full.colnames[2 * n + 3]
    ferr_col = cosmos_full.colnames[2 * n + 4]
    # By default lephare uses column order so names are irrelevant
    input_table[f"f_{filter_name}"] = cosmos_full[f_col]
    input_table[f"ferr_{filter_name}"] = cosmos_full[ferr_col]
# The context is a binary flag. Here we set it to use all filters.
input_table["context"] = np.sum(2 ** np.arange(len(filter_names)))
input_table["zspec"] = cosmos_full[specz_colname]
input_table["string_data"] = "arbitrary_info"

[7]:

# Look at the first 5 lines of the input table
input_table[:5]

[7]:
Table length=5
idf_cosmos/u_new.pbferr_cosmos/u_new.pbf_hsc/gHSC.pbferr_hsc/gHSC.pbf_hsc/rHSC.pbferr_hsc/rHSC.pbf_hsc/iHSC.pbferr_hsc/iHSC.pbf_hsc/zHSC.pbferr_hsc/zHSC.pbf_hsc/yHSC.pbferr_hsc/yHSC.pbcontextzspecstring_data
int64float64float64float64float64float64float64float64float64float64float64float64float64int64float64str14
24.232901366689564e-303.4341315142888603e-316.179738935719169e-301.386731273412228e-311.2142639224559102e-291.8245325069819158e-312.1114793014441938e-292.1601688228846318e-313.9365460376947166e-293.4133974396324463e-315.0357913173859444e-295.203234525553232e-31631.166arbitrary_info
31.8604787800453155e-283.6102558328725637e-311.0163707206619843e-272.5508479438832455e-302.5637553789165174e-273.907537695918684e-303.852181904261173e-274.270412613938428e-305.051842896248005e-275.651717781917373e-305.820737300549565e-276.786202238144557e-30630.1649arbitrary_info
63.883122772291653e-309.595010842153324e-324.647041003941889e-301.087246463423006e-314.8469179256155955e-301.178374194432367e-316.118486088007425e-301.2683990766417867e-319.150015677196151e-301.9130763838587345e-311.0166486142973662e-293.495575779500692e-31631.25arbitrary_info
92.8521064572984233e-301.897459353660885e-317.283799341197919e-301.7286878697910101e-312.3956967578910942e-292.8232988053219177e-315.597814087271151e-293.7781712042067346e-318.467793964385691e-295.472733062400153e-311.0885331971552352e-287.893832949105172e-31630.6233arbitrary_info
104.0113693940454606e-301.4061914648406559e-315.3481882134164066e-301.4193075132576959e-316.221652738450281e-301.5878330837093787e-317.330388037942477e-301.735529599561101e-311.1108868491203026e-292.6930229981999465e-311.5213763225065445e-295.031524893553477e-31631.4005arbitrary_info

Run process

Finally we run the main fitting process which is equivalent to zphota when using the command line. We also need to update some of the config values to make them consistent with the number of filters.

[8]:

# Compute physical parameters
output, _ = lp.process(config, input_table)

Using user columns from input table assuming they are in the standard order.
Processing 1000 objects with 6 bands
AUTO_ADAPT is set to YES. Computing offsets.
# Offsets added to the modeled magnitudes (or substracted to the observed): 0.0,0.0,0.0,0.0,0.0,0.0


[9]:

# the output is an astropy tabel that can be manipulated in the standard ways.
output[:5]

[9]:
Table length=5
IDENTZ_BESTZ_BEST68_LOWZ_BEST68_HIGHZ_MEDZ_MED68_LOWZ_MED68_HIGHCHI_BESTMOD_BESTEXTLAW_BESTEBV_BESTZ_SECCHI_SECMOD_SECEBV_SECZQ_BESTCHI_QSOMOD_QSOMOD_STARCHI_STARMAG_OBS()ERR_MAG_OBS()K_COR()MAG_ABS()EMAG_ABS()MABS_FILT()SCALE_BESTNBAND_USEDCONTEXTZSPECAGE_BESTAGE_INFAGE_MEDAGE_SUPLDUST_BESTLDUST_INFLDUST_MEDLDUST_SUPLUM_TIR_BESTLUM_TIR_INFLUM_TIR_MEDLUM_TIR_SUPMASS_BESTMASS_INFMASS_MEDMASS_SUPSFR_BESTSFR_INFSFR_MEDSFR_SUPSSFR_BESTSSFR_INFSSFR_MEDSSFR_SUPCOL1_INFCOL1_MEDCOL1_SUPCOL2_INFCOL2_MEDCOL2_SUPLUM_NUV_BESTLUM_R_BESTLUM_K_BESTMAG_PRED()STRING_INPUTPDF_BAY_ZG()
str4float64float64float64float64float64float64float64int64float64float64float64float64int64float64float64float64int64int64float64float64[6]float64[6]float64[6]float64[6]float64[6]float64[6]float64int64int64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64float64[0]str14float64[31]
21.1660.03.01.20000000000000021.13199999928474451.26800000071525597.35354703032218172.00.3-99.9-99.0-99-99.01.1666.6722574210142087-991000000000.024.83340462789454 .. 22.144830688261890.1355221450661712 .. 0.034399193768563791.4027514359493192 .. 0.22746431903521272-20.76039123476376 .. -22.5954226606309160.35799910340307406 .. 3.2172561811241213.0 .. 5.0508394575538.15454601.1669.4771212547196639.0708269318097649.3408080339188379.58513834904918811.26268890679380711.26377591465958411.3192132439027711.430530743344463-999.0-99.9-99.9-99.910.45890690930761810.30549804158810910.46044985622804910.6173546822949221.18565017251188021.14607003912986841.2548519657317121.4173108543425346-9.273256736795737-9.386450745969796-9.182778986875206-8.908110590004203-99.9-99.9-99.9-99.9-99.9-99.910.0987818785969710.0790565461137249.26246804596414arbitrary_info0.0 .. 0.0
30.16490.03.00.20.131999999284744260.268000000715255761.127109729785312311.00.35-99.9-99.0-99-99.00.164949.5480158652725251-991000000000.020.725938197553823 .. 16.987555001743210.03016607985901451 .. 0.0300600556726643640.8065954644069318 .. 0.11396514031072706-19.553034052624728 .. -22.6180552150121451.2252872539779265 .. 0.90432282852835751.0 .. 5.062291433485.1377600.16498.856548244299958.8147150588527848.8827870497872689.00248083712686410.94651707516501411.10895697656815511.16243968410352711.238848432982804-999.0-99.9-99.9-99.910.59712857876488510.73954078228709210.78164733015263410.82537362398034-0.32685034785477257-1.4055377938750315-0.17612927796832045-0.00037526675499194306-10.923978926619657-12.205537469043191-10.976131415901882-10.800382201257246-99.9-99.9-99.9-99.9-99.9-99.99.4257039461882110.1870586224178339.425068101607351arbitrary_info0.0 .. 0.0
61.250.03.01.20000000000000021.13199999928474451.26800000071525590.8753132980664218111.00.15-99.9-99.0-99-99.01.2521.0419584863924721-991000000000.024.92704719548275 .. 23.8820728174009960.05020167671229883 .. 0.06353871767418252-0.16111586808040457 .. -0.21214526248971308-19.57709975782467 .. -20.6054874287882440.3375439166770793 .. 3.60448565322461434.0 .. 5.013803624804.673458601.258.856548244299958.5360178765046948.8036085850528469.10745508340753110.58152924947810210.21356521685788510.48942330548668510.574905411891562-999.0-99.9-99.9-99.99.191862492678668.9755600337548769.1157784805432279.2617805804977030.6844134764143610.315524022095906630.50998746401594550.6591685767866572-8.507449016264298-8.922962444876912-8.563317879883106-8.395134335338103-99.9-99.9-99.9-99.9-99.9-99.99.5486541024710129.2149218611475488.340639502400958arbitrary_info0.0 .. 0.0
90.62330.03.00.60.53199999928474420.668000000715255810.0855310742551391.00.15-99.9-99.0-99-99.00.62332.10154248404085874-991000000000.025.262085670267126 .. 21.3078958041973470.11245030713537811 .. 0.032242077373582721.3022229798649088 .. 0.3272615330764923-19.084924703360304 .. -21.8588645799560130.26711059816815386 .. 1.72114404558625012.0 .. 5.060861552909.72917600.62339.8129133392710759.650048891058829.7802274788023939.84483344431772210.18721797469725110.68545032085513610.87473823552697110.983753138037462-999.0-99.9-99.9-99.910.50632946118216510.3895390781017910.46271495100871510.531521287516288-0.22565755144390526-0.4854679750949731-0.148899394692318780.37485509386084137-10.731987012626071-10.887791840258355-10.614074825331386-10.108976115916684-99.9-99.9-99.9-99.9-99.9-99.98.779466221639159.5649476395696998.954704685600117arbitrary_info0.0 .. 0.0
101.40050.03.01.41.33199999928474421.46800000071525560.3484112087385771792.00.1-99.9-99.0-99-99.01.400535.346032101527847-991000000000.024.891768358503363 .. 23.4444083676074650.06450552103875724 .. 0.061664191418595480.11232905902905901 .. -0.3339735279878466-20.218146548545384 .. -21.226587736339210.37713681002682975 .. 4.5819561094622194.0 .. 5.013236946761.090946601.40059.2065506376524068.8201187681319419.1611407166271959.3738599477541210.51534553709543710.47447466747156910.53502438244534610.976592245618587-999.0-99.9-99.9-99.99.6061090624153699.5451397100518749.698758101424399.880570102809850.62956733356090940.59229867864888330.69584462086975910.9426267673077545-8.97654172885446-9.188513913247936-9.000918573678264-8.730313145618705-99.9-99.9-99.9-99.9-99.9-99.99.5457756428621439.4246508925250198.531288470037632arbitrary_info0.0 .. 0.0

Next we can perform some simple plots to check the output

[10]:

logmass = output["MASS_MED"]
logSFR = output["SFR_MED"]
z = output["Z_BEST"]
cond = (z > 0.5) & (z < 1) & (logmass > 0)
plt.hist(logmass[cond], bins=20)
plt.xlabel("log(stellar mass)")

[10]:

Text(0.5, 0, 'log(stellar mass)')
../_images/notebooks_Typical_use_case_physicalParameters_16_1.png 
[11]:

plt.scatter(logmass[cond], logSFR[cond], s=2.0)
plt.xlabel("log(stellar mass)")
plt.ylabel("log(SFR)")

[11]:

Text(0, 0.5, 'log(SFR)')
../_images/notebooks_Typical_use_case_physicalParameters_17_1.png

You can use some plotting utilities provided with the code. Some examples are provided below. sel_filt is the index of the filter used to select objects by observed magnitude (starting at 0). pos_filt provides the index (starting at 0) of the filters corresponding to u, g, r, z, J, and Ks bands. You can use some plotting utilities provided with the code. Some examples are provided below. If you want to create all the plots and store them in a pdf file, you can use: utils.save_phys_plots_pdf(filename=”all_phys.pdf”)

[12]:

utils = lp.PlotUtils(
    output,
    sel_filt=3,
    pos_filt=[0, 1, 2, 4, 5, 5],
    range_z=[0, 0.5, 1, 1.5, 3],
    range_mag=[19, 20.5, 21.5, 22.5, 25],
)

[13]:

utils.dist_chi2()

<Figure size 640x480 with 0 Axes>
../_images/notebooks_Typical_use_case_physicalParameters_20_1.png 
[14]:

utils.dist_mass()

<Figure size 640x480 with 0 Axes>
../_images/notebooks_Typical_use_case_physicalParameters_21_1.png 
[15]:

utils.mass_sfr()

<Figure size 640x480 with 0 Axes>
../_images/notebooks_Typical_use_case_physicalParameters_22_1.png 
[ ]: