ATClean

Interactive Jupyter Notebooks and Python Scripts for Cleaning and Analyzing ATLAS Light Curves

View our paper here for details on our cleaning and binning method, as well as our pre-SN outburst detection process!

Table of Contents

• Python Scripts

  • Install dependencies: Install the necessary dependencies.
  • Setup in config.ini: Set the default configuration for the convert_to_atclean.py, download.py, and clean.py scripts.
  • download.py: Download one or more SNe and their control light curves from the ATLAS forced photometry server.
  • convert_to_atclean.py: Convert a non-ATLAS light curve (and, optionally, its control light curves) into an ATClean-readable format, so that it may be run through any of the following scripts.
  • clean.py: Apply one or more default and/or custom cuts and binning to one or more SNe and their control light curves.
  • plotloop.py: Generate a summary PDF file of diagnostic plots.
  • zip.py: For one or more SNe, zip files from input and output directories.
  • unzip.py: For one or more SNe, unzip files generated by zip.py and place them into the correct directories.
  • step1_generate_sim_tables.py (WIP): Part of our pre-SN outburst detection analysis. Generate tables of simulations (SimTables) by specifying the type of model and possible parameter values.
  • step2_generate_detec_tables.py (WIP): Part of our pre-SN outburst detection analysis. For each row in each SimTable, add the simulation to a random control light curve and record its max FOM and MJD, then update the rows and save as SimDetecTables.
  • step3_calculate_efficiencies.py (WIP): For a given set of SimDetecTables, determine best FOM limits, then calculate contamination, efficiencies, and magnitude thresholds.

• Jupyter Notebooks

  • clean.ipynb: An in-depth walkthrough of our cleaning and binning process for a single SN and its control light curves.
  • atlas_template_correction.ipynb: A standalone walkthrough of our ATLAS template change correction.
  • simdetec_analysis.ipynb (WIP): Part of our pre-SN outburst detection analysis. An in-depth walkthrough analysis of the generated SimDetecTables and efficiencies for a given SN and its control light curves.

• Dependencies

Python Scripts

Install dependencies

View Dependencies for a comprehensive list of dependencies. Run ./install_dependencies.sh to install Python dependencies using pip.

Setup in config.ini

Open the config.ini file, which contains configuration for input/output directory paths, convert_to_atclean.py, download.py, and clean.py. The following toggles describe each field section by section. Bolded fields denote fields that we urge the user to change before attempting to use the scripts.

Note that these configurations may also be overridden by command line arguments.

Input/output directory paths: dir config section

• atclean_input: This parameter specifies the path to the directory where light curves that are in ATClean-readable format are stored. These files have either been directly downloaded from the ATLAS server by download.py or converted from their raw formats by convert_to_atclean.py.

• output: This parameter designates the path to the directory where all output data, including cleaned and binned light curves, plots, efficiency tables, and other results, will be saved. This directory serves as the main repository for the results produced by the ATClean pipeline.

• sninfo_filename: This parameter provides the name of the SN info file located inside the output directory. This space-separated .txt file contains essential information about SNe, including the TNS name and optionally the RA, Dec, and MJD0 (MJD at which the transient begins). This file may be provided manually with the correct column names (tnsname, ra, dec, and mjd0, with blank fields denoted by NaN) or generated and updated automatically if TNS credentials are provided.

Column name presets for ATLAS, Rubin, and TESS: column_name_preset.<SURVEY NAME> config sections

These sections specify the arbitrary column names for the supported surveys. Column names are customizable and need to be accessed in convert_to_atclean.py, clean.py, etc. You can specify which preset to use via the -p or --preset argument (default is atlas). We recommend that you leave the ATLAS column name preset untouched!

Columns required for the scripts to run are MJD, flux, and dflux (i.e., flux error or uncertainty). Optional columns like chi-square, filter, magnitude, etc. can be None, but the script will throw an error if a process you run requires it. When using convert_to_atclean.py to convert a file to ATClean-readable format, the extra_columns field allows you to keep any other columns relevant to you that may not be used in ATClean scripts.

Credentials for TNS API and ATLAS server: credentials config section

ATLAS credentials:

• atlas_username: The username required to download the ATLAS data from the server.

• atlas_password: The password required to download the ATLAS data from the server.

TNS credentials (these are optional, but if not provided, the user must provide a SN info file and set the sninfo_filename field):

• tns_api_key: The API key for accessing the TNS API.

• tns_id: The user ID for the TNS account.

• tns_bot_name: The name of the bot used for retrieving data from the TNS.

To set up a TNS bot and get an API key, login to TNS, then navigate here and click on "Add bot".

Getting "Permission denied" errors when trying to run a script?

Navigate to your cloned repository and run the following command to add executable permission to all .py files: chmod a+x *.py.

Tests

To run tests, run the following command from inside the atclean directory:

python -m unittest discover -s tests

download.py

This script allows you to download ATLAS light curve(s) using ATLAS's REST API and TNS's API (to optionally fetch RA, Dec, and discovery date information for the SN). All downloaded files will be storied in the directory specified by the atclean_input field in config.ini. Make sure to add your ATLAS username and password to the credentials section in config.ini.

The user can download a single SN or a batch of SNe, as well as their optional control light curves.

We give users the chance to either specify the control light curve coordinates through a table, or have them calculated automatically as a circle pattern.

• To download control light curves, use the -c argument.
• To specify the number of control light curves to download, verify that the num_controls field in config.ini is set to the correct number, or use the --num_controls argument.
• To specify the radius in arcseconds of the circle pattern of control light curves from the center location, verify that the radius field in config.ini is set to the correct number, or use the --radius argument.
• To manually specify all control light curve coordinates, you must provide a control coordinates file name using the --ctrl_coords_filepath argument.
  • The file must be space-separated and include at least the following columns: ra and dec.

We additionally allow the user to change the center location of the control light curve circle pattern. By default, we use the SN location. However, this location can be changed via the --center_coords argument, or by filling out the center_ra and center_dec columns in the SN info file.

We allow the user to either specify certain RA and Dec coordinates, or to query the TNS API and automatically retrieve and save the coordinates.

• To manually specify RA, Dec, and MJD0 for a single SN, use the --sn_coords and --mjd0 arguments.
• To manually specify RA, Dec, and MJD0 for one or more SNe, you must provide a SN info file in the output directory.
  • The file must be space-separated and include at least the following columns: tnsname, ra, dec, and mjd0. Any blank or unknown fields should be denoted by NaN.
  • If TNS credentials are provided, the script will query TNS for any blank or unknown fields and update the SN info file with the missing information.
• To automatically retrieve RA, Dec, and MJD0 from TNS, simply provide your credentials in config.ini. The SN info file will be automatically generated and maintained inside the output directory.

download config section in config.ini

• flux2mag_sigmalimit: The sigma limit used when converting flux to magnitude. Magnitudes are set as limits when their uncertainties are NaN.

• num_controls: The default number of control light curves to be downloaded and used for analysis.

• radius: The default radius in arcseconds for the circle pattern of control light curves around a center location (default is SN location).

• sn_min_dist: The minimum distance in arcseconds from the SN location to a control light curve location. This distance is used when the center of the circle pattern is set to a nearby bright object, and helps avoid any control locations landing on top of or too close to the SN.

Arguments

Arguments will override default config file settings if specified.

• First provide TNS name(s) of the object(s) to download.
• --sninfo_file: Specifies the SN info file name.
  • Type: str
  • Default: None (i.e., the sninfo_filename field in config.ini)
  • Usage: --sninfo_file sninfo.txt
• --config_file: Specifies the file name of the .ini file with settings for this script.
  • Type: str
  • Default: config.ini
  • Usage: --config_file config.ini
• -o, --overwrite: If specified, existing files with the same name will be overwritten.
  • Type: bool
  • Default: False
  • Usage: -o or --overwrite
• -l, --lookbacktime: Sets the lookback time in days.
  • Type: int
  • Default: None (i.e., look back as far as possible)
  • Usage: -l 100 or --lookbacktime 100
• --max_mjd: Specifies the maximum MJD to download data up to.
  • Type: float
  • Default: None
  • Usage: --max_mjd 59500.0
• --sn_coords: comma-separated RA and Dec of the SN light curve to download.
  • Type: str
  • Default: None (i.e., reference the SN info file or query TNS)
  • Usage: --sn_coords 10.684,41.269
• --mjd0: The start date of the SN in MJD.
  • Type: float
  • Default: None (i.e., reference the SN info file or query TNS)
  • Usage: --mjd0 58800.0
• -c, --download_controls: If specified, control light curves will be downloaded in addition to the SN light curve.
  • Type: bool
  • Default: False
  • Usage: -c or --download_controls
• -n, --num_controls: The number of control light curves to download per SN.
  • Type: int
  • Default: None (i.e., the num_controls field in config.ini)
  • Usage: -n 5 or --num_controls 5
• -r, --radius: The radius of the control light curve circle pattern around the center location (by default the SN location), in arcseconds.
  • Type: float
  • Default: None
  • Usage: -r 20.0 or --radius 20.0
• --ctrl_coords_filepath: Specifies the file path of the control coordinates table to load (instead of generating one).
  • Type: str
  • Default: None (i.e., calculate the coordinates of each control light curve)
  • Usage: --ctrl_coords_filepath ctrl_coords.txt
• --center_coords: Comma-separated RA and Dec of a nearby bright object interfering with the SN light curve. This object becomes the center of the control light curve circle pattern.
  • Type: str
  • Default: None (i.e., use the SN location as the center of the circle pattern)
  • Usage: --center_coords 10.684,41.269

Filename scheme

• All downloaded files will be storied in the directory specified by the atclean_input field in config.ini.
• Files for a single location are separated by filter (c for cyan, o for orange).
• Control light curves will contain the control ID in the filename.
• SN filenames: [ATCLEAN_INPUT]/[TNSNAME]/[TNSNAME].[FILTER].lc.txt
  • Examples: /path/to/atclean_input/2020lse/2020lse.c.lc.txt and /path/to/atclean_input/2020lse/2020lse.o.lc.txt
• Control filenames: [ATCLEAN_INPUT]/[TNSNAME]/controls/[TNSNAME]_i[CONTROL_INDEX].[FILTER].lc.txt
  • Examples: /path/to/atclean_input/2020lse/controls/2020lse_i001.c.lc.txt and /path/to/atclean_input/2020lse/controls/2020lse_i001.o.lc.txt

Example commands

• Download a single SN: ./download.py 2020lse -o
• Download a batch of SNe: ./download.py 2020lse 2019vxm 2023ixf -o
• Specify SN coordinates and MJD0: ./download.py 2020lse --sn_coords 10:41:02.190,-27:05:00.42 --mjd0 58985.264 -o
• Specify SN info table: ./download.py 2020lse 2019vxm 2023ixf --sninfo_file my_custom_SN_info.txt -o
• Download only the last 10 days of data: ./download.py 2020lse -l 10 -o
• Download control light curves: ./download.py 2020lse -c -o
• Specify radius and number of control light curves: ./download.py 2020lse -c --radius 34 --num_controls 16 -o
• Change the center location of the control light curve circle pattern: ./download.py 2020lse -c --center_coords 10:41:02.290,-27:05:00.52 -o
• Specify control light curve coordinates ./download.py 2020lse -c --ctrl_coords /path/to/control_coordinates_table.txt -o

convert_to_atclean.py

This script converts non-ATLAS light curves (and optionally their control light curves) into an ATClean-readable format. The resulting files can then be processed by other ATClean scripts, such as clean.py and plotloop.py.

Functionality:

• Converts light curve data from different surveys (e.g., Rubin, TESS) into a standardized format using the specified preset column name configuration (column_name_preset.<SURVEY_NAME> in config.ini).
• Allows users to retain additional columns via the extra_columns field in config.ini.
• Searches for RA, Dec, and discovery date information from the command line and the light curves' RA and Dec columns. Saves this information to snlist.txt if found.
• Outputs files to the directory specified by the atclean_input field in config.ini. Files will adhere to the ATClean filename scheme, will be split by filter, and undergo several data preprocessing steps.

column_name_preset config sections in config.ini

Visit Setup in config.ini to read more about these sections.

Arguments

Arguments will override default config file settings if specified.

• First provide the name of the object to convert.
• -p, --preset: Specifies the column name preset from the config file (e.g., atlas, rubin, tess).
  • Type: str
  • Default: atlas
  • Usage: -p rubin or --preset tess
• --mjd0: The start date of the SN in MJD.
  • Type: float
  • Default: None
  • Usage: --mjd0 58800.0
• --ra: The Right Ascension (RA) of the SN in degrees or HMS.
  • Type: float
  • Default: None
  • Usage: --ra 210.91067124999998
• --dec: The Declination (Dec) of the SN in degrees or DMS.
  • Type: float
  • Default: None
  • Usage: --dec 54.31165527777777
• -f, --filenames: Specifies one or more file paths to convert.
  • Type: str (one or more)
  • Required: Yes
  • Usage: -f path/to/sn_lc.txt path/to/control_lc1.txt path/to/control_lc2.txt
• -i, --control_indices: Specifies one or more ordered control indices, each corresponding to a file path in -f.
  • Type: int (one or more)
  • Default: [0]
  • Usage: -i 0 1 2
• --config_file: Specifies the file name of the .ini file with settings for this script.
  • Type: str
  • Default: config.ini
  • Usage: --config_file config.ini
• -o, --overwrite: If specified, existing files with the same name will be overwritten.
  • Type: bool
  • Default: False
  • Usage: -o or --overwrite

Example commands

• Convert a single ATLAS light curve: convert_to_atclean.py 2023ixf -f path/to/2023ixf_lc.txt -o
• Convert a single Rubin light curve CSV file: convert_to_atclean.py 2023ixf -f path/to/2023ixf_lc.csv -p rubin -o
• Specify RA and Dec: convert_to_atclean.py 2023ixf -f path/to/2023ixf_lc.csv -p rubin -o --ra 210.91067124999998 --dec 54.31165527777777
• Convert ATLAS SN and control light curves: convert_to_atclean.py 2023ixf -f path/to/2023ixf_lc.txt path/to/2023ixf_control1.txt path/to/2023ixf_control2.txt -i 0 1 2 -o

Use the following runnable command to convert an example Rubin light curve at extern/rubin_example.csv to ATClean-readable format:

./convert_to_atclean.py example_SN -f extern/rubin_example.csv -p rubin -o

clean.py

This script allows you to run a series of customized cuts and binning on ATClean-readable files located in directory specified by the atclean_input field in config.ini. All cleaned and binned light curve, plots, and an informative output README file will be storied in the directory specified by the output field in config.ini.

Configuration for each cut, as well as an option to add custom cuts for other columns, is located in config.ini. Arguments are used to determine which cuts are actually applied.

True uncertainties estimation: uncert_est config section in config.ini

Click to read more

We also attempt to account for an extra noise source in the data by estimating the true typical uncertainty, deriving the additional systematic uncertainty, and applying this extra noise to a new uncertainty column. This new uncertainty column will be used in the cuts following this section.

• temp_x2_max_value: A temporary, very high chi-square cut value used to eliminate the most egregious outliers from the data. This is an initial step in uncertainty estimation to ensure grossly incorrect data points are removed.

To read more about how we calculate the true uncertainties, please refer to Section 3.1.2 of our paper (link at the top of this documentation).

Uncertainty cut: uncert_cut config section in config.ini

Click to read more

The uncertainty cut flags any measurements with $\delta \mu \text{Jy}$ > `max_value` (currently set to a default value of 160, or the the typical uncertainty of bright stars just below the saturation limit).

• max_value: The maximum allowable value for the uncertainties (duJy column). Measurements with uncertainties above this threshold will be flagged.

• flag: The flag value in hex assigned to measurements that exceed the maximum allowable uncertainty.

PSF chi-square cut: x2_cut config section in config.ini

Click to read more

The PSF chi-square cut flags any measurements with $\chi^2_\text{PSF}$ > `max_value`.

• max_value: The maximum allowable value for the PSF chi-squares (chi/N column). Measurements with a chi-square above this threshold will be flagged.

• flag: The flag value in hex assigned to measurements that exceed the maximum allowable chi-square.

We use two factors, contamination and loss, to analyze the effectiveness of a given PSF chi-square cut for the target SN, with $|\frac{\text{flux}}{\text{dflux}}|$ as the deciding factor of what constitutes a good measurement vs. a bad measurement. We calculate contamination and loss for a range of possible chi-square cuts in order to examine the relative effectiveness of the chosen cut:

• snr_bound: $|\frac{\text{flux}}{\text{dflux}}|$ bound that determines a "good" measurement vs. "bad" measurement.

• min_cut: The minimum chi-square value, inclusive, for which to calculate contamination and loss.

• min_cut: The maximum chi-square value, inclusive, for which to calculate contamination and loss.

• cut_step: The step size for the [min_cut, max_cut] range.

• use_pre_mjd0_lc: If set to True, we use the pre-MJD0 light curve to calculate contamination and loss. If set to False (recommended), we instead use the control light curves.

To read more about how we calculate contamination and loss, please refer to Section 3.1.3 of our paper (link at the top of this documentation).

Warning: For very bright SNe, the chi-square values may increase during the SN even for good measurements due to imperfection in PSF fitting. Therefore, we recommend that the user double-check the chi-square values or the output plots to verify that the cut is working as intended, and change max_value if needed.

Control light curve cut: controls_cut config section in config.ini

Click to read more

The control light curve cut uses a set of quality control light curves to determine the reliability of each SN measurement. Since we know that control light curve flux must be consistent with 0, any lack of consistency may indicate something wrong with the SN measurement at this epoch.

Note that this cut may not greatly affect certain SNe depending on the quality of the light curve. Its main purpose is to account for inconsistent flux in the case of systematic interference from bright objects, etc. that also affect the area around the SN. Therefore, normal SN light curves will usually see <1%-2% of data flagged as bad in this cut.

We perform a $3\sigma$-clipped average on each epoch across control light curves. Then, based on the calculated statistics of each epoch, we either flag that epoch as "bad" or "questionable" across the SN and control light curves, or leave it unflagged.

• bad_flag: The flag value in hex assigned to measurements identified as bad.

• questionable_flag: The flag value in hex assigned to measurements identified as questionable.

The following criteria are used on the calculated statistics of each epoch, not the statistics of the individual measurements, to identify SN measurements in that epoch as "bad" or "questionable". Measurements violating all of the following thresholds will be flagged as "bad".

• x2_max: The chi-square threshold for an epoch.

  • x2_flag: The flag value in hex assigned to epochs with chi-square values exceeding x2_max.

• snr_max: The $|\frac{\text{flux}}{\text{dflux}}|$ threshold for an epoch.

  • snr_flag: The flag value in hex assigned to epochs with $|\frac{\text{flux}}{\text{dflux}}|$ exceeding snr_max.

• Nclip_max: The threshold of clipped control measurements for an epoch.

  • Nclip_flag: The flag value in hex assigned to epochs with the number of clipped control measurements exceeding Nclip_max.

• Ngood_min: The threshold of good control measurements for an epoch.

  • Ngood_flag: The flag value in hex assigned to epochs with the number of good control measurements falling below Ngood_min.

To read more about how we determine which epochs are "bad" or "questionable", please refer to Section 3.1.4 of our paper (link at the top of this documentation).

Additional static cuts: custom cuts config section in config.ini

Click to read more

Custom cuts run during cleaning allow you to define additional filtering criteria based on specific column values. We provide an example template for specifying custom cuts below.

[example_cut]
    column: duJy
    max_value: 160
    min_value: None
    flag: 0x1000000

Note that the section title of the cut (in the above example, [example_cut]) must end in _cut in order to be properly parsed by clean.py.

• column: The name of the column in the data that the custom cut will be applied to.

• max_value: The maximum allowable value for the specified column. Measurements with values exceeding this threshold will be flagged. This parameter can be set to None if no upper limit is required.

• min_value: The minimum allowable value for the specified column. Measurements with values falling below this threshold will be flagged. This parameter can be set to None if no lower limit is required.

• flag: The flag value in hex assigned to measurements that do not meet the defined criteria. Flag values of 0x1000000 and above are available for use in custom cuts.

Averaging light curves and the bad day cut: averaging config section in config.ini

Click to read more

Our goal with the averaging procedure is to identify and cut out bad days. For both the SN and control locations, we bin the light curve and perform a $3\sigma$-clipped average on the unflagged measurements in each bin. We use the calculated average and its error as flux and uncertainty values in the averaged light curves.

• bad_flag: The flag value in hex assigned to measurements identified as "bad".

• ixclip_flag: The flag value in hex assigned to bins where one or more measurements have been clipped during the $3\sigma$-clipped average.

• smallnum_flag: The flag value in hex assigned to bins with 2 or less unflagged measurements in a bin.

• mjd_bin_size: The bin size in days.

The following criteria are used on the calculated statistics of each bin, not the statistics of the individual measurements, to identify measurements in that bin as "bad". Measurements violating all of the following thresholds will be flagged as "bad".

• x2_max: The chi-square threshold for a bin.

• Nclip_max: The threshold of clipped measurements for a bin.

• Ngood_min: The threshold of good measurements for a bin.

To read more about how we determine which bins are "bad", please refer to Section 3.3 of our paper (link at the top of this documentation).

Arguments

Arguments will override default config file settings if specified.

• First provide TNS name(s) of the object(s) to clean.
• -p, --preset: Specifies the column name preset from the config file (e.g., atlas, rubin, tess).
  • Type: str
  • Default: atlas
  • Usage: -p rubin or --preset tess
• --sninfo_file: Specifies the SN info file name.
  • Type: str
  • Default: None (i.e., the sninfo_filename field in config.ini)
  • Usage: --sninfo_file sninfo.txt
• --config_file: Specifies the file name of the .ini file with settings for this script.
  • Type: str
  • Default: config.ini
  • Usage: --config_file config.ini
• -o, --overwrite: If specified, existing files with the same name will be overwritten.
  • Type: bool
  • Default: False
  • Usage: -o or --overwrite
• --plot: If specified, a summary PDF file of diagnostic plots will be saved.
  • Type: bool
  • Default: False
  • Usage: --plot
• --filters: Specifies a comma-separated list of filters to clean.
  • Type: str
  • Default: None (i.e., the filters field in config.ini)
  • Usage: --filters c,o
• --mjd0: The start date of the SN in MJD.
  • Type: float
  • Default: None (i.e., reference the SN info file or query TNS)
  • Usage: --mjd0 58800.0
• -n, --num_controls: The number of control light curves to clean per SN.
  • Type: int
  • Default: None (i.e., the num_controls field in config.ini)
  • Usage: -n 5 or --num_controls 5
• -t, --template_correction: If specified, apply automatic ATLAS template change correction.
  • Type: bool
  • Default: False
  • Usage: -t or --template_correction
• -e, --uncert_est: If specified, apply true uncertainty estimation.
  • Type: bool
  • Default: False
  • Usage: -e or --uncert_est
• -u, --uncert_cut: If specified, apply uncertainty cut.
  • Type: bool
  • Default: False
  • Usage: -u or --uncert_cut
• -x, --x2_cut: If specified, apply chi-square cut.
  • Type: bool
  • Default: False
  • Usage: -x or --x2_cut
• -c, --controls_cut: If specified, apply control light curve cut.
  • Type: bool
  • Default: False
  • Usage: -c or --controls_cut
• -g, --averaging: If specified, average light curves and cut bad days.
  • Type: bool
  • Default: False
  • Usage: -g or --averaging
• -m, --mjd_bin_size: Specifies the MJD bin size in days for averaging.
  • Type: float
  • Default: None (i.e., the mjd_bin_size field in config.ini if needed)
  • Usage: -m 1.0 or --mjd_bin_size 1.0
• --custom_cuts: If specified, scan the config file for custom cuts.
  • Type: bool
  • Default: False
  • Usage: --custom_cuts

Filename scheme

• All cleaned files will be storied in the directory specified by the output field in config.ini.
• We denote the "cleaned" status of a non-binned file by adding a "clean" suffix to the original filename.
  • SN filenames: [ATCLEAN_INPUT]/[TNSNAME]/[TNSNAME].[FILTER].clean.lc.txt
    • Examples: /path/to/atclean_input/2020lse/2020lse.c.clean.lc.txt and /path/to/atclean_input/2020lse/2020lse.o.clean.lc.txt
  • Control filenames: [ATCLEAN_INPUT]/[TNSNAME]/controls/[TNSNAME]_i[CONTROL_INDEX].[FILTER].clean.lc.txt
    • Examples: /path/to/atclean_input/2020lse/controls/2020lse_i001.c.clean.lc.txt and /path/to/atclean_input/2020lse/controls/2020lse_i001.o.clean.lc.txt
• We denote binned light curves by adding the MJD bin size to the original filename.
  • Binned SN filenames: [ATCLEAN_INPUT]/[TNSNAME]/[TNSNAME].[FILTER].[MJD_BIN_SIZE]days.lc.txt
    • Examples: /path/to/atclean_input/2020lse/2020lse.c.1.00days.lc.txt and /path/to/atclean_input/2020lse/2020lse.o.1.00days.lc.txt
  • Binned control filenames: [ATCLEAN_INPUT]/[TNSNAME]/controls/[TNSNAME]_i[CONTROL_INDEX].[FILTER].[MJD_BIN_SIZE]days.lc.txt
    • Examples: /path/to/atclean_input/2020lse/controls/2020lse_i001.c.1.00days.lc.txt and /path/to/atclean_input/2020lse/controls/2020lse_i001.o.1.00days.lc.txt

Example commands

• Apply the uncertainty cut to a single SN and its control light curves: ./clean.py 2020lse -u -o
• Apply the uncertainty cut to a single SN and its control light curves, then generate a summary PDF file of diagnostic plots: ./clean.py 2020lse -u -o --plot
• Specify the number of control light curves to clean (can be 0): ./clean.py 2020lse -u --num_controls 5 -o
• Apply all custom cuts listed in config.ini: ./clean.py 2020lse --custom_cuts -o
• Apply the uncertainty cut, true uncertainties estimation, chi-square cut, control light curve cut, and all custom cuts listed in config.ini: ./clean.py 2020lse -u -e -x -c --custom_cuts -o
• Apply the uncertainty cut, true uncertainties estimation, chi-square cut, control light curve cut, and all custom cuts listed in config.ini, then bin the light curves and apply the bad day cut: ./clean.py 2020lse -u -e -x -c -g --custom_cuts -o

plotloop.py

This script allows you to generate a summary PDF file of diagnostic plots based on existing cleaned and/or averaged light curves.

To instead generate this summary PDF file while running the cleaning script, add --plot to your clean.py command.

Arguments

Arguments will override default config file settings if specified.

• First provide TNS name(s) of the object(s) to download.
• -p, --preset: Specifies the column name preset from the config file (e.g., atlas, rubin, tess).
  • Type: str
  • Default: atlas
  • Usage: -p rubin or --preset tess
• --config_file: Specifies the file name of the .ini file with settings for this script.
  • Type: str
  • Default: config.ini
  • Usage: --config_file config.ini
• -o, --overwrite: If specified, existing PDF files with the same name will be overwritten.
  • Type: bool
  • Default: False
  • Usage: -o or --overwrite
• --filters: Specifies a comma-separated list of filters to plot. Each filter gets its own PDF file.
  • Type: str
  • Default: None (i.e., the filters field in config.ini)
  • Usage: --filters c,o
• -n, --num_controls: The number of control light curves to load and plot per SN.
  • Type: int
  • Default: None (i.e., the num_controls field in config.ini)
  • Usage: -n 5 or --num_controls 5
• --mjd0: The start date of the SN in MJD.
  • Type: float
  • Default: None (i.e., reference the SN info file or query TNS)
  • Usage: --mjd0 58800.0
• -e, --uncert_est: If specified, plot the true uncertainty estimation.
  • Type: bool
  • Default: False
  • Usage: -e or --uncert_est
• -u, --uncert_cut: If specified, plot the uncertainty cut.
  • Type: bool
  • Default: False
  • Usage: -u or --uncert_cut
• -x, --x2_cut: If specified, plot the chi-square cut.
  • Type: bool
  • Default: False
  • Usage: -x or --x2_cut
• -c, --controls_cut: If specified, plot the control light curve cut.
  • Type: bool
  • Default: False
  • Usage: -c or --controls_cut
• -g, --averaging: If specified, plot the bad day cut and the averaged light curves.
  • Type: bool
  • Default: False
  • Usage: -g or --averaging
• --custom_cuts: If specified, scan the config file for custom cuts and plot them.
  • Type: bool
  • Default: False
  • Usage: --custom_cuts

Example commands

• Generate default plots of SN 2019vxm and its control light curves: ./plotloop.py 2019vxm -o
• Generate default plots of SN 2019vxm without its control light curves: ./plotloop.py 2019vxm -o --num_controls 0
• Generate defaults plots of SN 2019vxm and its control light curves, as well as plots of the true uncertainties estimation, the uncertainty cut, the chi-square cut, the control light curve cut, and the bad day cut: ./plotloop.py 2019vxm -o -e -u -x -c -g

zip.py

Zip multiple processed light curve files for one or more supernovae (SNe) into one or more .zip archives.

This script is intended to package all relevant files for a given list of supernovae, using the input and output directories specified in config.ini. It can output either individual zip files per SN or a single bulk archive.

Arguments

• First provide TNS name(s) of the object(s) to zip.
• --config_file: Specifies the file name of the .ini file with settings for this script.
  • Type: str
  • Default: config.ini
  • Usage: --config_file config.ini
• -b, --bulk: Store multiple SN folders within one zip file.
  • Type: bool
  • Default: False
  • Usage: -b or --bulk

Example commands

• Zip a single SN: ./zip.py SN2020abc
• Zip multiple SNe individually: ./zip.py SN2020abc SN2021xyz
• Zip multiple SNe into one batch file: ./zip.py SN2020abc SN2021xyz --bulk

unzip.py

Unzip previously packaged light curve files and restore them into the appropriate input or output directories based on file naming conventions.

This script complements zip.py by placing each file in the correct location depending on whether it is a raw (input) or processed (output) light curve.

Only files with the following extensions are included in the zip archives: .txt, .pdf, .jpg, .png, .md, .ipynb, .py, .json, .ini

Arguments

• First paste in the file paths of each zip file (generated by zip.py) to unzip.
• --config_file: Specifies the file name of the .ini file with settings for this script.
  • Type: str
  • Default: config.ini
  • Usage: --config_file config.ini

Example commands

• Unzip a single archive: ./unzip.py /path/to/SN2020abc.zip
• Unzip multiple archives: ./unzip.py /path/to/SN2020abc.zip /path/to/SN2021xyz.zip

step1_generate_sim_tables.py

WIP

Configuration file: step1_settings.json

WIP

Arguments

WIP

Example commands

WIP

step2_generate_detec_tables.py

WIP

Arguments

WIP

Example commands

./step2_generate_detec_tables.py 2019vxm o gaussian --sigma_kerns 1.0 1.31607 1.73205 2.27951 3.0 --num_controls 7 --mjd_ranges "[[57233.5, 57328.5],[57466.5, 57535.5],[57914.5, 58122.5],[58206.5, 58486.5],[58594.5, 58836.5],[58910.5, 59212.5],[59304.5, 59564.5],[59637.5, 59915.5],[60023.5, 60309.5],[60387.5, 60671.5],[60727.5, 60770.5]]" 

WIP

step3_calculate_efficiencies.py

WIP

Arguments

WIP

Example commands

./step3_calculate_efficiencies.py 2019vxm o gaussian --sigma_kerns 1.0 1.31607 1.73205 2.27951 3.0 --num_controls 7 --mjd_ranges "[[57233.5, 57328.5],[57466.5, 57535.5],[57914.5, 58122.5],[58206.5, 58486.5],[58594.5, 58836.5],[58910.5, 59212.5],[59304.5, 59564.5],[59637.5, 59915.5],[60023.5, 60309.5],[60387.5, 60671.5],[60727.5, 60770.5]]"

WIP

Jupyter Notebooks

clean.ipynb

Perform the same procedures that clean.py can, in a more user-friendly and step-by-step format. This notebook works on a single ATClean-readable SN and its control light curves. After cleaning and binning, the light curves and an informative output README are saved to user-specified output directories.

atlas_template_correction.ipynb

WIP

simdetec_analysis.ipynb

WIP

Dependencies

• Python 3.11.6 or higher
• typing
• os
• sys
• requests
• argparse
• configparser
• math
• pandas
• numpy
• getpass
• astropy
• abc
• re
• time
• json
• io
• copy
• itertools
• scipy
• matplotlib