Roman-Supernova-PIT · rknop · Aug 26, 2025 · Aug 20, 2025 · Aug 21, 2025 · Aug 22, 2025
@@ -0,0 +1 @@
+Massive refactor to use snappl for getting images and objects, removing phrosty explicit OU2024 dependence.
@@ -5,7 +5,7 @@ Development
 ===========
 
 .. contents::
-               
+
 
 If you're one of the phrosty developers, or are otherwise interested in contributing, here is some useful information.
 
@@ -20,7 +20,7 @@ Running Tests
 
 Running all of the tests requires an NVIDIA GPU with enough memory.  We are able to run them on 40GB NVIDIA GPUs, a GPU with only 12GB is not enough.  (TODO: figure out the actual cutoff.)
 
-To run  the tests, forst make sure you've set up your environment and pulled down the necessary docker images as described in :ref:`phrosty installation prerequisites<phrosty-installation-prerequisites>`.  
+To run  the tests, forst make sure you've set up your environment and pulled down the necessary docker images as described in :ref:`phrosty installation prerequisites<phrosty-installation-prerequisites>`.
 
 If you haven't already, get a copy of phrosty::
 
@@ -34,7 +34,7 @@ Make a couple of necessary directories::
 
   mkdir dia_out_dir
   mkdir phrosty_temp
-  
+
 Run the container with::
 
   docker run --gpus=all -it \
@@ -74,7 +74,7 @@ Run the container with::
 
 Once inside the container, cd into ``/home/phrosty/phrosty/tests`` and run::
 
-  SNPIT_CONFIG=phrosty_test_config.yaml pytest -v
+  pytest -v
 
 
 Manually running a test lightcurve
@@ -126,7 +126,7 @@ With these directories in place, run a container with::
     /bin/bash
 
 If you placed any of the new directories anywhere other than underneath your current working directory, modify the ``source=...`` parts of the command above to reflect that.
-    
+
 Inside the container, cd into ``/home/phrosty`` and try running::
 
   nvidia-smi

@@ -58,15 +58,15 @@ Next, try running::
 
   SNPIT_CONFIG=phrosty/tests/phrosty_test_config.yaml python phrosty/pipeline.py --help | less
 
-You should see all the options you can pass to phrosty.  There are a lot, because there are (verbose) options for everything that's in the config file.  Press ``q`` to get out of ``less``.
+You should see all the options you can pass to phrosty.  There are a lot, because there are (verbose) options for everything that's in the config file.  The options you need to think about most are at the top.  Press ``q`` to get out of ``less``.
 
 Try running::
 
   SNPIT_CONFIG=phrosty/tests/phrosty_test_config.yaml python phrosty/pipeline.py \
+    --oc ou2024 \
     --oid 20172782 \
-    --ra 7.551093401915147 \
-    --dec -44.80718106491529 \
     -b Y106 \
+    --ic ou2024 \
     -t phrosty/tests/20172782_instances_templates_1.csv \
     -s phrosty/tests/20172782_instances_science_2.csv \
     -p 3 -w 3 \
@@ -182,16 +182,16 @@ The main Python executable for running the pipeline is ``phrosty/phrosty/pipelin
 
   SNPIT_CONFIG=phrosty/examples/perlmutter/phrosty_config.yaml python phrosty/phrosty/pipeline.py --help
 
-to see how it works, and to see what the various parameters you can specify are.
+to see how it works, and to see what the various parameters you can specify are.  The output will be long, becasue everything that's in the config file is included as something you can override on the command line.  The arguments near the top are the ones you're more likely to want to think about.  You might want to pipe the output of this ``-help`` into ``less`` so you can see what's going on.
 
 Run this on your example lightcurve with::
 
   python phrosty/phrosty/pipeline.py \
     -c phrosty/examples/perlmutter/phrosty_config.yaml \
+    --oc ou2024 \
     --oid 20172782 \
-    -r 7.551093401915147 \
-    -d -44.80718106491529 \
     -b R062 \
+    --ic ou2024 \
     -t phrosty/examples/perlmutter/20172782_instances_templates_1.csv \
     -s phrosty/examples/perlmutter/20172782_instances_science_2.csv \
     -p 3 \
@@ -222,20 +222,22 @@ When developing/debugging the pipeline, it's useful to run with a profiler, so y
     --trace=cuda,nvtx,cublas,cusparse,cudnn,cudla,cusolver,opengl,openacc,openmp,osrt,mpi,nvvideo,vulkan,python-gil \
     python phrosty/phrosty/pipeline.py \
       -c phrosty/examples/perlmutter/phrosty_config.yaml \
+      --oc ou2024 \
       --oid 20172782 \
-      -r 7.551093401915147 \
-      -d -44.80718106491529 \
       -b R062 \
+      --ic ou2024 \
       -t phrosty/examples/perlmutter/20172782_instances_templates_1.csv \
       -s phrosty/examples/perlmutter/20172782_instances_science_2.csv \
       -p 3 \
       -w 3
 
-*Ideally*, this would create a file ``report1.nsys-rep``, but something about that is broken; I'm not sure what.  It does create a file ``report1.qdstrm``, which you can then somehow somewhere else convert to a ``.nsys-rep`` file.  On a Linux system, if you've installed the ``nsight-compute`` and ``nsight-systems`` packages (see `Nvidia's Nsight Systems installation guide <https://docs.nvidia.com/nsight-systems/InstallationGuide/index.html)>`_), you can download the ``.qdstrm`` file to your system and run::
+*Ideally*, this would create a file ``report1.nsys-rep`` (or ``report2.nsys-rep``, or higher numbers based on what files are already in the directory), but something about that is broken; I'm not sure what.  If that file is created, be happy.  If not, should leave behind a file ``report<n>.qdstrm``.  You can couple that file to another system and manually convert it to a ``.nsys-rep`` file.  On a Linux system, if you've installed the ``nsight-compute`` and ``nsight-systems`` packages (see `Nvidia's Nsight Systems installation guide <https://docs.nvidia.com/nsight-systems/InstallationGuide/index.html)>`_), you can download the ``.qdstrm`` file to your system and run::
 
-  /opt/nvidia/nsight-systems/2024.4.2/host-linux-x64/QdstrmImporter -i <name>.qdstrm
+  /opt/nvidia/nsight-systems/2025.3.2/host-linux-x64/QdstrmImporter -i <name>.qdstrm
 
-where ``<name>.qstrm`` is the file you downloaded.  (Note that the directory may have something other than ``2024.4.2`` in it, depending on what version you've installed.  For best comptibility with the version of Nsight in the current (as of this writing) snpit docker image, I recommend trying to install something close to ``nsight-compute-2024.3.1`` and  ``nsight-systems-2024.4.2``; exactly what is avialable seems to vary with time.)  This should produce a file ``<name>.nsys-rep``.  Then, on your local desktop, run::
+where ``<name>.qstrm`` is the file you downloaded.  (Note that the directory may have something other than ``2025.3.2`` in it, depending on what version you've installed.  For best comptibility with the version of Nsight in the current (as of this writing) snpit docker image, I recommend trying to install something close to ``nsight-compute-2025.3.0`` and  ``nsight-systems-2025.3.2``; exactly what is avialable seems to vary with time.)  This should produce a file ``<name>.nsys-rep``.
+
+Once you, somehow, have a ``<name>.nsys-rep`` file, copy it down to your local desktop if it's not there already, and run::
 
   nsys-ui <name>.nsys-rep
 
@@ -304,6 +306,75 @@ and, ideally, there should be no lines anywhere in the file with ``ERROR`` near
 
 Note that ``/lc_out_dir/...`` is the absolute path _inside_ the container; it maps to ``lc_out_dir/...`` underneath your working directory where you ran ``sbatch``.  You will find the lightcurve in that ``.csv`` file.  There will also be a number of files written to the ``dia_out_dir`` directory.
 
+Running on a HPC system that uses apptainer/singularity
+-------------------------------------------------------
+
+Everything below assumes that the apptainer/singulariy executable is named ``apptainer``.  That's the newer name; the older name was singlarity.  If your system doesn't have ``apptainer``, try running ``singularity`` in its place.
+
+Set up the environment
+^^^^^^^^^^^^^^^^^^^^^^
+
+You need to import the roman SNPIT docker environment into singularity.  First, cd to a directory that will be fast to read or write to (this may be a scratch partition or some such on your cluster) and run::
+
+  apptainer pull docker://rknop/roman-snpit-env:cuda-dev
+
+(You can also try pulling from ``registry.nersc.gov``, but to do that you'll have to figure out how to log into the repository with ``apptainer``; try something like ``apptainer remote login --username <yourusername> docker://registry.nersc.gov``.)
+
+That will take a long time.  When it's done, there should be a file::
+
+  roman-snpit-env_cuda-dev.sif
+
+Pick a directory to work in; I will henceforth call this your "parent" directory.  Make some necessary directories here::
+
+  mkdir phrosty_temp
+  mkdir dia_out_dir
+  mkdir lc_out_dir
+  mkdir ou2024_images
+
+Copy the data
+^^^^^^^^^^^^^
+
+This example assumes you're just going to use the data available in ``photometry_test_data``.  Pull it down with::
+
+  git clone https://github.com/Roman-Supernova-PIT/photometry_test_data
+
+If you want to run on more than the images that are there, figure out which images you're going to run on.  Look at the ``.csv`` files you'll be feeding to phrosty.  For all of the images in those files, copy the data file from the host system to ``ou2024_images`` (preserving the relative path that's in the ``.csv`` files.)  On Perlmutter, you can find the files underneath::
+
+  /dvs_ro/cfs/cdirs/lsst/shared/external/roman-desc-sims/Roman_data/RomanTDS/images/simple_model/
+
+
+Check out the code
+^^^^^^^^^^^^^^^^^^
+
+In a directory which I will henceforth call your "parent" directory, get a copy of phrosty::
+
+  git clone https://github.com/Roman-Supernova-PIT/phrosty.git
+
+Run the container
+^^^^^^^^^^^^^^^^^
+
+Do::
+
+  apptainer shell --nv \
+    --bind $PWD:/home \
+    --bind $PWD/photometry_test_data:/photometry_test_data \
+    --bind $PWD/dia_out_dir:/dia_out_dir \
+    --bind $PWD/lc_out_dir:/lc_out_dir \
+    --bind $PWD/phrosty_temp:/phrosty_temp \
+    --env LD_LIBRARY_PATH=/usr/lib64:/usr/lib/x86_64-linux-gnu:/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs \
+    --env PYTHONPATH=/roman_imsim \
+    --env OPENBLAS_NUM_THREADS=1 \
+    --env MKL_NUM_THREADS=1 \
+    --env NUMEXPR_NUM_THREADS=1 \
+    --env OMP_NUM_THREADS=1 \
+    --env VECLIB_MAXIMUM_THREADS=1 \
+    --env TERM=xterm \
+    roman-snpit-env_cuda-dev.sif
+
+
+If you ran the ``apptainer pull`` command above in a different place from where you are now, replaced ``roman-snpit-env_cuda-dev.sif`` above with the full path to that ``.sif`` file.
+
+
 
 Phrosty Functionality
 =====================

@@ -1,54 +1,54 @@
 path pointing sca mjd band
-simple_model/R062/35083/Roman_TDS_simple_model_R062_35083_8.fits.gz 35083 8 62455.174 R062
-simple_model/R062/35088/Roman_TDS_simple_model_R062_35088_2.fits.gz 35088 2 62455.19 R062
-simple_model/R062/35857/Roman_TDS_simple_model_R062_35857_3.fits.gz 35857 3 62465.187 R062
-simple_model/R062/36242/Roman_TDS_simple_model_R062_36242_5.fits.gz 36242 5 62470.187 R062
-simple_model/R062/36626/Roman_TDS_simple_model_R062_36626_15.fits.gz 36626 15 62475.185 R062
-simple_model/R062/37391/Roman_TDS_simple_model_R062_37391_6.fits.gz 37391 6 62485.168 R062
-simple_model/R062/37395/Roman_TDS_simple_model_R062_37395_16.fits.gz 37395 16 62485.182 R062
-simple_model/R062/37396/Roman_TDS_simple_model_R062_37396_8.fits.gz 37396 8 62485.185 R062
-simple_model/R062/38155/Roman_TDS_simple_model_R062_38155_11.fits.gz 38155 11 62495.15 R062
-simple_model/R062/38160/Roman_TDS_simple_model_R062_38160_4.fits.gz 38160 4 62495.166 R062
-simple_model/R062/38535/Roman_TDS_simple_model_R062_38535_15.fits.gz 38535 15 62500.134 R062
-simple_model/R062/38920/Roman_TDS_simple_model_R062_38920_3.fits.gz 38920 3 62505.134 R062
-simple_model/R062/39680/Roman_TDS_simple_model_R062_39680_15.fits.gz 39680 15 62515.102 R062
-simple_model/R062/40060/Roman_TDS_simple_model_R062_40060_18.fits.gz 40060 18 62520.086 R062
-simple_model/R062/40820/Roman_TDS_simple_model_R062_40820_10.fits.gz 40820 10 62530.053 R062
-simple_model/R062/40825/Roman_TDS_simple_model_R062_40825_15.fits.gz 40825 15 62530.07 R062
-simple_model/R062/41200/Roman_TDS_simple_model_R062_41200_1.fits.gz 41200 1 62535.037 R062
-simple_model/R062/41205/Roman_TDS_simple_model_R062_41205_12.fits.gz 41205 12 62535.053 R062
-simple_model/R062/42356/Roman_TDS_simple_model_R062_42356_18.fits.gz 42356 18 62550.04 R062
-simple_model/R062/43122/Roman_TDS_simple_model_R062_43122_13.fits.gz 43122 13 62560.027 R062
-simple_model/R062/43507/Roman_TDS_simple_model_R062_43507_17.fits.gz 43507 17 62565.027 R062
-simple_model/R062/43893/Roman_TDS_simple_model_R062_43893_1.fits.gz 43893 1 62570.029 R062
-simple_model/R062/44278/Roman_TDS_simple_model_R062_44278_17.fits.gz 44278 17 62575.029 R062
-simple_model/R062/44279/Roman_TDS_simple_model_R062_44279_9.fits.gz 44279 9 62575.032 R062
-simple_model/R062/44664/Roman_TDS_simple_model_R062_44664_12.fits.gz 44664 12 62580.032 R062
-simple_model/R062/44669/Roman_TDS_simple_model_R062_44669_5.fits.gz 44669 5 62580.048 R062
-simple_model/R062/45054/Roman_TDS_simple_model_R062_45054_14.fits.gz 45054 14 62585.048 R062
-simple_model/R062/47004/Roman_TDS_simple_model_R062_47004_7.fits.gz 47004 7 62610.128 R062
-simple_model/R062/47394/Roman_TDS_simple_model_R062_47394_8.fits.gz 47394 8 62615.144 R062
-simple_model/R062/47784/Roman_TDS_simple_model_R062_47784_5.fits.gz 47784 5 62620.16 R062
-simple_model/R062/48174/Roman_TDS_simple_model_R062_48174_11.fits.gz 48174 11 62625.176 R062
-simple_model/R062/48943/Roman_TDS_simple_model_R062_48943_9.fits.gz 48943 9 62635.174 R062
-simple_model/R062/48948/Roman_TDS_simple_model_R062_48948_3.fits.gz 48948 3 62635.19 R062
-simple_model/R062/49333/Roman_TDS_simple_model_R062_49333_1.fits.gz 49333 1 62640.19 R062
-simple_model/R062/50102/Roman_TDS_simple_model_R062_50102_2.fits.gz 50102 2 62650.187 R062
-simple_model/R062/50487/Roman_TDS_simple_model_R062_50487_8.fits.gz 50487 8 62655.187 R062
-simple_model/R062/50871/Roman_TDS_simple_model_R062_50871_11.fits.gz 50871 11 62660.185 R062
-simple_model/R062/51256/Roman_TDS_simple_model_R062_51256_4.fits.gz 51256 4 62665.185 R062
-simple_model/R062/51635/Roman_TDS_simple_model_R062_51635_17.fits.gz 51635 17 62670.166 R062
-simple_model/R062/52015/Roman_TDS_simple_model_R062_52015_14.fits.gz 52015 14 62675.15 R062
-simple_model/R062/52020/Roman_TDS_simple_model_R062_52020_1.fits.gz 52020 1 62675.166 R062
-simple_model/R062/52395/Roman_TDS_simple_model_R062_52395_18.fits.gz 52395 18 62680.134 R062
-simple_model/R062/52400/Roman_TDS_simple_model_R062_52400_2.fits.gz 52400 2 62680.15 R062
-simple_model/R062/52780/Roman_TDS_simple_model_R062_52780_12.fits.gz 52780 12 62685.134 R062
-simple_model/R062/53160/Roman_TDS_simple_model_R062_53160_15.fits.gz 53160 15 62690.118 R062
-simple_model/R062/53540/Roman_TDS_simple_model_R062_53540_18.fits.gz 53540 18 62695.102 R062
-simple_model/R062/53920/Roman_TDS_simple_model_R062_53920_17.fits.gz 53920 17 62700.086 R062
-simple_model/R062/54300/Roman_TDS_simple_model_R062_54300_16.fits.gz 54300 16 62705.07 R062
-simple_model/R062/54305/Roman_TDS_simple_model_R062_54305_15.fits.gz 54305 15 62705.086 R062
-simple_model/R062/54685/Roman_TDS_simple_model_R062_54685_14.fits.gz 54685 14 62710.07 R062
-simple_model/R062/55065/Roman_TDS_simple_model_R062_55065_11.fits.gz 55065 11 62715.053 R062
-simple_model/R062/55440/Roman_TDS_simple_model_R062_55440_8.fits.gz 55440 8 62720.021 R062
-simple_model/R062/55445/Roman_TDS_simple_model_R062_55445_3.fits.gz 55445 3 62720.037 R062
+R062/35083/Roman_TDS_simple_model_R062_35083_8.fits.gz 35083 8 62455.174 R062
+R062/35088/Roman_TDS_simple_model_R062_35088_2.fits.gz 35088 2 62455.19 R062
+R062/35857/Roman_TDS_simple_model_R062_35857_3.fits.gz 35857 3 62465.187 R062
+R062/36242/Roman_TDS_simple_model_R062_36242_5.fits.gz 36242 5 62470.187 R062
+R062/36626/Roman_TDS_simple_model_R062_36626_15.fits.gz 36626 15 62475.185 R062
+R062/37391/Roman_TDS_simple_model_R062_37391_6.fits.gz 37391 6 62485.168 R062
+R062/37395/Roman_TDS_simple_model_R062_37395_16.fits.gz 37395 16 62485.182 R062
+R062/37396/Roman_TDS_simple_model_R062_37396_8.fits.gz 37396 8 62485.185 R062
+R062/38155/Roman_TDS_simple_model_R062_38155_11.fits.gz 38155 11 62495.15 R062
+R062/38160/Roman_TDS_simple_model_R062_38160_4.fits.gz 38160 4 62495.166 R062
+R062/38535/Roman_TDS_simple_model_R062_38535_15.fits.gz 38535 15 62500.134 R062
+R062/38920/Roman_TDS_simple_model_R062_38920_3.fits.gz 38920 3 62505.134 R062
+R062/39680/Roman_TDS_simple_model_R062_39680_15.fits.gz 39680 15 62515.102 R062
+R062/40060/Roman_TDS_simple_model_R062_40060_18.fits.gz 40060 18 62520.086 R062
+R062/40820/Roman_TDS_simple_model_R062_40820_10.fits.gz 40820 10 62530.053 R062
+R062/40825/Roman_TDS_simple_model_R062_40825_15.fits.gz 40825 15 62530.07 R062
+R062/41200/Roman_TDS_simple_model_R062_41200_1.fits.gz 41200 1 62535.037 R062
+R062/41205/Roman_TDS_simple_model_R062_41205_12.fits.gz 41205 12 62535.053 R062
+R062/42356/Roman_TDS_simple_model_R062_42356_18.fits.gz 42356 18 62550.04 R062
+R062/43122/Roman_TDS_simple_model_R062_43122_13.fits.gz 43122 13 62560.027 R062
+R062/43507/Roman_TDS_simple_model_R062_43507_17.fits.gz 43507 17 62565.027 R062
+R062/43893/Roman_TDS_simple_model_R062_43893_1.fits.gz 43893 1 62570.029 R062
+R062/44278/Roman_TDS_simple_model_R062_44278_17.fits.gz 44278 17 62575.029 R062
+R062/44279/Roman_TDS_simple_model_R062_44279_9.fits.gz 44279 9 62575.032 R062
+R062/44664/Roman_TDS_simple_model_R062_44664_12.fits.gz 44664 12 62580.032 R062
+R062/44669/Roman_TDS_simple_model_R062_44669_5.fits.gz 44669 5 62580.048 R062
+R062/45054/Roman_TDS_simple_model_R062_45054_14.fits.gz 45054 14 62585.048 R062
+R062/47004/Roman_TDS_simple_model_R062_47004_7.fits.gz 47004 7 62610.128 R062
+R062/47394/Roman_TDS_simple_model_R062_47394_8.fits.gz 47394 8 62615.144 R062
+R062/47784/Roman_TDS_simple_model_R062_47784_5.fits.gz 47784 5 62620.16 R062
+R062/48174/Roman_TDS_simple_model_R062_48174_11.fits.gz 48174 11 62625.176 R062
+R062/48943/Roman_TDS_simple_model_R062_48943_9.fits.gz 48943 9 62635.174 R062
+R062/48948/Roman_TDS_simple_model_R062_48948_3.fits.gz 48948 3 62635.19 R062
+R062/49333/Roman_TDS_simple_model_R062_49333_1.fits.gz 49333 1 62640.19 R062
+R062/50102/Roman_TDS_simple_model_R062_50102_2.fits.gz 50102 2 62650.187 R062
+R062/50487/Roman_TDS_simple_model_R062_50487_8.fits.gz 50487 8 62655.187 R062
+R062/50871/Roman_TDS_simple_model_R062_50871_11.fits.gz 50871 11 62660.185 R062
+R062/51256/Roman_TDS_simple_model_R062_51256_4.fits.gz 51256 4 62665.185 R062
+R062/51635/Roman_TDS_simple_model_R062_51635_17.fits.gz 51635 17 62670.166 R062
+R062/52015/Roman_TDS_simple_model_R062_52015_14.fits.gz 52015 14 62675.15 R062
+R062/52020/Roman_TDS_simple_model_R062_52020_1.fits.gz 52020 1 62675.166 R062
+R062/52395/Roman_TDS_simple_model_R062_52395_18.fits.gz 52395 18 62680.134 R062
+R062/52400/Roman_TDS_simple_model_R062_52400_2.fits.gz 52400 2 62680.15 R062
+R062/52780/Roman_TDS_simple_model_R062_52780_12.fits.gz 52780 12 62685.134 R062
+R062/53160/Roman_TDS_simple_model_R062_53160_15.fits.gz 53160 15 62690.118 R062
+R062/53540/Roman_TDS_simple_model_R062_53540_18.fits.gz 53540 18 62695.102 R062
+R062/53920/Roman_TDS_simple_model_R062_53920_17.fits.gz 53920 17 62700.086 R062
+R062/54300/Roman_TDS_simple_model_R062_54300_16.fits.gz 54300 16 62705.07 R062
+R062/54305/Roman_TDS_simple_model_R062_54305_15.fits.gz 54305 15 62705.086 R062
+R062/54685/Roman_TDS_simple_model_R062_54685_14.fits.gz 54685 14 62710.07 R062
+R062/55065/Roman_TDS_simple_model_R062_55065_11.fits.gz 55065 11 62715.053 R062
+R062/55440/Roman_TDS_simple_model_R062_55440_8.fits.gz 55440 8 62720.021 R062
+R062/55445/Roman_TDS_simple_model_R062_55445_3.fits.gz 55445 3 62720.037 R062
@@ -1,3 +1,3 @@
 path pointing sca mjd band
-simple_model/R062/35083/Roman_TDS_simple_model_R062_35083_8.fits.gz 35083 8 62455.174 R062
-simple_model/R062/41205/Roman_TDS_simple_model_R062_41205_12.fits.gz 41205 12 62535.053 R062
+R062/35083/Roman_TDS_simple_model_R062_35083_8.fits.gz 35083 8 62455.174 R062
+R062/41205/Roman_TDS_simple_model_R062_41205_12.fits.gz 41205 12 62535.053 R062
@@ -1,2 +1,2 @@
 path pointing sca mjd band
-simple_model/R062/6/Roman_TDS_simple_model_R062_6_17.fits.gz 6 17 62000.04011 R062
+R062/6/Roman_TDS_simple_model_R062_6_17.fits.gz 6 17 62000.04011 R062
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Massive refactor to use snappl for getting images and objects, removing phrosty explicit OU2024 dependence.