45 save intermediate files

[laldoroty]

Checked both keep_intermediate true and keep_intermediate false. Works.

[wmwv]

I feel like I only half-understand the logic flow here of what determines when the images get written out. I think I expected more something that would wrap each step and then optionally write things out. My understanding (incomplete) of this PR is that it gathers all of the data files, keeps them in memory, and then writes them out again. That sounds like (a) a lot of memory usage; and (b) frustrating if the code crashes before it reaches the point where it writes out the intermediate files.

[wmwv]

Does waiting until here mean that all the steps have to have been successful to get any of the images to be written out?

[laldoroty]

Only for the portions on GPU.

[wmwv]

Add a test that exercises these functionality. Somewhere in this repo or in examples at least to capture it.

[wmwv]

A little philosophy here, but I'd prefer the get_psf function to be simpler and not have to know about the config object and that rather the thing calling get_psf explicitly set save_file=True in the call if self.keep_itermediate is set.

I'm open to discussions, but I'd like to not end up where all functions have few arguments but have lots of implicit dependencies on the config object even having an attribute defined.

[wmwv]

And no need to actually edit the commit, but

"[Move intermediate file save location to /pscratch/sd/l/laldoroty..."

should be

"Move intermediate file save location to $SCRATCH/phrosty_intermediate"

I was worried at first that you had hardcoded something.

[laldoroty]

And no need to actually edit the commit, but

"[Move intermediate file save location to /pscratch/sd/l/laldoroty..."

should be

"Move intermediate file save location to $SCRATCH/phrosty_intermediate"

I was worried at first that you had hardcoded something.

...I have no memory of typing this, where is it?

[rknop]

A little philosophy here, but I'd prefer the get_psf function to be simpler and not have to know about the config object and that rather the thing calling get_psf explicitly set save_file=True in the call if self.keep_itermediate is set.

I'm open to discussions, but I'd like to not end up where all functions have few arguments but have lots of implicit dependencies on the config object even having an attribute defined.

So.... in this specific case, get_psf may well need to know about the config object, because one of the things that gets configed is the type of PSF we're using. It's cleaner to have that ugliness buried inside a get_psf function than to have something outside know about that and pass a type variable to get_psf. (But, perhaps, then actual ugliness should be even deeper, in snappl. I would have to look at what's happening here to have an informed opinion.)

[wmwv]

And no need to actually edit the commit, but
"[Move intermediate file save location to /pscratch/sd/l/laldoroty..."
should be
"Move intermediate file save location to $SCRATCH/phrosty_intermediate"
I was worried at first that you had hardcoded something.

...I have no memory of typing this, where is it?

This was in the git commit message.

[wmwv]

A little philosophy here, but I'd prefer the get_psf function to be simpler and not have to know about the config object and that rather the thing calling get_psf explicitly set save_file=True in the call if self.keep_itermediate is set.
I'm open to discussions, but I'd like to not end up where all functions have few arguments but have lots of implicit dependencies on the config object even having an attribute defined.

So.... in this specific case, get_psf may well need to know about the config object, because one of the things that gets configed is the type of PSF we're using. It's cleaner to have that ugliness buried inside a get_psf function than to have something outside know about that and pass a type variable to get_psf. (But, perhaps, then actual ugliness should be even deeper, in snappl. I would have to look at what's happening here to have an informed opinion.)

I'd like to separate things like "detailed configuration of PSF" from pipeline operational choices like whether or not to save an intermediate image. The former seems like a good reason to pass an object with a bunch of stuff, the later seems like an something that is a different kind of config .

[wmwv]

I'm fine with this.

[laldoroty]

And no need to actually edit the commit, but
"[Move intermediate file save location to /pscratch/sd/l/laldoroty..."
should be
"Move intermediate file save location to $SCRATCH/phrosty_intermediate"
I was worried at first that you had hardcoded something.

...I have no memory of typing this, where is it?

This was in the git commit message.

That's so weird. I definitely typed $SCRATCH/phrosty_intermediate. It must have interpreted $SCRATCH as the above hardcoded path.

[wmwv]

That's so weird. I definitely typed $SCRATCH/phrosty_intermediate. It must have interpreted $SCRATCH as the above hardcoded path.

Yes, if you directly typed that on the command-line, e.g., git commit -m "Move $SCRATCH/phrosty_intermediate" without escaping the $, the shell would have expanded $SCRATCH into the full path before passing the argument list to git.