[help] Target statuses are different after transferring files

[wzbillings]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I apologize if this isn't a targets specific question, but I figured it couldn't hurt to ask here.

After a series of other answered help questions here, I got my targets pipeline to successfully run on my institution's Slurm cluster 🥳. After a small celebration I started the process of scp-ing my results off of the cluster so I can do everything else I need to do. I downloaded the files like this, if it matters (I think the -p is important to my issue).
scp -rp r_project_directory_on_cluster my_local_directory
Notably the cluster is Linux and my local machine is Windows.

Here's the dependency network generated by targets::tar_mermaid(TRUE) in an interactive session on the cluster.

graph LR
  style Legend fill:#FFFFFF00,stroke:#000000;
  style Graph fill:#FFFFFF00,stroke:#000000;
  subgraph Legend
    direction LR
    xf1522833a4d242c5([""Up to date""]):::uptodate --- xd03d7c7dd2ddda2b([""Stem""]):::none
    xd03d7c7dd2ddda2b([""Stem""]):::none --- x6f7e04ea3427f824[""Pattern""]:::none
  end
  subgraph Graph
    direction LR
    x83714660cbce9386(["model_data"]):::uptodate --> x4c2862de23e7fd96(["annual_vaccine_table"]):::uptodate
    xa6072ae7a43caffa(["antigenic_distance_raw_data_file"]):::uptodate --> x0e4681951393f524(["antigenic_distance_raw_data"]):::uptodate
    x2e8b0c0c9b08c7be(["model_metadata"]):::uptodate --> x29c21bee92c8aef2(["brms_model_file_names"]):::uptodate
    x29c21bee92c8aef2(["brms_model_file_names"]):::uptodate --> x90e81719f83e0844["brms_model_fit_files"]:::uptodate
    x47a4a6a335617863["brms_model_fits"]:::uptodate --> x90e81719f83e0844["brms_model_fit_files"]:::uptodate
    x2e8b0c0c9b08c7be(["model_metadata"]):::uptodate --> x47a4a6a335617863["brms_model_fits"]:::uptodate
    xfdbd654a1f2948e6(["prepped_cohort_data"]):::uptodate --> x210a25102d7d27e6(["civics_reporting_data"]):::uptodate
    xfdbd654a1f2948e6(["prepped_cohort_data"]):::uptodate --> x4924950c1a58945c(["cleaned_cohort_data"]):::uptodate
    x24daf91e66aafddf(["gap_sd_plot"]):::uptodate --> xb43eae5219050c3c(["combined_metrics_plot"]):::uptodate
    x6e1636bf5a8795a9(["metrics_pc_plot"]):::uptodate --> xb43eae5219050c3c(["combined_metrics_plot"]):::uptodate
    x698be06d7ad5f88a(["distinct_measurements"]):::uptodate --> x4223e83f6c8563f1(["counts_by_study_table"]):::uptodate
    xa44bcdc18fc68732(["distinct_new_participants"]):::uptodate --> x4223e83f6c8563f1(["counts_by_study_table"]):::uptodate
    xa72f89a62d09b7da(["distinct_personyears"]):::uptodate --> x4223e83f6c8563f1(["counts_by_study_table"]):::uptodate
    x698be06d7ad5f88a(["distinct_measurements"]):::uptodate --> x0f9acf7151c5194a(["data_counts"]):::uptodate
    xa44bcdc18fc68732(["distinct_new_participants"]):::uptodate --> x0f9acf7151c5194a(["data_counts"]):::uptodate
    xa72f89a62d09b7da(["distinct_personyears"]):::uptodate --> x0f9acf7151c5194a(["data_counts"]):::uptodate
    x83714660cbce9386(["model_data"]):::uptodate --> xddf50c9e7a4a0a7b(["demographics_table"]):::uptodate
    x83714660cbce9386(["model_data"]):::uptodate --> x698be06d7ad5f88a(["distinct_measurements"]):::uptodate
    x83714660cbce9386(["model_data"]):::uptodate --> xa44bcdc18fc68732(["distinct_new_participants"]):::uptodate
    x83714660cbce9386(["model_data"]):::uptodate --> xa72f89a62d09b7da(["distinct_personyears"]):::uptodate
    x645cc27bc82afdb8(["gap_sd_bootstraps"]):::uptodate --> x1a6e83f46fcebb0b(["gap_sd_bootstrap_file"]):::uptodate
    x7bc9c97777d80aca(["normalized_icc_data"]):::uptodate --> x645cc27bc82afdb8(["gap_sd_bootstraps"]):::uptodate
    x645cc27bc82afdb8(["gap_sd_bootstraps"]):::uptodate --> x24daf91e66aafddf(["gap_sd_plot"]):::uptodate
    x46ae7b79bb331fe3(["normalized_correlation_data"]):::uptodate --> x0eb069b24b54e832(["icc_data"]):::uptodate
    x0eb069b24b54e832(["icc_data"]):::uptodate --> xf8ede9d380dee3b2(["icc_data_nested"]):::uptodate
    xf8ede9d380dee3b2(["icc_data_nested"]):::uptodate --> xfeb99c9ab8909745["icc_results_per_strain_type"]:::uptodate
    xf8ede9d380dee3b2(["icc_data_nested"]):::uptodate --> x5303ef923ea08c5a(["icc_summary"]):::uptodate
    xfeb99c9ab8909745["icc_results_per_strain_type"]:::uptodate --> x5303ef923ea08c5a(["icc_summary"]):::uptodate
    x5303ef923ea08c5a(["icc_summary"]):::uptodate --> xb4a47366b8866ea7(["icc_summary_file"]):::uptodate
    x5303ef923ea08c5a(["icc_summary"]):::uptodate --> xe800f59476a52fa9(["icc_summary_table"]):::uptodate
    x0e4681951393f524(["antigenic_distance_raw_data"]):::uptodate --> xcfbad4104fe5c31f(["joined_data"]):::uptodate
    x4924950c1a58945c(["cleaned_cohort_data"]):::uptodate --> xcfbad4104fe5c31f(["joined_data"]):::uptodate
    x698be06d7ad5f88a(["distinct_measurements"]):::uptodate --> xfaa8ca0b7c12318d(["measurements_per_person_year"]):::uptodate
    x7bc9c97777d80aca(["normalized_icc_data"]):::uptodate --> x6e1636bf5a8795a9(["metrics_pc_plot"]):::uptodate
    xcfbad4104fe5c31f(["joined_data"]):::uptodate --> x83714660cbce9386(["model_data"]):::uptodate
    x83714660cbce9386(["model_data"]):::uptodate --> x77841919662e7158(["model_data_files"]):::uptodate
    x73d419b4476eac11(["model_fitting_seeds_file"]):::uptodate --> xccad9c3796b4eebc(["model_fitting_seeds"]):::uptodate
    xccad9c3796b4eebc(["model_fitting_seeds"]):::uptodate --> x2e8b0c0c9b08c7be(["model_metadata"]):::uptodate
    x33d7a8c39b877444(["nested_model_data"]):::uptodate --> x2e8b0c0c9b08c7be(["model_metadata"]):::uptodate
    x83714660cbce9386(["model_data"]):::uptodate --> x33d7a8c39b877444(["nested_model_data"]):::uptodate
    x83714660cbce9386(["model_data"]):::uptodate --> x46ae7b79bb331fe3(["normalized_correlation_data"]):::uptodate
    x0eb069b24b54e832(["icc_data"]):::uptodate --> x7bc9c97777d80aca(["normalized_icc_data"]):::uptodate
    xc9bc30827a59d139(["UGAFluVac_raw_data"]):::uptodate --> xfdbd654a1f2948e6(["prepped_cohort_data"]):::uptodate
    x83714660cbce9386(["model_data"]):::uptodate --> xdd3ac2d8aae8b204(["strain_names_table"]):::uptodate
    x83714660cbce9386(["model_data"]):::uptodate --> x041afc9f57e7881c(["strain_panel_table"]):::uptodate
    x4e3a18b7b03e484b(["UGAFluVac_raw_data_file"]):::uptodate --> xc9bc30827a59d139(["UGAFluVac_raw_data"]):::uptodate
    x46ae7b79bb331fe3(["normalized_correlation_data"]):::uptodate --> x0bf166d166c2d12b(["vaccine_normalized_correlation_pairplot"]):::uptodate
  end
  classDef uptodate stroke:#000000,color:#ffffff,fill:#354823;
  classDef none stroke:#000000,color:#000000,fill:#94a4ac;
  linkStyle 0 stroke-width:0px;
  linkStyle 1 stroke-width:0px;

Loading

And here's the dependency network in my local directory after scp-ing off the cluster.

graph LR                                                      
  style Legend fill:#FFFFFF00,stroke:#000000;
  style Graph fill:#FFFFFF00,stroke:#000000;
  subgraph Legend
    direction LR
    x2db1ec7a48f65a9b([""Outdated""]):::outdated --- xd03d7c7dd2ddda2b([""Stem""]):::none
    xd03d7c7dd2ddda2b([""Stem""]):::none --- x6f7e04ea3427f824[""Pattern""]:::none
  end
  subgraph Graph
    direction LR
    x83714660cbce9386(["model_data"]):::outdated --> x4c2862de23e7fd96(["annual_vaccine_table"]):::outdated
    xa6072ae7a43caffa(["antigenic_distance_raw_data_file"]):::outdated --> x0e4681951393f524(["antigenic_distance_raw_data"]):::outdated
    x2e8b0c0c9b08c7be(["model_metadata"]):::outdated --> x29c21bee92c8aef2(["brms_model_file_names"]):::outdated
    x29c21bee92c8aef2(["brms_model_file_names"]):::outdated --> x90e81719f83e0844["brms_model_fit_files"]:::outdated
    x47a4a6a335617863["brms_model_fits"]:::outdated --> x90e81719f83e0844["brms_model_fit_files"]:::outdated
    x2e8b0c0c9b08c7be(["model_metadata"]):::outdated --> x47a4a6a335617863["brms_model_fits"]:::outdated
    xfdbd654a1f2948e6(["prepped_cohort_data"]):::outdated --> x210a25102d7d27e6(["civics_reporting_data"]):::outdated
    xfdbd654a1f2948e6(["prepped_cohort_data"]):::outdated --> x4924950c1a58945c(["cleaned_cohort_data"]):::outdated
    x24daf91e66aafddf(["gap_sd_plot"]):::outdated --> xb43eae5219050c3c(["combined_metrics_plot"]):::outdated
    x6e1636bf5a8795a9(["metrics_pc_plot"]):::outdated --> xb43eae5219050c3c(["combined_metrics_plot"]):::outdated
    x698be06d7ad5f88a(["distinct_measurements"]):::outdated --> x4223e83f6c8563f1(["counts_by_study_table"]):::outdated
    xa44bcdc18fc68732(["distinct_new_participants"]):::outdated --> x4223e83f6c8563f1(["counts_by_study_table"]):::outdated
    xa72f89a62d09b7da(["distinct_personyears"]):::outdated --> x4223e83f6c8563f1(["counts_by_study_table"]):::outdated
    x698be06d7ad5f88a(["distinct_measurements"]):::outdated --> x0f9acf7151c5194a(["data_counts"]):::outdated
    xa44bcdc18fc68732(["distinct_new_participants"]):::outdated --> x0f9acf7151c5194a(["data_counts"]):::outdated
    xa72f89a62d09b7da(["distinct_personyears"]):::outdated --> x0f9acf7151c5194a(["data_counts"]):::outdated
    x83714660cbce9386(["model_data"]):::outdated --> xddf50c9e7a4a0a7b(["demographics_table"]):::outdated
    x83714660cbce9386(["model_data"]):::outdated --> x698be06d7ad5f88a(["distinct_measurements"]):::outdated
    x83714660cbce9386(["model_data"]):::outdated --> xa44bcdc18fc68732(["distinct_new_participants"]):::outdated
    x83714660cbce9386(["model_data"]):::outdated --> xa72f89a62d09b7da(["distinct_personyears"]):::outdated
    x645cc27bc82afdb8(["gap_sd_bootstraps"]):::outdated --> x1a6e83f46fcebb0b(["gap_sd_bootstrap_file"]):::outdated
    x7bc9c97777d80aca(["normalized_icc_data"]):::outdated --> x645cc27bc82afdb8(["gap_sd_bootstraps"]):::outdated
    x645cc27bc82afdb8(["gap_sd_bootstraps"]):::outdated --> x24daf91e66aafddf(["gap_sd_plot"]):::outdated
    x46ae7b79bb331fe3(["normalized_correlation_data"]):::outdated --> x0eb069b24b54e832(["icc_data"]):::outdated
    x0eb069b24b54e832(["icc_data"]):::outdated --> xf8ede9d380dee3b2(["icc_data_nested"]):::outdated
    xf8ede9d380dee3b2(["icc_data_nested"]):::outdated --> xfeb99c9ab8909745["icc_results_per_strain_type"]:::outdated
    xf8ede9d380dee3b2(["icc_data_nested"]):::outdated --> x5303ef923ea08c5a(["icc_summary"]):::outdated
    xfeb99c9ab8909745["icc_results_per_strain_type"]:::outdated --> x5303ef923ea08c5a(["icc_summary"]):::outdated
    x5303ef923ea08c5a(["icc_summary"]):::outdated --> xb4a47366b8866ea7(["icc_summary_file"]):::outdated
    x5303ef923ea08c5a(["icc_summary"]):::outdated --> xe800f59476a52fa9(["icc_summary_table"]):::outdated
    x0e4681951393f524(["antigenic_distance_raw_data"]):::outdated --> xcfbad4104fe5c31f(["joined_data"]):::outdated
    x4924950c1a58945c(["cleaned_cohort_data"]):::outdated --> xcfbad4104fe5c31f(["joined_data"]):::outdated
    x698be06d7ad5f88a(["distinct_measurements"]):::outdated --> xfaa8ca0b7c12318d(["measurements_per_person_year"]):::outdated
    x7bc9c97777d80aca(["normalized_icc_data"]):::outdated --> x6e1636bf5a8795a9(["metrics_pc_plot"]):::outdated
    xcfbad4104fe5c31f(["joined_data"]):::outdated --> x83714660cbce9386(["model_data"]):::outdated
    x83714660cbce9386(["model_data"]):::outdated --> x77841919662e7158(["model_data_files"]):::outdated
    x73d419b4476eac11(["model_fitting_seeds_file"]):::outdated --> xccad9c3796b4eebc(["model_fitting_seeds"]):::outdated
    xccad9c3796b4eebc(["model_fitting_seeds"]):::outdated --> x2e8b0c0c9b08c7be(["model_metadata"]):::outdated
    x33d7a8c39b877444(["nested_model_data"]):::outdated --> x2e8b0c0c9b08c7be(["model_metadata"]):::outdated
    x83714660cbce9386(["model_data"]):::outdated --> x33d7a8c39b877444(["nested_model_data"]):::outdated
    x83714660cbce9386(["model_data"]):::outdated --> x46ae7b79bb331fe3(["normalized_correlation_data"]):::outdated
    x0eb069b24b54e832(["icc_data"]):::outdated --> x7bc9c97777d80aca(["normalized_icc_data"]):::outdated
    xc9bc30827a59d139(["UGAFluVac_raw_data"]):::outdated --> xfdbd654a1f2948e6(["prepped_cohort_data"]):::outdated
    x83714660cbce9386(["model_data"]):::outdated --> xdd3ac2d8aae8b204(["strain_names_table"]):::outdated
    x83714660cbce9386(["model_data"]):::outdated --> x041afc9f57e7881c(["strain_panel_table"]):::outdated
    x4e3a18b7b03e484b(["UGAFluVac_raw_data_file"]):::outdated --> xc9bc30827a59d139(["UGAFluVac_raw_data"]):::outdated
    x46ae7b79bb331fe3(["normalized_correlation_data"]):::outdated --> x0bf166d166c2d12b(["vaccine_normalized_correlation_pairplot"]):::outdated
  end
  classDef outdated stroke:#000000,color:#000000,fill:#78B7C5;
  classDef none stroke:#000000,color:#000000,fill:#94a4ac;
  linkStyle 0 stroke-width:0px;
  linkStyle 1 stroke-width:0px;

Loading

That is...exactly the same, but all my targets are outdated!

So my question is how do I make sure the targets status is the same between the cluster and my local machine?

I ask because I've run the computationally expensive part on the cluster, and now I have some relatively fast post-processing I need to do, which would be easier to run on my local machine. If there is no way to do this, I can run everything on the cluster but I wanted to asked because it's a bit faster if I can do quick steps on my local machine without rerunning the expensive parts that I need to do on the cluster.

I thought that scp -rp would work because it makes an "exact" copy of the files on the cluster, but I guess that isn't quite true. Is there another way to copy files that will make my targets show up as up-to-date on my local machine? Or is there some command I can run to try and force targets to recognize the targets as up-to-date?

Thanks in advance (again)!

[wlandau]

tar_sitrep() can give you more information about why certain targets are not up to date. I would start there.

Transferring from Windows to Linux might be an issue because locales etc. can cause subtle discrepancies. You also might have different versions of R and/or packages. An renv library could help with the latter.

[wzbillings]

Thanks for the response -- I can reproduce this issue on the same repo across multiple machines, but I've been working on a reprex and haven't been able to reproduce it in a simpler example so far. The good news is that if I run tar_make(), it seems like despite saying all the targets are being run, they don't actually get computed (e.g. it took ~20 seconds to tar_make() my pipeline that said it was outdated, even though it took ~3 days to run the same pipeline previously). Not sure if that behavior is expected.

I'm still interested in trying to figure out what's going on here, but in the meantime it seems that there is a feasible solution. When I hopefully have more time to keep working on the reprex I'll try to figure out what's causing this.

[wlandau]

I wonder if your pipeline depends on global objects that contain functions or environments (such as R6 classes), or functions that need compilation with e.g. inline::cfunction()? Those objects can have meaningless changes that can throw off the hash. What does the tar_visnetwork() graph look like when you transfer the project to a new machine? Are there any outdated upstream functions or global objects?

[wzbillings]

Sorry this took so long to reply to! I don't have any inline::cfunction() calls -- and when I look at the entire network including upstream functions/objects, those appear to be up-to-date after the file transfer.

The main problem seems to be the actual data files. All of the pipelines where I've had this problem start with a tarchetypes::tar_file_read() target, and the small example pipeline where I couldn't replicate this problem didn't have any targets like that.

I think the problem is probably a fundamental issue in transferring files between Unix and Windows, maybe something inconsequential like line returns. The data files appear outdated when I copy the entire repo from the Unix cluster, so all downstream targets (i.e. almost all of them) are marked as outdated. But then when I run targets::tar_make(), the targets can be loaded correctly from the metadata store and there are no cues indicating that they need to be rerun, so they get marked as up-to-date without actually rerunning the targets.

So I think this is actually a non-issue, even though it was confusing to me. I'm not sure there's a good way to implement a check, warning, or advice for this situation.

[wlandau]

The main problem seems to be the actual data files.

I agree, that could be the reason. Looking back at your screenshot from #1476 (reply in thread), it looks like tar_sitrep() reported file = TRUE for a few targets, which indicates files somehow changed when you downloaded the project from Unix to Windows. You can confirm that the hash changed using secretbase::siphash13(file = "...") on a file and comparing output between platforms. As you say, that would point to an issue outside targets (preserving file contents en route to Windows).

[wzbillings]

Looking back at your screenshot from #1476 (reply in thread), it looks like tar_sitrep() reported file = TRUE for a few targets.

Yes, it does -- I should've looked at it a bit more carefully! It is TRUE for all of the raw data files at the beginning of the pipeline.

You can confirm that the hash changed using secretbase::siphash13(file = "...") on a file and comparing output between platforms.

Yep, they are different between the raw data files on the two platforms.

So that definitely seems to be what's going on here. Like I said, I can just run tar_make() after transferring the file to my windows machine and it still avoids rerunning the computationally expensive targets, so there's no real issue but it's nice to know what's going on.

Thanks for your help!