Pseudo PR against TEMPLATE for checking the pipeline

.github/actions/nf-test/action.yml

@@ -59,8 +59,6 @@ runs:
     - name: Run nf-test
       shell: bash
       env:
-        NFT_DIFF: ${{ env.NFT_DIFF }}
-        NFT_DIFF_ARGS: ${{ env.NFT_DIFF_ARGS }}
         NFT_WORKDIR: ${{ env.NFT_WORKDIR }}
       run: |
         nf-test test \

.github/workflows/awsfulltest.yml

@@ -24,7 +24,7 @@ jobs:
 
       - name: Launch workflow via Seqera Platform
         uses: seqeralabs/action-tower-launch@v2
-        # TODO nf-core: You can customise AWS full pipeline tests as required
+        # You can customise AWS full pipeline tests as required
         # Add full size test data (but still relatively small datasets for few samples)
         # on the `test_full.config` test runs with only one set of parameters
         with:

.github/workflows/nf-test.yml

@@ -99,8 +99,6 @@ jobs:
       - name: Run nf-test
         uses: ./.github/actions/nf-test
         env:
-          NFT_DIFF: ${{ env.NFT_DIFF }}
-          NFT_DIFF_ARGS: ${{ env.NFT_DIFF_ARGS }}
           NFT_WORKDIR: ${{ env.NFT_WORKDIR }}
         with:
           profile: ${{ matrix.profile }}

.github/workflows/template-version-comment.yml

@@ -2,7 +2,8 @@ name: nf-core template version comment
 # This workflow is triggered on PRs to check if the pipeline template version matches the latest nf-core version.
 # It posts a comment to the PR, even if it comes from a fork.
 
-on: pull_request_target
+on:
+  pull_request:
 
 jobs:
   template_version:

.gitignore

@@ -7,3 +7,4 @@ testing/
 testing*
 *.pyc
 null/
+.idea/

CHANGELOG.md

@@ -5,12 +5,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## v1.1.0 - [date]
 
+- Preprint is out now! Linking it in the documentation.
+- Added support for AWS: changed the structure of load response and parameter check to conform more to Nextflow
+  best practices.
+- Simplified visualization: multiple short processes were creating overhang → more efficient in one process.
+- Fixed errors that arose from the latest drevalpy version.
+- Added authors and licenses to the python scripts.
+- Moved all publishDir directives to modules.config.
+- Fixed drevalpy versions in conda and docker to 1.3.4: now supporting Python 3.13
+- Added no_hyperparameter_tuning option for quick runs without hyperparameter tuning.
+- Flag final_model_on_full data: if True, a final/production model is saved in the results directory. If hyperparameter_tuning is true, the final model is tuned, too. The model can later be loaded using the implemented load functions of the drevalpy models.
+
+## v1.0.0 - [date]
+
 Initial release of nf-core/drugresponseeval, created with the [nf-core](https://nf-co.re/) template.
 
 ### `Added`
 
+- Updated to the new template
+- Added tests that run with docker, singularity, apptainer, and conda
+- Added the docker container and the conda env.yml in the nextflow.config. We just need one container for all
+  processes as this pipeline automates the PyPI package drevalpy.
+- Added usage and output documentation.
+- Added CurveCurator to preprocess curves of custom datasets
+
 ### `Fixed`
 
+- Fixed linting issues
+- Fixed bugs with path_data: can now be handled as absolute and relative paths
+
 ### `Dependencies`
 
 ### `Deprecated`

CITATIONS.md

@@ -1,5 +1,9 @@
 # nf-core/drugresponseeval: Citations
 
+## [DrugResponseEval](https://github.com/nf-core/drugresponseeval/)
+
+> Bernett, J., Iversen, P., Picciani, M., Wilhelm, M., Baum, K., & List, M. **From Hype to Health Check: Critical Evaluation of Drug Response Prediction Models with DrEval.** > [10.1101/2025.05.26.655288](doi.org/10.1101/2025.05.26.655288) > _bioRxiv_, 2025-05.
+
 ## [nf-core](https://pubmed.ncbi.nlm.nih.gov/32055031/)
 
 > Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031.
@@ -10,6 +14,30 @@
 
 ## Pipeline tools
 
+- [DrEvalPy](https://github.com/daisybio/drevalpy): The pipeline mostly automates the individual steps of the DrEvalPy PyPI package.
+
+  > Bernett, J, Iversen, P, Picciani, M, Wilhelm, M, Baum, K, List, M. Will be published soon.
+
+- [CurveCurator](https://www.nature.com/articles/s41467-023-43696-z): For custom curve fitting on custom datasets. We also used it to re-process the response curves of GDSC1, GDSC2, CCLE, and CTRP.
+
+  > Bayer, F.P., Gander, M., Kuster, B., The, M. CurveCurator: a recalibrated F-statistic to assess, classify, and explore significance of dose–response curves. Nature Communications. 2023 Nov;14(7902).
+
+- [DIPK](https://doi.org/10.1093/bib/bbae153): Implemented model in the pipeline.
+
+  > Li P, Jiang Z, Liu T, Liu X, Qiao H, Yao X. Improving drug response prediction via integrating gene relationships with deep learning. Briefings in Bioinformatics. 2024 May;25(3):bbae153.
+
+- [MOLI](https://doi.org/10.1093/bioinformatics/btz318): Implemented model in the pipeline.
+
+  > Sharifi-Noghabi H, Zolotareva O, Collins CC, Ester M. MOLI: multi-omics late integration with deep neural networks for drug response prediction. Bioinformatics. 2019 Jul;35(14):i501-9.
+
+- [SRMF](https://doi.org/10.1186/s12885-017-3500-5): Implemented model in the pipeline.
+
+  > Wang L, Li X, Zhang L, Gao Q. Improved anticancer drug response prediction in cell lines using matrix factorization with similarity regularization. BMC cancer. 2017 Dec;17:1-2.
+
+- [SuperFELT](https://doi.org/10.1186/s12859-021-04146-z): Implemented model in the pipeline.
+
+  > Park S, Soh J, Lee H. Super. FELT: supervised feature extraction learning using triplet loss for drug response prediction with multi-omics data. BMC bioinformatics. 2021 May 25;22(1):269.
+
 ## Software packaging/containerisation tools
 
 - [Anaconda](https://anaconda.com)

README.md

@@ -6,7 +6,8 @@
 </h1>
 
 [![GitHub Actions CI Status](https://github.com/nf-core/drugresponseeval/actions/workflows/ci.yml/badge.svg)](https://github.com/nf-core/drugresponseeval/actions/workflows/ci.yml)
-[![GitHub Actions Linting Status](https://github.com/nf-core/drugresponseeval/actions/workflows/linting.yml/badge.svg)](https://github.com/nf-core/drugresponseeval/actions/workflows/linting.yml)[![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/drugresponseeval/results)[![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.XXXXXXX-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.XXXXXXX)
+[![GitHub Actions Linting Status](https://github.com/nf-core/drugresponseeval/actions/workflows/linting.yml/badge.svg)](https://github.com/nf-core/drugresponseeval/actions/workflows/linting.yml)[![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/drugresponseeval/results)[![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.14779984-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.14779984)
+
 [![nf-test](https://img.shields.io/badge/unit_tests-nf--test-337ab7.svg)](https://www.nf-test.com)
 
 [![Nextflow](https://img.shields.io/badge/version-%E2%89%A524.04.2-green?style=flat&logo=nextflow&logoColor=white&color=%230DC09D&link=https%3A%2F%2Fnextflow.io)](https://www.nextflow.io/)
@@ -20,48 +21,43 @@
 
 ## Introduction
 
-**nf-core/drugresponseeval** is a bioinformatics pipeline that ...
-
-<!-- TODO nf-core:
-   Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the
-   major pipeline sections and the types of output it produces. You're giving an overview to someone new
-   to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction
--->
-
-<!-- TODO nf-core: Include a figure that guides the user through the major workflow steps. Many nf-core
-     workflows use the "tube map" design for that. See https://nf-co.re/docs/guidelines/graphic_design/workflow_diagrams#examples for examples.   -->
-<!-- TODO nf-core: Fill in short bullet-pointed list of the default steps in the pipeline -->
+# ![drevalpy_summary](assets/dreval_summary.svg)
+
+**DrEval** is a bioinformatics framework that includes a PyPI package (drevalpy) and a Nextflow
+pipeline (this repo). DrEval ensures that evaluations are statistically sound, biologically
+meaningful, and reproducible. DrEval simplifies the implementation of drug response prediction
+models, allowing researchers to focus on advancing their modeling innovations by automating
+standardized evaluation protocols and preprocessing workflows. With DrEval, hyperparameter
+tuning is fair and consistent. With its flexible model interface, DrEval supports any model type,
+ranging from statistical models to complex neural networks. By contributing your model to the
+DrEval catalog, you can increase your work's exposure, reusability, and transferability.
+
+1. The response data is loaded
+2. All models are trained and evaluated in a cross-validation setting
+3. For each CV split, the best hyperparameters are determined using a grid search per model
+4. The model is trained on the full training set (train & validation) with the best
+   hyperparameters to predict the test set
+5. If randomization tests are enabled, the model is trained on the full training set with the best
+   hyperparameters to predict the randomized test set
+6. If robustness tests are enabled, the model is trained N times on the full training set with the
+   best hyperparameters
+7. Plots are created summarizing the results
+
+For baseline models, no randomization or robustness tests are performed.
 
 ## Usage
 
 > [!NOTE]
 > If you are new to Nextflow and nf-core, please refer to [this page](https://nf-co.re/docs/usage/installation) on how to set-up Nextflow. Make sure to [test your setup](https://nf-co.re/docs/usage/introduction#how-to-run-a-pipeline) with `-profile test` before running the workflow on actual data.
 
-<!-- TODO nf-core: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
-     Explain what rows and columns represent. For instance (please edit as appropriate):
-
-First, prepare a samplesheet with your input data that looks as follows:
-
-`samplesheet.csv`:
-
-```csv
-sample,fastq_1,fastq_2
-CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz
-```
-
-Each row represents a fastq file (single-end) or a pair of fastq files (paired end).
-
--->
-
 Now, you can run the pipeline using:
 
-<!-- TODO nf-core: update the following command to include all required parameters for a minimal example -->
-
 ```bash
 nextflow run nf-core/drugresponseeval \
    -profile <docker/singularity/.../institute> \
-   --input samplesheet.csv \
-   --outdir <OUTDIR>
+   --models <RandomForest,model2,...> \
+   --baselines <NaiveMeanEffectsPredictor,baseline2,...> \
+   --dataset_name <CTRPv2|CTRPv1|CCLE|GDSC1|GDSC2|custom_dataset>
 ```
 
 > [!WARNING]
@@ -77,24 +73,36 @@ For more details about the output files and reports, please refer to the
 
 ## Credits
 
-nf-core/drugresponseeval was originally written by Judith Bernett.
+nf-core/drugresponseeval was originally written by Judith Bernett (TUM) and Pascal Iversen (FU
+Berlin).
 
 We thank the following people for their extensive assistance in the development of this pipeline:
 
-<!-- TODO nf-core: If applicable, make list of people who have also contributed -->
-
 ## Contributions and Support
 
+Contributors to nf-core/drugresponseeval and the drevalpy PyPI package:
+
+- [Judith Bernett](https://github.com/JudithBernett) (TUM)
+- [Pascal Iversen](https://github.com/PascalIversen) (FU Berlin)
+- [Mario Picciani](https://github.com/picciama) (TUM)
+
 If you would like to contribute to this pipeline, please see the [contributing guidelines](.github/CONTRIBUTING.md).
 
 For further information or help, don't hesitate to get in touch on the [Slack `#drugresponseeval` channel](https://nfcore.slack.com/channels/drugresponseeval) (you can join with [this invite](https://nf-co.re/join/slack)).
 
 ## Citations
 
-<!-- TODO nf-core: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. -->
-<!-- If you use nf-core/drugresponseeval for your analysis, please cite it using the following doi: [10.5281/zenodo.XXXXXX](https://doi.org/10.5281/zenodo.XXXXXX) -->
+If you use nf-core/drugresponseeval for your analysis, please cite it using the following doi: [10.5281/zenodo.14779984](https://doi.org/10.5281/zenodo.14779984)
+
+> Our corresponding publication is at doi [10.1101/2025.05.26.655288](doi.org/10.1101/2025.05.26.655288)
+>
+> Bernett, J., Iversen, P., Picciani, M., Wilhelm, M., Baum, K., & List, M. **From Hype to Health Check: Critical Evaluation of Drug Response Prediction Models with DrEval.**
+>
+> _bioRxiv_, 2025-05.
+
+The underlying data is available at doi: [10.5281/zenodo.12633909](https://doi.org/10.5281/zenodo.12633909).
 
-<!-- TODO nf-core: Add bibliography of tools and data used in your pipeline -->
+The underlying python package is drevalpy, availably on [PyPI](https://pypi.org/project/drevalpy/) as standalone, for which we also have an extensive [ReadTheDocs Documentation](https://drevalpy.readthedocs.io/en/latest/).
 
 An extensive list of references for the tools used by the pipeline can be found in the [`CITATIONS.md`](CITATIONS.md) file.
 

bin/collect_results.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python
+
+# Written by Judith Bernett and released under the MIT License.
+
+import argparse
+import pandas as pd
+import pathlib
+
+from drevalpy.visualization.utils import prep_results, write_results
+
+
+def get_parser():
+    parser = argparse.ArgumentParser(description="Collect results and write to single files.")
+    parser.add_argument("--outfiles", type=str, nargs="+", required=True, help="Output files.")
+    parser.add_argument("--path_data", type=str, default="data", help="Data directory path")
+    return parser
+
+
+def parse_results(outfiles):
+    # get all files with the pattern f'{model_name}_evaluation_results.csv' from outfiles
+    result_files = [file for file in outfiles if "evaluation_results.csv" in file]
+    # get all files with the pattern f'{model_name}_evaluation_results_per_drug.csv' from outfiles
+    result_per_drug_files = [file for file in outfiles if "evaluation_results_per_drug.csv" in file]
+    # get all files with the pattern f'{model_name}_evaluation_results_per_cl.csv' from outfiles
+    result_per_cl_files = [file for file in outfiles if "evaluation_results_per_cl.csv" in file]
+    # get all files with the pattern f'{model_name}_true_vs_pred.csv' from outfiles
+    t_vs_pred_files = [file for file in outfiles if "true_vs_pred.csv" in file]
+    return result_files, result_per_drug_files, result_per_cl_files, t_vs_pred_files
+
+
+def collapse_file(files):
+    out_df = None
+    for file in files:
+        if out_df is None:
+            out_df = pd.read_csv(file, index_col=0)
+        else:
+            out_df = pd.concat([out_df, pd.read_csv(file, index_col=0)])
+    if out_df is not None and "drug" in out_df.columns:
+        out_df["drug"] = out_df["drug"].astype(str)
+    return out_df
+
+
+if __name__ == "__main__":
+    args = get_parser().parse_args()
+    # parse the results from outfiles.outfiles
+    outfiles = args.outfiles
+    path_data = pathlib.Path(args.path_data)
+    eval_result_files, eval_result_per_drug_files, eval_result_per_cl_files, true_vs_pred_files = parse_results(outfiles)
+
+    # collapse the results into single dataframes
+    eval_results = collapse_file(eval_result_files)
+    eval_results_per_drug = collapse_file(eval_result_per_drug_files)
+    eval_results_per_cell_line = collapse_file(eval_result_per_cl_files)
+    t_vs_p = collapse_file(true_vs_pred_files)
+
+    # prepare the results through introducing new columns algorithm, rand_setting, LPO_LCO_LDO, split, CV_split
+    eval_results, eval_results_per_drug, eval_results_per_cell_line, t_vs_p = prep_results(
+        eval_results=eval_results,
+        eval_results_per_drug=eval_results_per_drug,
+        eval_results_per_cell_line=eval_results_per_cell_line,
+        t_vs_p=t_vs_p,
+        path_data=path_data
+    )
+
+    # save the results to csv files
+    write_results(
+        path_out="",
+        eval_results=eval_results,
+        eval_results_per_drug=eval_results_per_drug,
+        eval_results_per_cl=eval_results_per_cell_line,
+        t_vs_p=t_vs_p,
+    )

bin/consolidate_results.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python
+
+# Written by Judith Bernett and released under the MIT License.
+
+import os
+import argparse
+from drevalpy.models import MODEL_FACTORY
+from drevalpy.experiment import consolidate_single_drug_model_predictions
+
+
+def get_parser():
+    parser = argparse.ArgumentParser(description="Consolidate results for SingleDrugModels")
+    parser.add_argument('--run_id', type=str, required=True, help="Run ID")
+    parser.add_argument("--test_mode", type=str, required=True, help="Test mode (LPO, LCO, LDO)")
+    parser.add_argument("--model_name", type=str, required=True, help="All Model "
+                                                                                  "names")
+    parser.add_argument("--outdir_path", type=str, required=True, help="Output directory path")
+    parser.add_argument("--n_cv_splits", type=int, required=True, help="Number of CV splits")
+    parser.add_argument("--cross_study_datasets", type=str, nargs="+", help="All "
+                                                                                          "cross-study "
+                                                                                          "datasets")
+    parser.add_argument("--randomization_modes", type=str, required=True, help="All "
+                                                                                     "randomizations")
+    parser.add_argument("--n_trials_robustness", type=int, required=True, help="Number of trials")
+    return parser
+
+
+def main():
+    parser = get_parser()
+    args = parser.parse_args()
+    results_path = os.path.join(
+        args.outdir_path,
+        args.run_id,
+        args.test_mode,
+    )
+    if args.randomization_modes == "[None]":
+        randomizations = None
+    else:
+        randomizations = args.randomization_modes.split('[')[1].split(']')[0].split(', ')
+    model = MODEL_FACTORY[args.model_name]
+    if args.cross_study_datasets is None:
+        args.cross_study_datasets = []
+    consolidate_single_drug_model_predictions(
+        models=[model],
+        n_cv_splits=args.n_cv_splits,
+        results_path=results_path,
+        cross_study_datasets=args.cross_study_datasets,
+        randomization_mode=randomizations,
+        n_trials_robustness=args.n_trials_robustness,
+        out_path=""
+    )
+
+
+if __name__ == "__main__":
+    main()