Google Drive and AWS S3 - Implementation, Tests and Documentation #556
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cs7-shrey
force-pushed
the
add_gdrive_aws_remote
branch
from
bcf5c2a to
2771400
Compare
JoeZiminski
reviewed
Aug 18, 2025
7 tasks
* google drive setup via python api first draft * enable google drive config setup via tui * minor compatibility and ui changes * protectedclient secret input box * google drive connection setup via TUI * add aws as remote storage via python api first draft * add: logging and connection check for aws s3 * update: type checking for aws regions * add: save aws configs via TUI * add: setup aws connection via TUI * feat: setup google drive on machines with no browser * fix: minor bug * fix: logical error * add: logging for google drive connections * refactor: move google drive client secret to be entered at runtime while setting up connection * refactor: aws_regions.py; provide aws secret access key at runtime * add: docstrings to gdrive.py * add: root_folder_id config to google drive; some refactor * refactor: radiobuttons switch in configs.py * edit: minor changes to SetupAwsScreen for setting up aws connection * refactor: SetupGdriveScreen and handle errors * add: some tooltips for google drive configs * fix: vanishing central path, radio button order, minor refactor * fix: minor bug * refactor: single button for setup connection * add: backwards compatibility to configs while load from config file * edit: raise error on bucket not present * rename: aws region config key * rename: connection method from aws_s3 to aws * add: utility function to remove duplicate code * add: docstrings to setup gdrive dialog * update: config dict inplace change for backward compatibility, use existing rclone function, moving type hint imports in the conditional block * add: docstrings to setup connection functions; remove: aws region class * add: docstrings to setup widgets function; use backwards compatibility * add: docstrings to rclone function, change arugment order * minor changes * refactor: centralize the get secret function * extend centralized function for sensitive information input to ssh connection method * convert stage from float to int * add: function for getting aws bucket name * refactor: connection methods list * move: widgets not match saved configs * Fix linting. * fix: overly long container in configs tab * add: first version of service account file setup method * remove: utility functions and docstrings in python api for config token setup method * fix: css * update: setup gdrive screen to use service account file and refactor state transition * rename: existing functions from config token -> service account * edit: docstrings in accordance with new connection setup * update: function for checking successful rclone connection * edit: setup aws screen css * Fix assert in CreateFolders. * Allow no central_path for aws or gdrive and refactor configs_content. * Fix tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix linting. * Add tests for new base path behaviour. * add: call rclone with popen; refactor: google drive connection to use popen * fix: failing typehint on python 3.9 * fix: another failing typehint * fix: failing monkeypatch * add: docstrings to functions invovled in killing rclone setup process * fix: minor bug --------- Co-authored-by: shrey <shrey@kali> Co-authored-by: JoeZiminski <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
JoeZiminski
force-pushed
the
add_gdrive_aws_remote
branch
from
48c7038 to
cdaacde
Compare
* add: gdrive and aws transfer tests * add: tui tests for setting up gdrive and aws connections * add: setup env variables from github secrets in the workflow file * Minor changes for null central path and distinct folders (#572) * Use different folder name per tests. * Play around with AWS see if it works. * Edit workflow. * fix: handle None central_path in tempfile path and correct aws bucket checking logic * fix: location constraint, add: failure test for aws setup via tui * minor change * add: gdrive tui connection setup failure tests * add: switch connection radiobutton test * extend: update tui config test for gdrive and aws * pre-commit: formatting * edit: code_test_and_deploy.yml * add: backwards compability tests for config.yaml * add: unit test for gdrive preliminary setup * Revert handling None case in get_aws_bucket_name * refactor: code_test_and_deploy.yml * Apply suggestions from code review Co-authored-by: Joe Ziminski <[email protected]> * fix: env in tui tests * use inbuilt rclone recursive search * add: docstrings to backwards compatiblity tests * minor changes * use rclone recursive search in ssh tests * use central function for ssh wildcard tests * fix: CI error on deleting temp.txt * add: docstrings to transfer tests * add: docstrings to tui tests * add: docstrings to unit test * remove: backwards compatibility test for incorrect configs * add: make new project configs tests for gdrive and aws * add: test for None central path in gdrive * edit: code_test_and_deploy.yml * add: central path fixtures to tui tests and tweak central path for transfer tests * fix: ci error on windows * Apply suggestions from code review Co-authored-by: Joe Ziminski <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add extra check to ssh setup and move message template function to tui_utils.py * fix: typo --------- Co-authored-by: Joe Ziminski <[email protected]> Co-authored-by: JoeZiminski <[email protected]> Co-authored-by: Shrey Singh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* add: docs for setting up google drive and aws connection * add: helpful links * Apply suggestions from code review Co-authored-by: Joe Ziminski <[email protected]> * minor changes * final changes * Small edits to docs. * Minor updates. * remove: miscellaneous file --------- Co-authored-by: Shrey Singh <[email protected]> Co-authored-by: Joe Ziminski <[email protected]> Co-authored-by: JoeZiminski <[email protected]>
* Use local only mode as default. * Capitalise Google Drive. * Capitalise Rclone. * Additional newline in tooltip. * Uncapitalise datashuttle, rename AWS S3 label to AWS S3 Bucket. * Extend tooltips and add example placeholders. * Small fixes * Fix missed uppercase datashuttle. * Uncomment test parameterisation. * Fix tests.
JoeZiminski
force-pushed
the
add_gdrive_aws_remote
branch
from
d48e420 to
40f2f4c
Compare
JoeZiminski
approved these changes
Aug 28, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This is the main PR to track the progress of adding Google Drive and AWS connection methods to datashuttle, including the code for implemenetation, testing and documentation.
TODOs:
preliminary_for_setup_without_browserwhich use some new rclone commandsChecklist: