Skip to content

ExpiredToken error when cp large dataset #678

@MqllR

Description

@MqllR

Some context:

  • large dataset: > 4TB, million of files
  • s5cmd running in AWS EKS with assumeRole

I'm facing intermittent problem to upload or download with the following error:

MultipartUpload: upload multipart failed upload id: xxx-- caused by: ExpiredToken: The provided token has expired. status code: 400

After looking at the code, there is an explicit deny for retrying this kind of error.

The PR introducing this code seems to tackle another kind of issue. So I'm wondering if that would make sense to revisit the s3 session management.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions