S3 FileDownload fails with file lock exception : FileLockException

[NitinPeddewad]

Upcoming End-of-Support

• I acknowledge the upcoming end-of-support for AWS SDK for Java v1 was announced, and migration to AWS SDK for Java v2 is recommended.

Describe the bug

Problem: We are encountering FileLockException errors when 42 applications attempt to download files in parallel from S3 for processing.

Error Details:

com.amazonaws.services.s3.transfer.exception.FileLockException: Fail to lock L9_NS_S3_TO_xxxxxxx_011-xxxxxx-test-xxxxx-3-buffer/xxxxx.00.xxxxxx_sxxxx.parquet
at com.amazonaws.services.s3.transfer.DownloadCallable.truncateDestinationFileIfNecessary(DownloadCallable.java:200) ~\[xxxxxx:?\]
at com.amazonaws.services.s3.transfer.DownloadCallable.downloadInParallel(DownloadCallable.java:139) ~\[xxxxxxxx:?\]
at com.amazonaws.services.s3.transfer.DownloadCallable.downloadInParallel(DownloadCallable.java:99) ~\[xxxxx:?\]
at com.amazonaws.services.s3.transfer.internal.AbstractDownloadCallable.call(AbstractDownloadCallable.java:100) ~\[xxxxx:?\]
at com.amazonaws.services.s3.transfer.internal.AbstractDownloadCallable.call(AbstractDownloadCallable.java:40) ~\[xxxxxx:?\]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~\[?:?\]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~\[?:?\]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~\[?:?\]
at java.lang.Thread.run(Thread.java:834) ~\[?:?\]

Context:

• Each of the 42 applications is designed to process unique files, meaning there should be no contention for the same file download to the same system file location.
• The issue is intermittent; after an application retries multiple times on a specific file, it eventually moves on to a different file, only to encounter the FileLockException again on that new file.
• We are always performing delete all downloaded files including File Lock file as well, before starting to download files again.

Java Code For reference

  File pathToDownload = new File(bufferedDirectoryName+filenameWithoutPrefix);
  transferManager.download(bucketName, filename, pathToDownload).waitForCompletion();
  boolean success = pathToDownload.renameTo(new File(localDirectoryName+filenameWithoutPrefix));

                            `

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Expected Behaviour: Applications should be able to download files from S3 without encountering file lock exceptions, given that each application is working with unique file paths.

Current Behavior

Problem: We are encountering FileLockException errors when 42 applications attempt to download files in parallel from S3 for processing.

Reproduction Steps

Run 42 multiples app to download files from S3, each file size ~160MB.
Download them to unique folder for each app.

Possible Solution

We have a static Map that maintains file lock mappings. If a file is being downloaded and gets erased during the process, who is responsible for clearing those downloadInParallel executors? Are we calling unlock file on failure, or are we ensuring file unlocking occurs in all cases (both failure and completion)?

Additional Information/Context

<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.12.589</version>

AWS Java SDK version used

1.12.589

JDK version used

java 11.0.21 2023-10-17 LTS

Operating System and version

Ubuntu