Progress listening with chunked encoding reads more bytes transferred than file size

[vincetran]

Describe the bug

Amplify Android has been investigating an issue (aws-amplify/amplify-android#2869) where the root cause is that when doing an S3 transfer with chunked encoding, we read the number of bytes transferred as larger than the actual file size itself.

I understand that there's technically byte transfer overhead holistically when it comes to chunked encoding passing additional headers however the numbers don't really add up. e.g. for a 10,000,000 byte file, there's an additional 13,941 bytes that get transferred which seems way higher than the headers overhead size for chunked encoding. Additionally, other SDKs don't have this issue. I cross-referenced with Amplify Swift and even the legacy AWS SDK for Android's S3 transfer and their progress is always reported accurately -- for a 10,000,000 byte file, 10,000,000 bytes are reported as transferred.

If we disable chunked encoding, the number of bytes that are reported is accurate.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected behavior

We understand that for situations where the length of the content is unknown, progress listening doesn't make sense -- we're not asking about that. We have a known file size, so we expect that we should know when the file has finished transferring.

Current behavior

If you read the number of bytes transferred for an S3 download/upload with chunked encoding enabled, the number of bytes transferred is larger than the file size.

Steps to Reproduce

See our implementation of a progress listener interceptor:
https://github.com/aws-amplify/amplify-android/blob/e31b961e97c5896f200ff5939fa6d249832c0d27/aws-storage-s3/src/main/java/com/amplifyframework/storage/s3/transfer/ProgressListenerHttpInterceptor.kt#L53-L72

That gets added as an interceptor to the S3 client like this:
https://github.com/aws-amplify/amplify-android/blob/e31b961e97c5896f200ff5939fa6d249832c0d27/aws-storage-s3/src/main/java/com/amplifyframework/storage/s3/transfer/worker/SinglePartUploadWorker.kt#L45-L51

From there, just initiate an S3 download or upload and read the number of bytes and it'll read a higher number of bytes with encoding than the file size itself.

Possible Solution

No response

Context

No response

Smithy-Kotlin version

1.3.23

Platform (JVM/JS/Native)

Android

Operating system and version

Android 11

[ianbotsf]

Hi @vincetran, thanks for the bug report. It's not clear to me from the description where/how the SDK is incorrectly reporting number of bytes transferred. Can you please clarify?

Also, the progress listener code adds quite a bit of noise to debugging. Do you have a minimal repro which shows the issue?

[vincetran]

Ultimately, the issue is that AWS SDK for Android (which largely uses HttpsURLConnection for its AWS service calls), and the Amplify Swift (which uses Swift's built-in URLSession - example usage that's used for upload) report the number of bytes transferred as "the number of bytes of the file itself" whereas the Amplify Android relies on AWS SDK for Kotlin S3 client (and thus Smithy-kotlin) to handle the actual service call.

The progress listener code that I linked to is the most direct line of communication (afaik) from the service call to us, the client. Our ProgressListener classes simply take the bytesTransferred value from the interceptor, adds up the bytes, and emits a progress event of "total bytes sent" and "total bytes of the file." Funny enough, the original implementation tried to "workaround" the issue of more bytes being sent with this line:

if (transferRecord.bytesCurrent > transferRecord.bytesTotal) {
    transferRecord.bytesCurrent = transferRecord.bytesTotal
}

But we're realizing now that it's actually disingenuous and incorrect.

So ultimately what we're trying to understand is what is this overhead? Is it headers? Is there some sort of handshaking that's happening that gets reported as "bytes transferred?" We thought originally it was just chunking and s3 multipart upload headers overhead but if we disable chunking and do a multipart upload, the progress is reported accurately (total bytes transferred === total bytes of the file). But when we do a transfer with chunking is when we see this issue. I tried to even napkin math to calculate what the size of the chunking headers would be using the aws documentation and found that the number was significantly smaller than what we see IRL.

[xinsong-cui]

Hi @vincetran I believe the overhead is from the chunk header. Each data chunk of 65,536 bytes includes a 90-byte header. This additional header size seems align with the extra bytes you're observing.

We are currently investigating this issue and will get back to you with our findings. Thank you for your patience and bringing this to our attention.

[vincetran]

Okay that's good information! So let me share some additional data that I saw which might help.

For a 10,000,000 byte file, I observe the following byte transfers:

1. 1,220 transfers of 8192 bytes (totals 9,994,240 bytes)
2. 1 transfer of 5849 bytes
3. 152 transfers of 90 bytes (totals 13680 bytes)
4. 2 transfers of 86 bytes (totals 172 bytes)

Bullet points 1 and 2 indicate the bytes of the file itself while 3 and 4 indicates the "overhead"
I wasn't aware of the 65,536 byte data chunks which helps me understand point 3 some more. If I divide the 10,000,000 bytes to those sized chunks, I get 152.355 chunks. So rounded down, I can understand point 2, the 152 transfers of 90 bytes.

What about the 86 byte transfers? Would that be a sort of, start and stop kind of handshake? And what about the .355 chunk?

So in lieu of observing only the bytes transferred for the file, I'm wondering if I can try and calculate the "true bytes sent" for a transfer. So the file transferred + the chunk header overhead.

[xinsong-cui]

Thanks for sharing this data! I am guessing the 86 bytes transfer could be the final chunk

The size of the final chunk data that you send is 0, although the chunk body still contains metadata, including the signature of the previous chunk.

For example: 0;chunk-signature=xxxxx...\r\n

Calculating the actual bytes transferred is challenging, and we don't want to put that burden on your implementation. Our team is currently investigating solutions to report the de-chunkified data size directly.

We'll update you when we have a solution.

[vincetran]

Thanks for the update! The de-chunkified data size would be greatly appreciated 🙏

Out of curiousity, I understand the concept of the final chunk and that makes total sense, but I'm curious why there would be two instances of the 86-byte transfer. Do you have a theory?

[xinsong-cui]

To be honest, I don't have a definite theory either. The other 86-bytes looks to me like it could be a header size for an empty chunk.