Skip to content

Scrapy errors due to decompression attempt based on Content-Encoding #11

@BurnzZ

Description

@BurnzZ

For Zyte API requests with httpResponseHeader=True, some websites would return headers like Content-Encoding: gzip.

In the current setup, all of the headers in the httpResponseHeader from Zyte API response is used to create the Response in the Download Handler.

Now, when the response containing the Content-Encoding: gzip header is processed by the Downloader Middlewares, the scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware errors out since it's attempting to decompress the gzipped response.

This is not the case since Zyte API already decompressed the contents.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions