http parse version error when `--release` version of `kafka_producer` attempts to download from ztf public alert archive

[jesaerys]

I get the following error when I run kafka_producer ztf 20250205:

Error downloading alerts: error sending request for url (https://ztf.uw.edu/alerts/public/ztf_public_20250205.tar.gz)
Error: reqwest::Error { kind: Request, url: "https://ztf.uw.edu/alerts/public/ztf_public_20250205.tar.gz", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(Parse(Version))) }

curl works, though:

$ curl -v https://ztf.uw.edu/alerts/public/ztf_public_20250205.tar.gz
* Host ztf.uw.edu:443 was resolved.
* IPv6: (none)
* IPv4: 128.95.0.72
*   Trying 128.95.0.72:443...
* Connected to ztf.uw.edu (128.95.0.72) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=epyc.astro.washington.edu
*  start date: Apr 21 09:51:48 2025 GMT
*  expire date: Jul 20 09:51:47 2025 GMT
*  subjectAltName: host "ztf.uw.edu" matched cert's "ztf.uw.edu"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
* using HTTP/1.x
> GET /alerts/public/ztf_public_20250205.tar.gz HTTP/1.1
> Host: ztf.uw.edu
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Date: Thu, 19 Jun 2025 13:35:20 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
< Last-Modified: Thu, 06 Feb 2025 15:11:12 GMT
< ETag: "3ed8962e-62d7aa70b5082"
< Accept-Ranges: bytes
< Content-Length: 1054381614
< Content-Type: application/x-gzip
< 
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.
* Failure writing output to destination, passed 16384 returned 4294967295
* Closing connection
* TLSv1.2 (OUT), TLS alert, close notify (256):

It looks like curl is falling back to http/1.1. Possibly we need to force reqwest to use http/1.1 (I believe it uses http2 by default), or the issue has to do with something else.

I have no idea when this stopped working because it's been several months since I tried downloading new data.

[jesaerys]

I believe the error traces back to request_builder.send().await? in boom::utils::data::download_to_file. I tried setting http1_only() on the ClientBuilder, but I got the same error and now I'm really confused.

[jesaerys]

I set connection_verbose(true) on the reqwest ClientBuilder and enabled logging at the TRACE level. Here's what I get from the producer:

2025-06-19T15:17:45.787933Z  INFO boom::kafka::ztf: Downloading alerts for date 20250205 (programid: Public)
2025-06-19T15:17:45.803067Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("https", ztf.uw.edu)
2025-06-19T15:17:45.803176Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("https"), host=Some("ztf.uw.edu"), port=None
2025-06-19T15:17:45.874780Z DEBUG hyper_util::client::legacy::connect::http: connecting to 128.95.0.72:443
2025-06-19T15:17:45.948783Z DEBUG hyper_util::client::legacy::connect::http: connected to 128.95.0.72:443
2025-06-19T15:17:46.134635Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-06-19T15:17:46.134756Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("https", ztf.uw.edu)
2025-06-19T15:17:46.225086Z ERROR boom::kafka::ztf: Error downloading alerts: error sending request for url (https://ztf.uw.edu/alerts/public/ztf_public_20250205.tar.gz)
Error: reqwest::Error { kind: Request, url: "https://ztf.uw.edu/alerts/public/ztf_public_20250205.tar.gz", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(Parse(Version))) }

And this is without http1_only(). hyper reports successfully going through the http1 handshake, so it seems like the client somehow figures out which version to use and then the error happens somewhere after that.

[Theodlz]

@jesaerys it definitely still works just fine on my laptop (mac) and Linux desktop. I suspect this might be some openssl issue? Might be the change Pete made to the cargo.toml for it to work on his machine, could be that the version of openssl it uses (vendor I think) is too old on your laptop? Anyways just a thought. We used to download with curl and I changed that to reqwest too so that could have been an issue (it does work for me but maybe reqwest runs into an issue I don't have myself).

Last week on Zoom I got Pete to run this commands which worked for him too, so it's definitely not that the code is broken, but there is something wrong based on the environment around it.

[jesaerys]

that's good info, thanks

[jesaerys]

Ok, I've got the producer working again. All I did was build from complete scratch with cargo clean && rm Cargo.lock && cargo build. This bumped a bunch of transitive dependencies in Cargo.lock and somehow that was enough to make the download request succeed. I have no deeper explanation for why this worked. I didn't touch anything on my system. I'll submit a PR with the updated Cargo.lock.