[BUG] Burp option not working with HTTPS

[thibon]

Describe the bug
When using '--burp' option or '--proxy http://localhost:8080' with an url in HTTPS, I get the following error:

WRN      2.224 feroxbuster::utils Error while making request: error sending request for url (https://target/)                                                                                           WRN      2.224 feroxbuster::utils err: error sending request for url (https://target/)                                                                                                                  Could not connect to https://www.dma.gp, skipping...
  => error sending request for url (https://target/)

However, looks like it's working with '--replay-prox' option. Note that I'm not sure to understand the difference between the two based on the documentation

I'm using feroxbuster 2.10.4

To Reproduce
Steps to reproduce the behavior:

1. Just try to connect to an https target

Expected behavior
Normal request going through the proxy

Traceback / Error Output

~ ❯ feroxbuster --url 'https://target' -w 'SecLists/Discovery/Web-Content/raft-large-words-lowercase.txt' --burp -vvv                                27/06/24 11:28:23 AST
DBG      0.001 feroxbuster::utils set open file descriptor limit to 8192                                                                                                                                                                                                                                                                                                                                                DBG      0.013 feroxbuster::event_handlers::filters filters: FiltersHandler { data: FeroxFilters { filters: RwLock { data: [], poisoned: false, .. } }, receiver: UnboundedReceiver { chan: Rx { inner: Chan { tx: Tx { block_tail: 0x14e80a400, tail_position: 1 }, semaphore: Semaphore(0), rx_waker: AtomicWaker, tx_count: 1, rx_fields: "..." } } } }
 ___  ___  __   __     __      __         __   ___
|__  |__  |__) |__) | /  `    /  \ \_/ | |  \ |__
|    |___ |  \ |  \ | \__,    \__/ / \ | |__/ |___
by Ben "epi" Risher 🤓                 ver: 2.10.4
───────────────────────────┬──────────────────────
 🎯  Target Url            │ https://target
 🚀  Threads               │ 50
 📖  Wordlist              │ SecLists/Discovery/Web-Content/raft-large-words-lowercase.txt
 👌  Status Codes          │ All Status Codes!
 💥  Timeout (secs)        │ 7
 🦡  User-Agent            │ feroxbuster/2.10.4
 💎  Proxy                 │ http://127.0.0.1:8080
 🔎  Extract Links         │ true
 🏁  HTTP methods          │ [GET]
 🔓  Insecure              │ true
 🔊  Verbosity             │ 3
 🔃  Recursion Depth       │ 4
───────────────────────────┴──────────────────────
 🏁  Press [ENTER] to use the Scan Management Menu™
──────────────────────────────────────────────────
WRN      2.176 feroxbuster::utils Error while making request: error sending request for url (https://target/)                                                                                           WRN      2.176 feroxbuster::utils err: error sending request for url (https://target/)                                                                                                                  Could not connect to https://target, skipping...
  => error sending request for url (https://target/)                                                                                                                                                    WRN      2.177 feroxbuster::heuristics error sending request for url (https://target/)                                                                                                                  INF      2.177 feroxbuster All scans complete!                                                                                                                                                              INF      2.178 feroxbuster::event_handlers::statistics Stats {
    kind: "statistics",
    timeouts: 0,
    requests: 2,
    expected_per_scan: 0,
    total_expected: 0,
    errors: 1,
    successes: 1,
    redirects: 0,
    client_errors: 0,
    server_errors: 0,
    total_scans: 0,
    initial_targets: 0,
    links_extracted: 0,
    extensions_collected: 0,
    status_200s: 1,
    status_301s: 0,
    status_302s: 0,
    status_401s: 0,
    status_403s: 0,
    status_429s: 0,
    status_500s: 0,
    status_503s: 0,
    status_504s: 0,
    status_508s: 0,
    wildcards_filtered: 0,
    responses_filtered: 0,
    resources_discovered: 0,
    url_format_errors: 0,
    redirection_errors: 0,
    connection_errors: 0,
    request_errors: 1,
    directory_scan_times: Mutex {
        data: [],
        poisoned: false,
        ..
    },
    total_runtime: Mutex {
        data: [
            0.0,
        ],
        poisoned: false,
        ..
    },
    json: false,
    targets: Mutex {
        data: [
            "https://target",
        ],
        poisoned: false,
        ..
    },
}                                                                                                                                                                                                           ERROR: Could not connect to any target provided

Environment (please complete the following information):

• feroxbuster version: 2.10.4
• OS: MacOS Ventura 13.4

Additional context
No additional context, it was working well in the past, did many test without any problem but can't find the cause here

[godylockz]

@thibon Did you add burp's cert to your systems cert store?
update-ca-certificates

HTTPS doesn't allow MITM interceptions - you have to enable trust.

I would first try to check if this works:

export HTTP_PROXY=http://127.0.0.1:8080
export HTTPS_PROXY=http://127.0.0.1:8080
curl https://target

[thibon]

Hi, thanks for your answer but this test works, and burp cert is in my cert store.
What is weird is that when I configure feroxbuster with --burp option or with --proxy, the first request is OK, I can see it in burp but after that the second fail and I get the error

[epi052]

@thibon can you elaborate on the first request? is it the request to github.com that checks for updates? or is it a request to the actual target? If so, can you post the server response to that first request?

additionally, a pcap of the conversation might shed some light on things as well.

[epi052]

Also, the --replay-proxy option waits until you get a response from the server that wasn't filtered out, and then passes it to burp. It's just a convenience option to only send interesting things to burp for further analysis, instead of sending all the 404s et al and cluttering up the history

[epi052]

just for the record, i'm able to run -u https://bitdiscovery.com --burp with no issues.

i looked through the replay proxy code vs the proxy code and i don't see any obvious differences that would lead to one working adn the other not.

[thibon]

Hi,

Let's forget about --replay-proxy option, it's not related to the problem and I didn't fully understood this option at first

@thibon can you elaborate on the first request? is it the request to github.com that checks for updates? or is it a request to the actual target? If so, can you post the server response to that first request?

additionally, a pcap of the conversation might shed some light on things as well.

Traffic is HTTPS so it's a bit hard to provide an deciphered pcap

However, digging in with your example, it works pretty well with some targets like https://bitdiscovery.com or https://google.com, but not with others like https://dma.gp or https://www.groupe-paralliance.fr

It looks like it is because the two other targets don't support HTTP/2 and when feroxbuster sends the second request, it's using HTTP/2.0 regarding of the response of the server to the first request (if the server respond with HTTP/1.1 for example). I was able to have it work by forcing burp to use HTTP/1 in the proxy options because I couldn't find an option to tell feroxbuster to use HTTP/1.1, and it didn't follow server instructions

[epi052]

thanks for the targets that exhibit the behavior!

regarding the pcap, i actually was interested in the tls conversation, since we can see the http stuff in burp.

for https://dma.gp, i got a bad certificate error.

so, i then added -k and the scan worked fine. (--dont-filter --dont-extract-links simply limit the number of requests sent to the target, they can be removed and the scan will still work).

feroxbuster -u https://dma.gp -k  --dont-filter --dont-extract-links

 ___  ___  __   __     __      __         __   ___
|__  |__  |__) |__) | /  `    /  \ \_/ | |  \ |__
|    |___ |  \ |  \ | \__,    \__/ / \ | |__/ |___
by Ben "epi" Risher 🤓                 ver: 2.10.4
───────────────────────────┬──────────────────────
 🎯  Target Url            │ https://dma.gp
 🚀  Threads               │ 50
 📖  Wordlist              │ twl.smol
 👌  Status Codes          │ All Status Codes!
 💥  Timeout (secs)        │ 7
 🦡  User-Agent            │ feroxbuster/2.10.4
 💉  Config File           │ /home/epi/PycharmProjects/feroxbuster/ferox-config.toml
 🔎  Extract Links         │ true
 🏁  HTTP methods          │ [GET]
 🔓  Insecure              │ true
 🔃  Recursion Depth       │ 4
───────────────────────────┴──────────────────────
 🏁  Press [ENTER] to use the Scan Management Menu™
──────────────────────────────────────────────────
403      GET        9l       28w      272c Auto-filtering found 404-like response and created new filter; toggle off with --dont-filter
301      GET        0l        0w        0c Auto-filtering found 404-like response and created new filter; toggle off with --dont-filter
[####################] - 2s         2/2       0s      found:0       errors:0      
[####################] - 2s         2/2       1/s     https://dma.gp/                                                                        

I get the same results for the other site provided

feroxbuster -u https://www.groupe-paralliance.fr -w small.txt --dont-filter --dont-extract-links

 ___  ___  __   __     __      __         __   ___
|__  |__  |__) |__) | /  `    /  \ \_/ | |  \ |__
|    |___ |  \ |  \ | \__,    \__/ / \ | |__/ |___
by Ben "epi" Risher 🤓                 ver: 2.10.4
───────────────────────────┬──────────────────────
 🎯  Target Url            │ https://www.groupe-paralliance.fr
 🚀  Threads               │ 50
 📖  Wordlist              │ twl.smol
 👌  Status Codes          │ All Status Codes!
 💥  Timeout (secs)        │ 7
 🦡  User-Agent            │ feroxbuster/2.10.4
 💉  Config File           │ /home/epi/PycharmProjects/feroxbuster/ferox-config.toml
 🏁  HTTP methods          │ [GET]
 🤪  Filter Wildcards      │ false
 🔃  Recursion Depth       │ 4
───────────────────────────┴──────────────────────
 🏁  Press [ENTER] to use the Scan Management Menu™
──────────────────────────────────────────────────
404      GET      651l     4403w    99427c https://www.groupe-paralliance.fr/test
200      GET     1553l     8152w   227424c https://www.groupe-paralliance.fr/
[####################] - 2s         2/2       0s      found:1       errors:0      
[####################] - 2s         2/2       1/s     https://www.groupe-paralliance.fr/                                                     

So, at this point, we can say for certain that feroxbuster can handle talking to the sites directly. When we add --proxy http://127.0.0.1:8080, then we see the problematic behavior.

I don't have the time right this minute to setup certs to see how burp handles its side of the conversation on the wire, but it appears to be a burp problem, or at worst a ferox problem realted to how it interacts with burp in certain configurations.

I'm also curious if we use zap or another intercepting proxy, what the behavior looks like.

[epi052]

small update: using ZAP as the intercepting proxy also works fine. it seems to be a burp issue

[thibon]

I can confirm, did the test too.
I don't understand at which point the combination of feroxbuster and Burp decide to send the other request in HTTP/2, I will try to dive into it in a few days. For now I will just force downgrade in Burp

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[epi052]

small update, here's some more verbose logging of the error

WRN      0.568 feroxbuster::utils exit: make_request -> error sending request for url (https://www.groupe-paralliance.fr/)
WRN      0.569 feroxbuster::utils exit: make_request -> client error (SendRequest)
WRN      0.569 feroxbuster::utils exit: make_request -> http2 error
WRN      0.569 feroxbuster::utils exit: make_request -> stream error detected: unspecific protocol error detected

[scar0x]

Hi there,

I had a hard time understanding what was happening when I tried Feroxbuster in a CTF lab where the target was HTTP/1.1 over TLS.

I’m running into the same issue described here when using Feroxbuster through Burp Suite.
After some tests and packet analysis with Wireshark, it looks like the problem happens when:

• Feroxbuster talks HTTP/2 to Burp (ALPN h2)
• Burp then tries to relay the request to the target, which only supports HTTP/1.1 over HTTPS
• The target responds with a TLS Alert (Fatal: Unexpected Message) and closes the connection
• Feroxbuster shows:
  Could not connect to https://<target>:8081, skipping...

If I disable HTTP/2 support on Burp’s proxy listener
(Settings > Proxy > Proxy Listeners > Edit > HTTP/2 > uncheck "Support HTTP/2"),
Burp forces HTTP/1.1, and Feroxbuster works perfectly.

So the issue seems to be related to HTTP/2 negotiation between Feroxbuster and the proxy when the target only supports HTTP/1.1.

Here is the command I used with Feroxbuster:

feroxbuster -u https://192.168.198.57:8081 \
  -w /usr/share/seclists/Discovery/Web-Content/raft-large-directories.txt \
  -d 1 -t 10 --rate-limit 50 --burp

Interestingly, with the same target, ffuf works fine without forcing HTTP/1.1 in burp proxy:

ffuf -w /usr/share/seclists/Discovery/Web-Content/raft-large-directories.txt \
  -u https://192.168.198.57:8081/FUZZ -t 10 \
  -x http://127.0.0.1:8080 -k

Even if Burp can be configured to work well in this case, maybe adding an option to force HTTP/1.1 in Feroxbuster could help in this situation.

Hope it helps. I can send wireshark datas if needed.

Thanks !