`@opentelemetry/instrumentation-express` breaks some HTTP requests

[hognevevle]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/nestjs

SDK Version

9.40.0

Framework Version

Nest 10.0.0

Link to Sentry event

No response

Reproduction Example/SDK Setup

Steps to Reproduce

I have been unable to produce a minimal reproduction. This error never occured until we introduced Sentry, and the error also disappears if we comment the Sentry.init() in the project. I realize that's not very helpful, and would be happy to provide additional details given your guidance.

Anyway, this is what we experience on our end:

With the default Sentry integrations enabled (set up following the official docs), endpoints of certain content types (e.g. application/x-www-form-urlencoded) fail with stream is not readable:

➜ curl -X POST -H "Content-Type: application/x-www-form-urlencoded"  http://localhost:4000/api/foo "foo=bar"
{"statusCode":500,"message":"stream is not readable"}%                                                                  

(See stacktrace under Actual Result).

This error does not get recorded by Sentry.

application/json requests, however, are fine.

➜ curl -X POST -H "Content-Type: application/json"  http://localhost:4000/api/foo -d '{}'                 
{"message":"Missing Mailgun signature","error":"Unauthorized","statusCode":401}%                                        

(This specific error is expected for my test request)

Seeing how there was a reference to @opentelemetry/instrumentation-express in the stack trace, I did some additional research where I found that disabling the Express integration will fix also the issue (while leaving Sentry and other integrations intact).

Sentry.init({
    dsn: process.env.SENTRY_ENDPOINT,
    environment: process.env.SENTRY_ENVIRONMENT,
    release: process.env.SENTRY_RELEASE,
    tracesSampleRate: 1.0,
    _experiments: { enableLogs: true },
    integrations: (defaults) => {
      return [
        // The below resolves the issue
        ...defaults.filter((i) => i.name !== 'Express')
      ];
    },
})

Expected Result

Would expect requests not to crash.

Actual Result

  ERROR [ExceptionsHandler] stream is not readable - {
  stack: [
    'InternalServerError: stream is not readable\n' +
      '    at readStream (/Users/hognevevle/Projects/myproject/node_modules/.pnpm/raw-body@2.5.2/node_modules/raw-body/index.js:185:17)\n' +
      '    at getRawBody (/Users/hognevevle/Projects/myproject/node_modules/.pnpm/raw-body@2.5.2/node_modules/raw-body/index.js:116:12)\n' +
      '    at read (/Users/hognevevle/Projects/myproject/node_modules/.pnpm/body-parser@1.20.2/node_modules/body-parser/lib/read.js:79:3)\n' +
      '    at urlencodedParser (/Users/hognevevle/Projects/myproject/node_modules/.pnpm/body-parser@1.20.2/node_modules/body-parser/lib/types/urlencoded.js:116:5)\n' +
      '    at patched (/Users/hognevevle/Projects/myproject/node_modules/.pnpm/@opentelemetry+instrumentation-express@0.47.1_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation-express/src/instrumentation.ts:286:27)\n' +
      '    at Layer.handle [as handle_request] (/Users/hognevevle/Projects/myproject/node_modules/.pnpm/express@4.19.2/node_modules/express/lib/router/layer.js:95:5)\n' +
      '    at trim_prefix (/Users/hognevevle/Projects/myproject/node_modules/.pnpm/express@4.19.2/node_modules/express/lib/router/index.js:328:13)\n' +
      '    at /Users/hognevevle/Projects/myproject/node_modules/.pnpm/express@4.19.2/node_modules/express/lib/router/index.js:286:9\n' +
      '    at Function.process_params (/Users/hognevevle/Projects/myproject/node_modules/.pnpm/express@4.19.2/node_modules/express/lib/router/index.js:346:12)\n' +
      '    at next (/Users/hognevevle/Projects/myproject/node_modules/.pnpm/express@4.19.2/node_modules/express/lib/router/index.js:280:10)'
  ]
}

[linear]

JS-765 `@opentelemetry/instrumentation-express` breaks some HTTP requests

[Divyanshu7001]

This issue and break may be happening because the body of the request (a readable stream) is consumed before it reaches the body-parser middleware,
due to overlapping instrumentation of Express from both OpenTelemetry and Sentry.

Both OpenTelemetry and Sentry provide middleware wrappers around Express routes to capture errors and traces. Some of these wrappers, depending on the order in which they’re applied, might intercept and consume the request stream too early.

In particular:

• @opentelemetry/instrumentation-express wraps each middleware and route handler, and if the timing is off, it may interfere with request stream access.
• Sentry’s integration also wrap the Express stack, but more passively.

When both are active, their combined instrumentation can interfere with stream lifecycle management, especially for requests with content types like application/x-www-form-urlencoded, which rely on raw-body and require the request stream to be untouched.
That's why disabling one of the options from Sentry & opentelemetry removes the occuring problem.

[chargome]

Hey @hognevevle thanks for reporting!

I have been unable to produce a minimal reproduction.

But you can consistently repro this in your own app?

[hognevevle]

Hey @hognevevle thanks for reporting!

I have been unable to produce a minimal reproduction.

But you can consistently repro this in your own app?

Correct

[chargome]

Any way you could isolate this within your app so we can look at a reproduction?