Fix unhandled exception in tiered compilation thread

[janvorli]

The detection the unhandled exception is not working in the case of the tiered compilation thread, because there is no indication that the exception is going to be caught in the CEEInfo::runWithErrorTrap. So the EH assumes the exception is going to be unhandled and exits the process.
The unhandled exception message was not seen in the repro case of the issue likely because the tiered compilation thread console output didn't get flushed. When running under a debugger and stepping over the InternalUnhandledExceptionFilter_Worker, it was actually printed out.

The fix is to add DebuggerU2MCatchHandlerFrame to the CEEInfo::runWithErrorTrap. That allows the EH to see that the exception will be caught there and let it flow through the CallDescrWorker to reach that trap.

I have also taken this opportunity and renamed the confusing invalidRevPInvoke local variable.

Closes #116676

[Copilot]

Pull Request Overview

This PR ensures that exceptions in the tiered compilation thread are correctly detected as handled by adding a debugger catch frame and improves clarity by renaming a confusing local variable.

• Register a DebuggerU2MCatchHandlerFrame in CEEInfo::runWithErrorTrap so EH knows exceptions will be caught.
• Rename invalidRevPInvoke to isPropagatingToNativeCode in SfiNext and update related logic and log messages.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/coreclr/vm/jitinterface.cpp	Added DebuggerU2MCatchHandlerFrame catchFrame(true) at the start of runWithErrorTrap.
src/coreclr/vm/exceptionhandling.cpp	Renamed invalidRevPInvoke to isPropagatingToNativeCode and updated assignments, checks, and logs.

Comments suppressed due to low confidence (3)

src/coreclr/vm/jitinterface.cpp:10588

• [nitpick] Add a brief comment explaining why this catch frame is needed (e.g., "// Ensure EH sees that this thread has a catch handler for tiered compilation").

DebuggerU2MCatchHandlerFrame catchFrame(true /* catchesAllExceptions */);

src/coreclr/vm/exceptionhandling.cpp:3913

• [nitpick] The name isPropagatingToNativeCode overlaps with isPropagatingToExternalNativeCode and may be ambiguous; consider a more specific name like hasReversePInvokeFrame to clarify its intent.

CLR_BOOL isPropagatingToNativeCode = FALSE;

src/coreclr/vm/jitinterface.cpp:10588

• No tests were added to verify that exceptions in the tiered compilation thread are now correctly caught; consider adding a repro unit test covering this scenario.

DebuggerU2MCatchHandlerFrame catchFrame(true /* catchesAllExceptions */);

[jkotas]

Should we add a regression test for this?

[AaronRobinsonMSFT]

Suggested change

	EH_LOG((LL_INFO100, "SfiNext: reached native frame at IP=%p, SP=%p, isPropagatingToNativeCode=%d\n",
	GetIP(pThis->m_crawl.GetRegisterSet()->pCurrentContext), GetSP(pThis->m_crawl.GetRegisterSet()->pCurrentContext), isPropagatingToNativeCode));
	EH_LOG((LL_INFO100, "SfiNext: reached native frame at IP=%p, SP=%p, isPropagatingToNativeCode=%s\n",
	GetIP(pThis->m_crawl.GetRegisterSet()->pCurrentContext), GetSP(pThis->m_crawl.GetRegisterSet()->pCurrentContext), (isPropagatingToNativeCode ? "true" : "false")));

[jkotas]

This will produce megabytes of output that will get copied around, etc. Can we do something less visible, like Thread.Sleep(10) and change the for-loop to go till like 200? We should only need to hit the loop about CallCountThreshold times (30) to trigger the background tiered compilation.

[janvorli]

It usually triggers on hundreds of thousands of iterations for me, so even a Sleep(1) was too long. Let me try if a longer sleep and less iterations would give the tiered compilation enough time to crash.

[janvorli]

I've modified it to use the Sleep(10) and less loops. It reproduces the problem without the fix reliably.

[MichalPetryka]

Would it make sense to use the env var for aggressive tiering here?

[janvorli]

I don't think it is needed, the test now repros the problem reliably (with the fix from this PR removed)

[jkotas]

Thanks