Callback pointer reference issue in AsyncTCP

[mhummels]

IDE / Tooling

PlatformIO

What happened?

I am running ESP-DASH-Pro 4.6.2 on an ESP32 using ESPAsyncWebServer 3.7.7 and AsyncTCP 3.4.0. My application requires that the ESP32 does not reset itself because it is connected to a relay holding the power up of the ESP32. So as soon as the ESP resets, the relay gets released and the ESP32 dies due to power down.

That said, I tried to figure out why the ESP32 dies from time to time (currently approx. once every 1-3 days). To do that I changed the power supply to keep it running even after a reset and added some code to write the coredumps to LittleFS to be able to check them regularly. I am already doing this for months now, but I was not able to catch a good coredump leading me closer to the issue.

As far as I got, it seems like there is a pointer of a callback which points to some very strange memory region (0x8018d038) which gets executed resulting in an InstFetchPorohibited Hardfault. As the thread with these strange pcb is named asyncTCP it seems like its happening somewhere in the AsyncTCP code. Unfortunately the stack trace is now showing the last moment before the strange memory region get called. So I am clueless where to start looking, as it could be a memory leak anywhere in the codebase, or the pointer gets called after it was released already.

I now updated AsyncTCP again to the new version and wait for the next reset to happen.

Any help is appreciated.

Stack Trace

===============================================================
==================== ESP32 CORE DUMP START ====================

Crashed task handle: 0x3ffe05ec, name: '', GDB name: 'process 1073612268'
Crashed task is not in the interrupt context

================== CURRENT THREAD REGISTERS ===================
exccause       0x14 (InstFetchProhibitedCause)
excvaddr       0x8018d038
epc1           0x40084d77
epc2           0x0
epc3           0x0
epc4           0x0
epc5           0x40082553
epc6           0x0
eps2           0x0
eps3           0x0
eps4           0x0
eps5           0x60a20
eps6           0x0
[New process 1073612268]
[New process 1073431312]
[New process 1073467184]
[New process 1073468712]
[New process 1073447304]
[New process 1073469084]
[New process 1073443912]
[New process 1073445456]
[New process 1073437912]
[New process 1073528948]
[New process 1073437540]
[New process 1073464584]
[New process 1073439140]
[Current thread is 1 (process 1073612268)]

==================== CURRENT THREAD STACK =====================


======================== THREADS INFO =========================
pc             0x8018d039          0x8018d039
lbeg           0x4008a448          1074308168
lend           0x4008a45e          1074308190
lcount         0xffffffff          4294967295
sar            0x14                20
ps             0x60920             395552
threadptr      <unavailable>
br             <unavailable>
scompare1      <unavailable>
acclo          <unavailable>
acchi          <unavailable>
m0             <unavailable>
m1             <unavailable>
m2             <unavailable>
m3             <unavailable>
expstate       <unavailable>
f64r_lo        <unavailable>
f64r_hi        <unavailable>
f64s           <unavailable>
fcr            <unavailable>
fsr            <unavailable>
a0             0x8018d500          -2145856256
a1             0x3ffdf990          1073609104
a2             0x312e3104          825110788
a3             0xfc                252
a4             0x3ffe0a54          1073613396
a5             0x31aa00            3254784
a6             0x3ffe20b8          1073619128
a7             0xf7e5d5e0          -135932448
a8             0x800e0074          -2146566028
a9             0x3ffdb9ac          1073592748
a10            0x3ffc7d08          1073511688
a11            0x1ffefcce          536804558
a12            0x3ffdf998          1073609112
a13            0x60123             393507
a14            0xb33fffff          -1287651329
a15            0xcdcd              52685
Retrying reading threads information...
Retrying reading threads information...


       TCB             NAME PRIO C/B  STACK USED/FREE
---------- ---------------- -------- ----------------
0x3ffe05ec Corrupted TCB data
0x3ffb4310        async_tcp1073430480/1073426444       178044/324
0x3ffbcf30                 1073466688/1073466028   1073467218/132
0x3ffbd528                 1073468208/1073467556   1073468764/140
0x3ffb8188                 1073421648/1073414944 1073447356/24236
0x3ffbd69c                 1073550944/1073547264 1073469135/82208
0x3ffb7448                 1073443328/1073442756   1073443963/188
0x3ffb7a50                 1073444864/1073444300   1073445484/196
0x3ffb5cd8                 1073411520/1073408156 1073437940/25724
0x3ffcc074                 1073585392/1073581796 1073528981/56888
0x3ffb5b64                 1073436704/1073432672   1073437591/328
0x3ffbc508                 1073463616/1073455364   1073464616/572
0x3ffb61a4                 1073558736/10735525961073439170/120056

==================== THREAD 1 (TCB: 0x3ffe05ec, name: '') =====================


==================== THREAD 2 (TCB: 0x3ffb4310, name: 'async_tcp') =====================
#0  0x8018d039 in ?? ()
#1  0x8018d500 in ?? ()
#2  0x8018d78d in ?? ()
#3  0x8018d7d5 in ?? ()
#4  0x8008fb81 in ?? ()

==================== THREAD 3 (TCB: 0x3ffbcf30, name: '') =====================
#0  0x4010750e in tcp_receive (pcb=0x3ffb4010) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:1264

==================== THREAD 4 (TCB: 0x3ffbd528, name: '') =====================
#0  0x4008d45a in esp_cpu_wait_for_intr () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_hw_support/cpu.c:64
#1  0x400f40ec in esp_vApplicationIdleHook () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/freertos_hooks.c:58
#2  0x40090de6 in prvIdleTask (pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:4353
#3  0x4008fb81 in vPortTaskWrapper (pxCode=0x40090d6c <prvIdleTask>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 5 (TCB: 0x3ffb8188, name: '') =====================
#0  0x4008d45a in esp_cpu_wait_for_intr () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_hw_support/cpu.c:64
#1  0x400f40ec in esp_vApplicationIdleHook () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/freertos_hooks.c:58
#2  0x40090de6 in prvIdleTask (pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:4353
#3  0x4008fb81 in vPortTaskWrapper (pxCode=0x40090d6c <prvIdleTask>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 6 (TCB: 0x3ffbd69c, name: '') =====================
#0  0x400835e0 in esp_crosscore_int_send_yield (core_id=1) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/crosscore_int.c:121
#1  0x40091cf1 in vTaskDelay (xTicksToDelay=10) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/xtensa/include/xt_utils.h:41
#2  0x400ecf3c in delay (ms=10) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-misc.c:210
#3  0x4019464c in HTTPClient::handleHeaderResponse (this=0x3ffb204c) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/HTTPClient/src/HTTPClient.cpp:1288
#4  0x401948b9 in HTTPClient::sendRequest (this=0x3ffb204c, type=<optimized out>, payload=<optimized out>, size=<optimized out>) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/HTTPClient/src/HTTPClient.cpp:617
#5  0x401949ba in HTTPClient::GET (this=0x3ffb204c) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/HTTPClient/src/HTTPClient.cpp:500
#6  0x400e0be2 in EasyDDNSClass::update (this=0x3ffc7e6c <EasyDDNS>, ddns_update_interval=10000, use_local_ip=<optimized out>) at .pio/libdeps/henry-central-control/EasyDDNS/EasyDDNS.cpp:40
#7  0x400d5d30 in loop () at src/main.cpp:823
#8  0x400ef3eb in loopTask (pvParameters=0x0) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp:74
#9  0x4008fb81 in vPortTaskWrapper (pxCode=0x400ef3b0 <loopTask(void*)>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 7 (TCB: 0x3ffb7448, name: '') =====================
#0  0x400835e0 in esp_crosscore_int_send_yield (core_id=1) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/crosscore_int.c:121
#1  0x40090754 in prvProcessTimerOrBlockTask (xNextExpireTime=0, xListWasEmpty=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/xtensa/include/xt_utils.h:41
#2  prvTimerTask (pvParameters=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/timers.c:685
#3  0x4008fb81 in vPortTaskWrapper (pxCode=0x40090644 <prvTimerTask>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 8 (TCB: 0x3ffb7a50, name: '') =====================
#0  0x4000bff0 in ?? ()
#1  0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560 
#2  vPortExitCritical (mux=0x3ffbdfcc <xKernelLock>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40092251 in ulTaskGenericNotifyTake (uxIndexToWait=0, xClearCountOnExit=1, xTicksToWait=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:5765
#4  0x40082749 in ipc_task (arg=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/esp_ipc.c:62
#5  0x4008fb81 in vPortTaskWrapper (pxCode=0x40082708 <ipc_task>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 9 (TCB: 0x3ffb5cd8, name: '') =====================
#0  0x4000bff0 in ?? ()
#1  0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560 
#2  vPortExitCritical (mux=0x3ffbdfcc <xKernelLock>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40092251 in ulTaskGenericNotifyTake (uxIndexToWait=0, xClearCountOnExit=1, xTicksToWait=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:5765
#4  0x40082749 in ipc_task (arg=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/esp_ipc.c:62
#5  0x4008fb81 in vPortTaskWrapper (pxCode=0x40082708 <ipc_task>, pvParameters=0x1) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 10 (TCB: 0x3ffcc074, name: '') =====================
#0  0x4000bff0 in ?? ()
#1  0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560 
#2  vPortExitCritical (mux=0x3ffb44d8) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x4008f833 in xQueueReceive (xQueue=0x3ffb4484, pvBuffer=0x3ffaf70c, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4  0x40189d48 in NetworkEvents::_checkForEvent (this=<optimized out>) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/Network/src/NetworkEvents.cpp:111
#5  NetworkEvents::_checkForEvent (this=0x3ffcbda4 <Network+4>) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/Network/src/NetworkEvents.cpp:100
#6  0x40189e18 in operator() (__closure=0x0, self=0x3ffcbda4 <Network+4>) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/Network/src/NetworkEvents.cpp:66
#7  _FUN () at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/Network/src/NetworkEvents.cpp:67
#8  0x4008fb81 in vPortTaskWrapper (pxCode=0x40189e10 <_FUN(void*)>, pvParameters=0x3ffcbda4 <Network+4>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 11 (TCB: 0x3ffb5b64, name: '') =====================
#0  0x4000bff0 in ?? ()
#1  0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560 
#2  vPortExitCritical (mux=0x3ffd8b5c) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x4008f833 in xQueueReceive (xQueue=0x3ffd8b08, pvBuffer=0x3ffd9e38, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4  0x4019d218 in _mdns_service_task (pvParameters=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/managed_components/espressif__mdns/mdns.c:5417
#5  0x4008fb81 in vPortTaskWrapper (pxCode=0x4019d1f4 <_mdns_service_task>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139       

==================== THREAD 12 (TCB: 0x3ffbc508, name: '') =====================
#0  0x4000bff0 in ?? ()
#1  0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560 
#2  vPortExitCritical (mux=0x3ffb45c4) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x4008f833 in xQueueReceive (xQueue=0x3ffb4570, pvBuffer=0x3ffb5970, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4  0x401bd708 in esp_event_loop_run (event_loop=0x3ffb2c54, ticks_to_run=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_event/esp_event.c:644
#5  0x401bd8c2 in esp_event_loop_run_task (args=0x3ffb2c54) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_event/esp_event.c:106
#6  0x4008fb81 in vPortTaskWrapper (pxCode=0x401bd8b8 <esp_event_loop_run_task>, pvParameters=0x3ffb2c54) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 13 (TCB: 0x3ffb61a4, name: '') =====================
#0  0x4000bff0 in ?? ()
#1  0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560 
#2  vPortExitCritical (mux=0x3ffbdfcc <xKernelLock>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40092251 in ulTaskGenericNotifyTake (uxIndexToWait=0, xClearCountOnExit=1, xTicksToWait=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:5765
#4  0x40137424 in timer_task (arg=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_timer/src/esp_timer.c:459
#5  0x4008fb81 in vPortTaskWrapper (pxCode=0x40137414 <timer_task>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139


======================= ALL MEMORY REGIONS ========================
Name   Address   Size   Attrs
.rtc.text 0x400c0000 0x0 RW
.rtc.dummy 0x3ff80000 0x0 RW
.rtc.force_fast 0x3ff80000 0x1c RW A
.rtc_noinit 0x50000200 0x0 RW
.rtc.force_slow 0x50000200 0x0 RW
.iram0.vectors 0x40080000 0x403 R XA
.iram0.text 0x40080404 0x16c83 R XA
.dram0.data 0x3ffbdb60 0x676c RW A
.ext_ram_noinit 0x3f800000 0x0 RW
.ext_ram.bss 0x3f800000 0x0 RW
.flash.appdesc 0x3f400020 0x100 R  A
.flash.rodata 0x3f400120 0x6a1e0 RW A
.flash.text 0x400d0020 0xf02c4 R XA
.iram0.data 0x40097088 0x0 RW
.iram0.bss 0x40097088 0x0 RW
.dram0.heap_start 0x3ffcf188 0x0 RW  
.coredump.tasks.data 0x3ffb4310 0x160 RW
.coredump.tasks.data 0x3ffb3fd0 0x230 RW
.coredump.tasks.data 0x3ffe05ec 0x160 RW
.coredump.tasks.data 0x3ffdf8d0 0x1f0 RW
.coredump.tasks.data 0x3ffbcf30 0x160 RW
.coredump.tasks.data 0x3ffbcd40 0x1a0 RW
.coredump.tasks.data 0x3ffbd528 0x160 RW
.coredump.tasks.data 0x3ffbd330 0x1a0 RW
.coredump.tasks.data 0x3ffb8188 0x160 RW
.coredump.tasks.data 0x3ffb1d50 0x5c0 RW
.coredump.tasks.data 0x3ffbd69c 0x160 RW
.coredump.tasks.data 0x3ffd1660 0x190 RW
.coredump.tasks.data 0x3ffb7448 0x160 RW
.coredump.tasks.data 0x3ffb7200 0x1c0 RW
.coredump.tasks.data 0x3ffb7a50 0x160 RW
.coredump.tasks.data 0x3ffb7800 0x1c0 RW
.coredump.tasks.data 0x3ffb5cd8 0x160 RW
.coredump.tasks.data 0x3ffaf5c0 0x2d0 RW
.coredump.tasks.data 0x3ffcc074 0x160 RW
.coredump.tasks.data 0x3ffd9cf0 0x1f0 RW
.coredump.tasks.data 0x3ffb5b64 0x160 RW
.coredump.tasks.data 0x3ffb5820 0x230 RW
.coredump.tasks.data 0x3ffbc508 0x160 RW
.coredump.tasks.data 0x3ffbc140 0x1c0 RW
.coredump.tasks.data 0x3ffb61a4 0x160 RW
.coredump.tasks.data 0x3ffd34d0 0x200 RW

===================== ESP32 CORE DUMP END =====================
===============================================================
Done!

Minimal Reproductible Example (MRE)

This is a hard one, as this is a very sporadic problem and only occurs once in a few days.

I confirm that:

• I have read the documentation.
• I have searched for similar discussions.
• I have searched for similar issues.
• I have looked at the examples.
• I have upgraded to the lasted version of AsyncTCP.

[mathieucarbou]

There are countless of apps running for days even months with asynctcp. I am pretty sure the culprit is how you use it or your app.

the provided stacks do not show any hint wehre asynctcp would have a bug somewhere.

I am sorry to tell you that, but you must find a way to isolate your issue.