Skip to content

[Bug]: DN injected 80% of the memory, CDC cannot be synchronized downstream #22655

@heni02

Description

@heni02

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Branch Name

3.0-dev

Commit ID

490d470

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

测试场景:
模拟dn内存占满80%故障场景,同步了快一个小时后再没有同步数据到下游(上游一直有新数据插入)

Image Image Image

日志:
https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22pNh%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-chaos-490d470-202510201135%5C%22%7D%20%7C%3D%20%60cdc_test%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221760944703772%22,%22to%22:%221760957702002%22%7D%7D%7D&schemaVersion=1&orgId=1

cdc监控:有error还有执行很慢的sql
Image
https://shanghai.idc.matrixorigin.cn:30001/d/ce3ttwv605r0gb/frontend-metrics?orgId=1&from=1760953043414&to=1760961026966&var-interval=1m&var-namespace=mo-chaos-490d470-202510201135&var-pod=All

Expected Behavior

No response

Steps to Reproduce

mo-chaos tool chaos_mem_dn test

Additional information

No response

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't workingseverity/s0Extreme impact: Cause the application to break down and seriously affect the use

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions