dead-letter cycle detection vs requeue

[maagy]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.0

Erlang version used

27.3.x

Operating system (distribution) used

RKE2 kubernetes, nodes running on ubuntu 24.04

How is RabbitMQ deployed?

Kubernetes Operator(s) from Team RabbitMQ

rabbitmq-diagnostics status output

Basics

Cluster name: rabbitmqcluster
Total CPU cores available cluster-wide: 704

Cluster Tags

(none)

Disk Nodes

rabbit@rabbitmqcluster-server-0.rabbitmqcluster-nodes.some-namespace
rabbit@rabbitmqcluster-server-1.rabbitmqcluster-nodes.some-namespace
rabbit@rabbitmqcluster-server-2.rabbitmqcluster-nodes.some-namespace

Running Nodes

rabbit@rabbitmqcluster-server-0.rabbitmqcluster-nodes.some-namespace
rabbit@rabbitmqcluster-server-1.rabbitmqcluster-nodes.some-namespace
rabbit@rabbitmqcluster-server-2.rabbitmqcluster-nodes.some-namespace

Versions

rabbit@rabbitmqcluster-server-0.rabbitmqcluster-nodes.some-namespace: RabbitMQ 4.1.0 on Erlang 27.3.3
rabbit@rabbitmqcluster-server-1.rabbitmqcluster-nodes.some-namespace: RabbitMQ 4.1.0 on Erlang 27.3.3
rabbit@rabbitmqcluster-server-2.rabbitmqcluster-nodes.some-namespace: RabbitMQ 4.1.0 on Erlang 27.3.3

CPU Cores

Node: rabbit@rabbitmqcluster-server-0.rabbitmqcluster-nodes.some-namespace, available CPU cores: 192
Node: rabbit@rabbitmqcluster-server-1.rabbitmqcluster-nodes.some-namespace, available CPU cores: 128
Node: rabbit@rabbitmqcluster-server-2.rabbitmqcluster-nodes.some-namespace, available CPU cores: 384

Maintenance status

Node: rabbit@rabbitmqcluster-server-0.rabbitmqcluster-nodes.some-namespace, status: not under maintenance
Node: rabbit@rabbitmqcluster-server-1.rabbitmqcluster-nodes.some-namespace, status: not under maintenance
Node: rabbit@rabbitmqcluster-server-2.rabbitmqcluster-nodes.some-namespace, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@rabbitmqcluster-server-0.rabbitmqcluster-nodes.some-namespace, interface: 0.0.0.0, port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmqcluster-server-0.rabbitmqcluster-nodes.some-namespace, interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Node: rabbit@rabbitmqcluster-server-0.rabbitmqcluster-nodes.some-namespace, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmqcluster-server-0.rabbitmqcluster-nodes.some-namespace, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmqcluster-server-1.rabbitmqcluster-nodes.some-namespace, interface: 0.0.0.0, port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmqcluster-server-1.rabbitmqcluster-nodes.some-namespace, interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Node: rabbit@rabbitmqcluster-server-1.rabbitmqcluster-nodes.some-namespace, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmqcluster-server-1.rabbitmqcluster-nodes.some-namespace, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmqcluster-server-2.rabbitmqcluster-nodes.some-namespace, interface: 0.0.0.0, port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmqcluster-server-2.rabbitmqcluster-nodes.some-namespace, interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Node: rabbit@rabbitmqcluster-server-2.rabbitmqcluster-nodes.some-namespace, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmqcluster-server-2.rabbitmqcluster-nodes.some-namespace, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: classic_queue_type_delivery_support, state: enabled
Flag: detailed_queues_endpoint, state: enabled
Flag: direct_exchange_routing_v2, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: feature_flags_v2, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: khepri_db, state: disabled
Flag: listener_records_in_ets, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: message_containers, state: enabled
Flag: message_containers_deaths_v2, state: enabled
Flag: quorum_queue, state: enabled
Flag: quorum_queue_non_voters, state: enabled
Flag: rabbit_exchange_type_local_random, state: enabled
Flag: rabbitmq_4.0.0, state: enabled
Flag: rabbitmq_4.1.0, state: enabled
Flag: restart_streams, state: enabled
Flag: stream_filtering, state: enabled
Flag: stream_queue, state: enabled
Flag: stream_sac_coordinator_unblock_group, state: enabled
Flag: stream_single_active_consumer, state: enabled
Flag: stream_update_config_command, state: enabled
Flag: tracking_records_in_ets, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

Logs from node 1 (with sensitive values edited out)

2025-09-24 07:00:28.575547+00:00 [warning] <0.8299038.0> Message dropped because the following list of queues (ordered by death recency) contains a dead letter cycle without reason 'rejected'. This list will not be logged again: [<<"dead-letter">>,
2025-09-24 07:00:28.575547+00:00 [warning] <0.8299038.0>                                                                                                                                                                              <<"src">>,
2025-09-24 07:00:28.575547+00:00 [warning] <0.8299038.0>                                                                                                                                                                              <<"dead-letter">>,
2025-09-24 07:00:28.575547+00:00 [warning] <0.8299038.0>                                                                                                                                                                              <<"src">>]

Logs from node 2 (if applicable, with sensitive values edited out)

No response

Logs from node 3 (if applicable, with sensitive values edited out)

No response

rabbitmq.conf

CRD based

Steps to deploy RabbitMQ cluster

operator used

Steps to reproduce the behavior in question

queues (quorum type)

┌─────────────┬────────┬──────────┬────────────────┬─────────────────────────┐
│ name        │ type   │ messages │ messages_ready │ messages_unacknowledged │
├─────────────┼────────┼──────────┼────────────────┼─────────────────────────┤
│ src         │ quorum │ 1        │ 1              │ 0                       │
├─────────────┼────────┼──────────┼────────────────┼─────────────────────────┤
│ dead-letter │ quorum │ 5        │ 0              │ 0                       │
└─────────────┴────────┴──────────┴────────────────┴─────────────────────────┘

exchanges

┌────────────────────┬─────────┐
│ name               │ type    │
├────────────────────┼─────────┤
│ amq.match          │ headers │
├────────────────────┼─────────┤
│ amq.rabbitmq.trace │ topic   │
├────────────────────┼─────────┤
│ src-by-dlx         │ direct  │
├────────────────────┼─────────┤
│ src-dlx            │ direct  │
├────────────────────┼─────────┤
│ amq.topic          │ topic   │
├────────────────────┼─────────┤
│ amq.headers        │ headers │
├────────────────────┼─────────┤
│ amq.fanout         │ fanout  │
├────────────────────┼─────────┤
│ amq.direct         │ direct  │
├────────────────────┼─────────┤
│                    │ direct  │
└────────────────────┴─────────┘

bindings

┌─────────────┬─────────────┬──────────────────┬──────────────────┬─────────────┬───────────┐
│ source_name │ source_kind │ destination_name │ destination_kind │ routing_key │ arguments │
├─────────────┼─────────────┼──────────────────┼──────────────────┼─────────────┼───────────┤
│             │ exchange    │ src              │ queue            │ src         │           │
├─────────────┼─────────────┼──────────────────┼──────────────────┼─────────────┼───────────┤
│             │ exchange    │ dead-letter      │ queue            │ dead-letter │           │
├─────────────┼─────────────┼──────────────────┼──────────────────┼─────────────┼───────────┤
│ src-by-dlx  │ exchange    │ src              │ queue            │ nejaky      │           │
├─────────────┼─────────────┼──────────────────┼──────────────────┼─────────────┼───────────┤
│ src-dlx     │ exchange    │ dead-letter      │ queue            │ dl          │           │
└─────────────┴─────────────┴──────────────────┴──────────────────┴─────────────┴───────────┘

policies

┌───────┬─────────────┬───────────────┬───────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────┐
│ vhost │ name        │ pattern       │ apply-to      │ definition                                                                                                                                                                        │ priority │
├───────┼─────────────┼───────────────┼───────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────┤
│ /     │ dead-letter │ ^dead-letter$ │ quorum_queues │ {"dead-letter-exchange":"src-by-dlx","dead-letter-routing-key":"nejaky","dead-letter-strategy":"at-least-once","max-length":2000,"message-ttl":30000,"overflow":"reject-publish"} │ 1        │
├───────┼─────────────┼───────────────┼───────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────┤
│ /     │ src-dlx     │ ^src$         │ quorum_queues │ {"dead-letter-exchange":"src-dlx","dead-letter-routing-key":"dl","dead-letter-strategy":"at-least-once","delivery-limit":2,"overflow":"reject-publish"}                           │ 1        │
└───────┴─────────────┴───────────────┴───────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────┘

advanced.config

No response

Application code

No response

Kubernetes deployment file

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmqcluster
  namespace: ....
spec:
  replicas: 3
  resources:
    limits:
      cpu: '6'
      memory: 6Gi
    requests:
      cpu: '4'
      memory: 4Gi
  persistence:
    storage: 15Gi
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - rabbitmqcluster
          topologyKey: kubernetes.io/hostname
  rabbitmq:
    additionalConfig: |
      disk_free_limit.absolute = 2GB
    additionalPlugins:
      - rabbitmq_shovel
      - rabbitmq_shovel_management          
  override:
    statefulSet:
      spec:
        #https://github.com/rabbitmq/cluster-operator/issues/480
        #https://stackoverflow.com/questions/60407082/rabbit-mq-error-while-waiting-for-mnesia-tables
        podManagementPolicy: Parallel

What problem are you trying to solve?

I try to create "retry" logic by using DLX with TTL. That is, I create intentional dead-letter cycle where source queue's "src" dead messages are sent to "dead-letter" queue. The dead-letter queue has TTL and DLX set so as the messages go back to src queue.

According to documentation, there has to be rejection so as the cycle is broken. It works great when i reject the message with requeue set to false.

However, the message is dropped when i nack or reject the message with requeue set to true. Let's take an example when I have multiple consumers of src queue and one consumer has some trouble processing the message (not enough resources, disconnected dependency, etc.). In that case I'd like to nack the message with requeue so another consumer can take it. The requeue works great, until delivery-limit is hit. Then the message should go to dlq, but is dropped instead because of dead-letter cycle detection.

I can see reason behind cycle detection, but I don't think it is rabbit's responsibility to enforce it this way. And what about multiple nacks? The reject doesn't support bulk processing.

[kjnilsson]

Then the message should go to dlq, but is dropped instead because of dead-letter cycle detection.

Are you sure it is the cycle detection that is tripping you up? I just did a very simple test where I returned messages to a qq until the delivery limit and the message was forwarded to the configured dead letter queue.

Can you give us the simplest possible runnable example of this issue?

btw, in RabbitMQ 4.3 we are planning a change to the behaviour of quorum queue deliver limits. It isn't correct to evaluate the delivery limit against explicit returns (requeue=true). The delivery limit from 4.3 will only be evaluated against failed processing (e.g. when the channel is terminated before an ack / settlement).