Skip to content

[Enhancement] massive memory allocation in the parsePublishMessageQueues #9926

@ShiKaiWi

Description

@ShiKaiWi

Before Creating the Enhancement Request

  • I have confirmed that this should be classified as an enhancement rather than a bug/feature.

Summary

During my benchmark (50K LMQ tps), the rocketmq client in my process shows massive memory allocation in the parsePublishMessageQueues.

Here is my profiling result about the memory allocation:

Image

Motivation

This multiple growths of the array list introduce a great influence on the performance under the high throughput workload.

Describe the Solution You'd Like

Here is a simple proposal:

  • do pre-allocation for the returned list of the parsePublishMessageQueues.
  • Just return the original List<MessageQueue> if the namespace is empty.

Describe Alternatives You've Considered

There are more complex solutions:

  • Upgrade the interface about the MessageQueueSelector to use size of the message queue and accessor by index to MessageQueue instead of List<MessageQueue>, so that no need to create another List<MessageQueue>.
  • Use thread-local buffer array list for the parsed List<MessageQueue> to avoid allocating memory for any to-be-sent message.

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions