kafka压测

一个kafka实例

创建一个主题，设置副本1，分区2
1
kafka-topics --create --topic raintest1 --bootstrap-server localhost:9092 --replication-factor 1 --partitions 2
生产者测试–一个kafka实例

kafka-producer-perf-test --help有空再查看详细参数

1
2
3
4
5
6
7
8
9
10
11
//先来个50个消息试试
kafka-producer-perf-test --topic raintest1 --num-records 50 --record-size 200 --throughput -1 --producer-props bootstrap.servers=localhost:9092 acks=-1
//返回如下吞吐量每秒0.04M，平均时延25.66ms，最大时延263ms，50%的消息延时在21ms内，95%消息延时在22ms内
50 records sent, 185.185185 records/sec (0.04 MB/sec), 25.66 ms avg latency, 263.00 ms max latency, 21 ms 50th, 22 ms 95th, 263 ms 99th, 263 ms 99.9th.


//好了，开始玩大点，用自己笔记本咯，统一50万消息，每个消息200字节
kafka-producer-perf-test --topic raintest1 --num-records 50000 --record-size 200 --throughput -1 --producer-props bootstrap.servers=localhost:9092 acks=-1

// out
500000 records sent, 241080.038573 records/sec (45.98 MB/sec), 304.23 ms avg latency, 478.00 ms max latency, 321 ms 50th, 459 ms 95th, 471 ms 99th, 475 ms 99.9th
docker compose修改，两个kafka实例
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  kafka2:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29093:29093
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9093,PLAINTEXT_HOST://localhost:29093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
尝试验证一下两个实例之间的连接
1
2
3
kafka-topics --list --bootstrap-server kafka_kafka2_1:9093
kafka-topics --list --bootstrap-server kafka_kafka_1:9092
kafka-topics --list --bootstrap-server kafka_kafka2_1:9093,kafka_kafka_1:9092
生产者测试–两个kafka实例
1
2
3
4
5
6
7
8
kafka-producer-perf-test --topic raintest1 --num-records 500000 --record-size 200 --throughput -1 --producer-props bootstrap.servers=kafka_kafka2_1:9093,kafka_kafka_1:9092 acks=-1

//out
500000 records sent, 114077.116131 records/sec (21.76 MB/sec), 1122.87 ms avg latency, 2508.00 ms max latency, 630 ms 50th, 2481 ms 95th, 2500 ms 99th, 2506 ms 99.9th.

//continue
[appuser@15e3a4abd18b ~]$ kafka-producer-perf-test --topic raintest1 --num-records 5000 --record-size 200 --throughput -1 --producer-props bootstrap.servers=kafka_kafka2_1:9093,kafka_kafka_1:9092 acks=-1
5000 records sent, 16393.442623 records/sec (3.13 MB/sec), 8.71 ms avg latency, 229.00 ms max latency, 6 ms 50th, 19 ms 95th, 21 ms 99th, 21 ms 99.9th.

50万的数据，两个kafka实例，明显压测数据比一个实例难看，根本原因可能受制于我这台笔记本的网络带宽、磁盘写入速度。于是采用5000数据来测试。一个实例的数据如下：

1
2
[appuser@774a362daac6 ~]$ kafka-producer-perf-test --topic raintest1 --num-records 5000 --record-size 200 --throughput -1 --producer-props bootstrap.servers=localhost:9092 acks=-1
5000 records sent, 15923.566879 records/sec (3.04 MB/sec), 13.58 ms avg latency, 231.00 ms max latency, 12 ms 50th, 25 ms 95th, 27 ms 99th, 27 ms 99.9th.

经过多次验证比对，5000数据下两个kafka实例确实会提升producer性能。

消费者测试–1个kafka实例
1
2
3
4
[appuser@774a362daac6 ~]$ kafka-consumer-perf-test --broker-list kafka_kafka_1:9092 --messages 5000 --topic raintest1 --num-fetch-threads 1
WARNING: option [threads] and [num-fetch-threads] have been deprecated and will be ignored by the test
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2023-09-17 09:46:30:869, 2023-09-17 09:46:34:230, 0.9537, 0.2837, 5000, 1487.6525, 3302, 59, 16.1640, 84745.7627

这些数据是关于Kafka消费者压力测试的结果。以下是每个字段的含义：

  • start.time: 测试开始时间。
  • end.time: 测试结束时间。
  • data.consumed.in.MB: 消费者在测试期间消费的数据量（以 MB 为单位）。
  • MB.sec: 消费者在测试期间的平均吞吐量（以 MB/秒为单位）。
  • data.consumed.in.nMsg: 消费者在测试期间消费的消息数量。
  • nMsg.sec: 消费者在测试期间的平均吞吐量（以消息/秒为单位）。
  • rebalance.time.ms: 消费者重新平衡所需的时间（以毫秒为单位）。
  • fetch.time.ms: 消费者从代理获取数据所需的时间（以毫秒为单位）。
  • fetch.MB.sec: 消费者从代理获取数据的平均速率（以 MB/秒为单位）。
  • fetch.nMsg.sec: 消费者从代理获取数据的平均速率（以消息/秒为单位）。
    消费者测试–2个kafka实例
1
2
3
4
5
//5000数据，两个实例对比，数据略有提升
[appuser@774a362daac6 ~]$ kafka-consumer-perf-test --broker-list kafka_kafka2_1:9093,kafka_kafka_1:9092 --messages 5000 --topic raintest1 --num-fetch-threads 1
WARNING: option [threads] and [num-fetch-threads] have been deprecated and will be ignored by the test
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2023-09-17 09:50:03:238, 2023-09-17 09:50:06:582, 0.9537, 0.2852, 5000, 1495.2153, 3288, 56, 17.0299, 89285.7143
中间件 kafka

本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议 ，转载请注明出处！