docker创建kafka，apache/kafka:latest，haproxy控制白名单
目录
docker创建kafka，apache/kafka:latest，haproxy控制白名单
1、创建kafka容器
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
docker run -d \ --name kafka \ --restart always \ -e TZ=Asia/Shanghai \ -e KAFKA_NODE_ID=1 \ -e KAFKA_PROCESS_ROLES=broker,controller \ -e KAFKA_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://158.178.1.1:9092 \ -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \ -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \ -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ -e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \ -e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \ -e KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0 \ -e KAFKA_NUM_PARTITIONS=3 \ -v /data/kafka/kafka-logs:/var/lib/kafka/data \ apache/kafka:latest |
2、通过haproxy控制9092访问白名单
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# 查询 Kafka 容器的内网 IP：docker inspect kafka | grep IPAddress "IPAddress": "172.17.0.8" # 修改 HAProxy 配置 /etc/haproxy/haproxy.cfg frontend kafka-in bind *:9092 mode tcp default_backend kafka-out backend kafka-out balance roundrobin mode tcp tcp-request content accept if { src -f /etc/haproxy/white_ip_list } tcp-request content reject server docker-kafka 172.17.0.8:9092 check maxconn 20480000 # 配置白名单文件 /etc/haproxy/white_ip_list 36.112.21.1 36.112.22.2 # 重启 HAProxy systemctl restart haproxy |
3、创建topic：test
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# 创建topic：test docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \ --create \ --topic test \ --partitions 1 \ --replication-factor 1 \ --bootstrap-server 158.178.1.1:9092 # 查看topic：test docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \ --list \ --bootstrap-server 158.178.1.1:9092 |
4、生产消息测试
|
1 2 3 4 5 6 7 8 9 |
# 执行生产者 docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \ --topic test \ --bootstrap-server 158.178.1.1:9092 # 输入消息 hello world kafka test |
5、消费消息测试
|
1 2 3 4 5 6 |
docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \ --topic test \ --from-beginning \ --bootstrap-server 158.178.1.1:9092 # 看到生产者生产的消息 |
6、模拟删除容器，重建容器，从头消费topic：test数据还在
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
# 删除容器 docker rm -f kafka # 重建kafka容器 docker run -d \ --name kafka \ --restart always \ -e TZ=Asia/Shanghai \ -e KAFKA_NODE_ID=1 \ -e KAFKA_PROCESS_ROLES=broker,controller \ -e KAFKA_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://158.178.1.1:9092 \ -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \ -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \ -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ -e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \ -e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \ -e KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0 \ -e KAFKA_NUM_PARTITIONS=3 \ -v /data/kafka/kafka-logs:/var/lib/kafka/data \ apache/kafka:latest # 再次从头消费test docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \ --topic test \ --from-beginning \ --bootstrap-server 158.178.1.1:9092 # 数据还在 |