Install Kafka in Linux

1.1 Java environment as a prerequisite
1.2 Zookeeper installation and configuration
1.3 Kafka installation and configuration
1.4 Start Kafka
1.5 Reopen a window and view the Zookeeper nodes
1.6 At this time, Kafka is started in foreground mode. To stop it, use Ctrl+C

2. Production and Consumption

2.1 kafka-topics.sh is used to manage topics
2.2 kafka-console-consumer.sh is used to consume messages
2.3 kafka-console-producer.sh is used to produce messages
2.4 Specific Operations

1. Environmental Preparation

JDK download address link: Download address

Zookeeper download address link: Download address

kafka download address link: download address

1.1 Java environment as a prerequisite

1.1.1 Upload jdk-8u261-linux-x64.rpm to the server and install it

# Installation command rpm -ivh jdk-8u261-linux-x64.rpm

insert image description here

1.1.2 Configuring environment variables

# Edit the configuration file, jdk's bin directory to the /etc/profile file, which is valid for all users' shells vim /etc/profile
# Take effect source /etc/profile

insert image description here

export JAVA_HOME=/usr/java/jdk1.8.0_261-amd64
export PATH=$PATH:$JAVA_HOME/bin

insert image description here

# Verify java -version

insert image description here

At this point, jdk is installed successfully.

1.2 Zookeeper installation and configuration

1.2.1 Upload zookeeper-3.4.14.tar.gz to the server and unzip it to /opt

# Unzip zk to the specified directory tar -zxf zookeeper-3.4.14.tar.gz -C /opt

insert image description here

1.2.2 Modify the directory where Zookeeper saves data, dataDir

# Enter the conf configuration directory cd /opt/zookeeper-3.4.14/conf
# Copy zoo_sample.cfg and name it zoo.cfg
cp zoo_sample.cfg zoo.cfg
# Edit the zoo.cfg file vim zoo.cfg
dataDir=/var/riemann/zookeeper/data

insert image description here

1.2.3 Edit /etc/profile to make the configuration effective

Set the environment variable ZOO_LOG_DIR to specify the location where Zookeeper saves logs;
ZOOKEEPER_PREFIX points to the directory where Zookeeper is unpacked;
Add Zookeeper 's bin directory to PATH :

insert image description here

export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.14
export PATH=$PATH:$ZOOKEEPER_PREFIX/bin
export ZOO_LOG_DIR=/var/riemann/zookeeper/log

After the configuration is completed, the configuration will take effect:

source /etc/profile

1.2.4 Start Zookeeper and confirm the status of Zookeeper

zkServer.sh start

insert image description here

At this point, zookeeper is installed successfully.

1.3 Kafka installation and configuration

1.3.1 Upload kafka_2.12-1.0.2.tgz to the server and decompress it

tar -zxf kafka_2.12-1.0.2.tgz -C /opt

insert image description here

1.3.2 Configure environment variables and make them effective

vim /etc/profile

export KAFKA=/opt/kafka_2.12-1.0.2
export PATH=$PATH:$KAFKA/bin

source /etc/profile

1.3.3 Configure the server.properties file in /opt/kafka_2.12-1.0.2/config

vi /opt/kafka_2.12-1.0.2/config/server.properties

insert image description here

The address where Kafka connects to Zookeeper . The connection address of the locally started Zookeeper instance is localhost:2181
The following myKafka is the root node path of Kafka in Zookeeper

insert image description here

Configure the Kafka storage persistent data directory

insert image description here

log.dirs=/var/riemann/kafka/kafka-logs

Create the above persistent data directory

mkdir -p /var/riemann/kafka/kafka-logs

1.4 Start Kafka

Enter the root directory of Kafka installation and execute the following command:

kafka-server-start.sh ../config/server.properties

insert image description here

If the startup is successful, you can see the started status in the last line of the console output: Kafka is installed successfully.

insert image description here

1.5 Reopen a window and view the Zookeeper nodes

insert image description here

1.6 At this time, Kafka is started in foreground mode. To stop it, use Ctrl+C

To start it in the background, use the command:

kafka-server-start.sh -daemon config/server.properties

View Kafka's background process:

ps -ef | grep kafka

insert image description here

Stop the Kafka running in the background:

kafka-server-stop.sh

2. Production and Consumption

Check the zookeeper status. If zookeeper is started successfully, start kafka.

insert image description here

2.1 kafka-topics.sh is used to manage topics

View command help information

[root@master1 bin]# kafka-topics.sh
Create, delete, describe, or change a topic.
Option Description                            
------ -----------                            
--alter Alter the number of partitions,        
                                           replica assignment, and/or           
                                           configuration for the topic.         
--config <String: name=value> A topic configuration override for the 
                                           topic being created or altered.   
                                           The following is a list of valid         
                                           configurations:                      
                                         	cleanup.policy                        
                                         	compression.type                      
                                         	delete.retention.ms                   
                                         	file.delete.delay.ms                  
                                         	flush.messages                        
                                         	flush.ms                              
                                         	follower.replication.throttled.       
                                           replicas                             
                                         	index.interval.bytes                  
                                         	leader.replication.throttled.replicas 
                                         	max.message.bytes                     
                                         	message.format.version                
                                         	message.timestamp.difference.max.ms   
                                         	message.timestamp.type                
                                         	min.cleanable.dirty.ratio             
                                         	min.compaction.lag.ms                 
                                         	min.insync.replicas                   
                                         	preallocate                           
                                         	retention.bytes                       
                                         	retention.ms                          
                                         	segment.bytes                         
                                         	segment.index.bytes                   
                                         	segment.jitter.ms                     
                                         	segment.ms                            
                                         	unclean.leader.election.enable        
                                         See the Kafka documentation for full   
                                           details on the topic configs.        
--create Create a new topic.                    
--delete Delete a topic                         
--delete-config <String: name> A topic configuration override to be   
                                           removed for an existing topic (see   
                                           the list of configurations under the 
                                           --config option).                    
--describe List details for the given topics.     
--disable-rack-aware Disable rack aware replica assignment  
--force Suppress console prompts               
--help Print usage information.               
--if-exists if set when altering or deleting       
                                           topics, the action will only execute 
                                           if the topic exists                  
--if-not-exists if set when creating topics, the       
                                           action will only execute if the      
                                           topic does not already exist         
--list List all available topics.             
--partitions <Integer: # of partitions> The number of partitions for the topic 
                                           being created or altered (WARNING:   
                                           If partitions are increased for a    
                                           topic that has a key, the partition  
                                           logic or ordering of the messages    
                                           will be affected                     
--replica-assignment <String: A list of manual partition-to-broker   
  broker_id_for_part1_replica1 : assignments for the topic being      
  broker_id_for_part1_replica2 , created or altered.                  
  broker_id_for_part2_replica1 :                                                
  broker_id_for_part2_replica2 , ...>                                           
--replication-factor <Integer: The replication factor for each        
  replication factor> partition in the topic being created.
--topic <String: topic> The topic to be created, altered or       
                                           describe. Can also accept a regular  
                                           expression except for --create option
--topics-with-overrides if set when describing topics, only    
                                           show topics that have overridden     
                                           configs                              
--unavailable-partitions if set when describing topics, only    
                                           show partitions whose leader is not  
                                           available                            
--under-replicated-partitions if set when describing topics, only    
                                           show under replicated partitions     
--zookeeper <String: urls> REQUIRED: The connection string for    
                                           the zookeeper connection in the form 
                                           host:port. Multiple URLS can be      
                                           given to allow fail-over.            
[root@master1 bin]#

# List existing topics [root@master1 ~]# kafka-topics.sh --list --zookeeper localhost:2181/myKafka
# Create a topic that contains one partition, which is the Leader partition and has no Follower partition replicas.
[root@master1 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --create --topic topic_test --partitions 1 --replication-factor 1
# View partition information [root@master1 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --list
# View detailed information of the specified topic [root@master1 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --describe --topic topic_test 
# Delete the specified topic [root@master1 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --delete --topic topic_test

List existing topics and create a topic that contains one partition, which is the leader partition and has no follower partition replicas.

insert image description here

View detailed information about a specific topic

insert image description here

Create a topic that contains multiple partitions

Multiple partitions: Horizontal expansion Multiple replicas: High availability

insert image description here

2.2 kafka-console-consumer.sh is used to consume messages

# Enable consumer [root@node1 ~]# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_test 
# Start consumer mode 2, consume from the beginning, not according to the offset [root@node1 ~]# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_test --from-beginning

2.3 kafka-console-producer.sh is used to produce messages

# Open the producer [root@node1 ~]# kafka-console-producer.sh --topic topic_test --broker-list localhost:9020

2.4 Specific Operations

Enable consumers and producers to produce and consume messages.

insert image description here

Consumers consume according to the offset

insert image description here

Consumers consume from the beginning, not according to the offset

insert image description here

This is the end of this article about installing Kafka in Linux environment. For more information about installing Kafka in Linux, please search 123WORDPRESS.COM's previous articles or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!

You may also be interested in: