Monitoring Kafka with Elastic Stack: Filebeat

Apache Kafka Stream processorKafak

Kafka clusters provide a number of opportunities for monitoring. At the network level, you can monitor connections between Kafka nodes, Zookeeper, and clients. At the host level, you can monitor Kafka resource usage, such as CPU, memory and disk usage. And Kafka itself provides log files, an API to query offsets, and JMX support to monitor internal process metrics.

Related Post:  How to avoid spyware, viruses on Windows 7, Vista and XP

In this blog post, the first in a series that show you how to use Beats for monitoring a Kafka cluster, we’ll focus on collecting and parsing Kafka logs by using Filebeat and Elasticsearch Ingest Node. After indexing the Kafka logs into Elasticsearch, we’ll finish this post by building Kibana dashboards for visualizing the data.

Related Post:  Dual-boot Windows 8 and Ubuntu 12.10 on UEFI hardware

This blog post is based on The Elastic Stack version 5.1.1. All configuration files, dashboards and sample log files can be found on GitHub.

Cluster Setup

Continue reading

Apache Kafka Stream Processor

Share:

Share on facebook
Facebook
Share on twitter
Twitter
Share on pinterest
Pinterest
Share on linkedin
LinkedIn

Newsletter: Subscribe for updates

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments

Get the latest

On social media

Security distros

Hacker
Linux distros for hacking and pentesting

Crypto mining OS

Bitcoin
Distros for mining bitcoin and other cryptocurrencies

Crypto hardware

MSI GeForce GTX 1070
Installing Nvidia GTX 1070 GPU drivers on Ubuntu

Disk guide

LVM
Beginner's guide to disks & disk partitions in Linux
Categories
Archives
0
Hya, what do you think? Please comment.x
()
x
Algorithm 2020

Did you get your ticket yet?

Algorithm 2022 is a 3-day conference on blockchain, cryptocurrencies and AI set for Feb. 10 – 12, 2022, in Dallas. Speakers from the US Air Force, Ministry of Digital Transformation, Ukraine, and more. click that button to learn more and get your ticket. Use BSD20 code for 20% off ticket price.