How To Improve Working With Application Logs using Kibana and Elasticsearch

Published in

Towards Dev

10 min readMar 1, 2024

Today I want to continue the article that I wrote about monitoring application metrics on Grafana with the help of Micrometer and Prometheus (see How to Monitor Your Application Metrics With Prometheus and Grafana). Monitoring is just one important part of nowadays application development, another one is logging.

Application logs are essential for maintaining, troubleshooting, securing, and optimizing modern software applications. They provide valuable insights that help to ensure software systems' reliability, performance, and security. So it is time to shed light on this topic too. Elasticsearch and Kibana are two tools that can support me in this way.

Before starting with the implementation part let’s first look at why application logs play such an important role in developing applications.

Importance of Logging

What are the key reasons for keeping logs in focus?

Troubleshooting and Debugging: Logs provide valuable information for diagnosing and fixing issues within an application. When something goes wrong, logs can reveal the sequence of events leading up to the problem, error messages, and other relevant details that help me as a developer to understand the root cause of the issue.
Monitoring and Performance Optimization: By analyzing logs, developers like me but also system administrators can monitor the health and performance of an application. It is possible to identify performance bottlenecks, monitor resource usage, and optimize the application for better efficiency.
Security: Logs play a vital role in detecting and investigating security incidents. They can capture suspicious activities, unauthorized access attempts, and other security-related events. Analyzing logs can help in identifying security threats, understanding the extent of a breach, and implementing measures to prevent future incidents.
Compliance and Auditing: Many industries have regulations and compliance standards that require organizations to maintain detailed logs of their systems and applications. These logs serve as a record of activities, helping organizations demonstrate compliance with regulations and facilitating audits by regulatory bodies.
Historical Analysis and Trend Identification: Logs contain historical data about the behavior of an application over time. By analyzing this data, I can identify patterns, trends, and anomalies that can inform decision-making, such as feature enhancements, capacity planning, or infrastructure changes.
Business Intelligence: Besides technical insights, application logs can also provide valuable business intelligence. They can capture user interactions, customer behavior, and usage patterns, which can be analyzed to make data-driven decisions, improve user experience, and drive business growth.

Now that there is a basic understanding of why logging is an important part of every application that goes to production, it is time to have a look at how this can work in practice. I don’t want to talk about storing logs locally in the environment, where the application is running and manually copying them to my local system for analysis but a centralized and automated approach. So let’s start…

Implementation

The application I use as the starting point for this article is the one used for the previous monitoring article. You can find a link to the repository at the end of this article.

Because the example application contains no custom logging, except for the startup process, I first add some log statements to the different components. To also increase the amount of logs (which is not necessarily what you want in production) I set the default log level to DEBUG in the application.properties file.

quarkus.log.level=DEBUG

The last thing that I change is to add a custom error handler for the NotFoundException. With this, I can add an error log message that is printed to the log.

@Provider
class NotFoundErrorHandler : ExceptionMapper<NotFoundException> {
    private val logger: Logger = LoggerFactory.getLogger(NotFoundErrorHandler::class.java)

    override fun toResponse(exception: NotFoundException): Response {
        logger.error("Request failed with exception.", exception)
        val originalErrorResponse: Response = exception.response
        return Response.fromResponse(originalErrorResponse)
            .entity(exception.message)
            .build()

    }
}

Now that the application is prepared with log content, that is produced during runtime, I can start by adding the first brick to manage the logs centrally - Elasticsearch

Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine built on top of Apache Lucene. It’s designed for horizontal scalability, meaning you can easily scale it up to handle large amounts of data across multiple nodes in a cluster.

Here are some key features and concepts of Elasticsearch:

Document-Oriented: Data is stored as JSON documents, which are organized into indices. Each document is a collection of fields with corresponding values.
Index: An index is a collection of documents that have similar characteristics. For example, you might have an index for storing logs from your application and another index for storing product information.
Node: A single instance of Elasticsearch running on a server is called a node. Multiple nodes can form a cluster, allowing for distributed data storage and query processing.
Cluster: A cluster is a collection of nodes that work together to store your data and provide search capabilities. Elasticsearch automatically manages cluster state and ensures data is distributed across nodes for fault tolerance and scalability.
Sharding: Elasticsearch divides each index into multiple shards, which are distributed across nodes in the cluster. Sharding allows for horizontal scalability and improves search performance by distributing the workload.
Replication: Elasticsearch provides data redundancy and high availability through index replication. Each shard can have one or more replicas, which are copies of the primary shard. Replicas provide failover in case a node goes down and improve read performance by distributing search requests across multiple nodes.
RESTful API: Elasticsearch exposes a RESTful API for interacting with the cluster. You can perform various operations such as indexing documents, searching, aggregations, and cluster management using simple HTTP requests.
Query DSL: Elasticsearch uses a powerful query language called Query DSL (Domain-Specific Language) for searching and filtering documents. It supports complex queries, aggregations, and filtering based on criteria such as text relevance, geolocation, and date ranges.

Elasticsearch can be added to the application by adding the following section to the existing docker-compose.yml. The current latest version of the Elasticsearch docker image at the time of writing this article is 8.12.2:

  elasticsearch:
    image: 'elasticsearch:8.12.2'
    container_name: elasticsearch
    ports:
      - '9200:9200'
      - '9300:9300'
    volumes:
      - 'elastic_data:/usr/share/elasticsearch/data/'
      - ./logging/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./logging/sysctl.conf:/etc/sysctl.conf
    environment:
      ELASTIC_PASSWORD: "password1234"


volumes:
  elastic_data:

Let’s have a quick look at the relevant parts:

I define a volume that will persist the data that is collected from the application.
I define an elasticsearch.yml file that is copied to the container at startup. There are a lot of configuration properties available (see settings) that can be set depending on the requirements. In the example case I only specify the below one:

cluster.name: "elasticsearch"
network.host: localhost
discovery.type: single-node
http.host: 0.0.0.0
transport.host: 0.0.0.0

Because of some memory problems that I have when running Elasticsearch on a single node, I specify the vm.max_map_count property that is copied to the /etc/sysctl.conf file.
I also set a default password for accessing Elasticsearch.
The last thing that I configure is a health check that is used by the other services running to wait until Elasticsearch is ready to accept requests, before starting themselves.

This is everything necessary for starting Elasticsearch successfully. Until now Elasticsearch is not connected to the application but running in isolation besides it.

Let’s continue with the next brick - Kibana.

Kibana

Kibana is a powerful tool used for visualizing and analyzing data stored in Elasticsearch. Kibana serves as the frontend interface for Elasticsearch, providing users with a user-friendly environment to interact with their data.

Here’s a more detailed breakdown of Kibana’s key features and functionalities:

Data Exploration: Kibana allows users to explore large volumes of data by querying Elasticsearch indices using a simple search interface. Users can search, filter, and aggregate data to identify patterns, trends, and outliers.
Visualization: Kibana provides a wide range of visualization options, including line charts, bar charts, pie charts, histograms, maps, and more. I can create custom visualizations to represent their data in a meaningful way, making it easier to understand and analyze complex datasets.
Dashboarding: I can combine multiple visualizations into interactive dashboards, allowing me to monitor key metrics and KPIs in real-time. Dashboards can be customized and shared with other team members, enabling collaborative data analysis and decision-making.
Time Series Analysis: Kibana is particularly well-suited for analyzing time-series data, such as log files, sensor data, and metrics. It offers powerful time-based aggregations and filtering options, making it easy to perform tasks like monitoring system performance, detecting anomalies, and troubleshooting issues.
Elasticsearch Integration: Kibana seamlessly integrates with Elasticsearch, leveraging its powerful search and indexing capabilities. I can create index patterns to define the data they want to visualize and analyze, and Kibana automatically retrieves data from Elasticsearch in response to user queries and requests.
Plugin Ecosystem: Kibana supports a vibrant ecosystem of plugins and extensions, allowing me to extend its functionality and integrate with other systems and services. This flexibility makes it possible to customize Kibana to suit specific use cases and workflows.

I can use Kibana in my application development in the same way as Elasticsearch. I add a new service configuration to the docker-compose.yml file:

  kibana:
    image: 'kibana:8.12.2'
    container_name: kibana
    ports:
      - '5601:5601'
    volumes:
      - './logging/kibana.yml:/usr/share/kibana/config/kibana.yml'
      - 'kibana_data:/usr/share/kibana/data'
    environment:
      ELASTICSEARCH_SERVICEACCOUNTTOKEN: "AAEAAWVsYXN0aWMva2liYW5hL3Rva2VuMTpkeEplSzVmRFN6NmUwWkRNdWFYUEh3"
    depends_on:
      - elasticsearch

volumes:
  kibana_data:

Let’s again have a look at the relevant parts:

There is its volume for the persistence of the data configured like the dashboards.
Also, there is a kibana.yml file defined with custom settings that is copied to the docker container. This is e.g. the place to configure the connection to the Elasticsearch instance.
The start of the Kibana instance depends on Elasticsearch because it consumes its data.

server.name: kibana
server.host: "0.0.0.0"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true

For accessing Elasticsearch I use a predefined service account that is available for Kibana (you can have a look at the details in the documentation). To get the access token that is necessary for the connection I run the Elasticsearch container and execute the below curl command:

curl -X POST -u elastic:password1234 "localhost:9200/_security/service/elastic/kibana/credential/token/token1?pretty"

The result is a JSON response with the token embedded like the below example:

{
  "created" : true,
  "token" : {
    "name" : "token1",
    "value" : "AAEAAWVsYXN0aWMva2liYW5hL3Rva2VuMTpkeEplSzVmRFN6NmUwWkRNdWFYUEh3"
  }
}

With this Kibana is ready for visualization of my application logs. So it is time for the last part. The logs of the sample application need to reach Kibana.

Pushing Application Logs

For bringing the application logs to Elasticsearch and Kibana there are multiple solutions available including the below options:

Logstash: A powerful open-source log ingestion and processing tool that can collect, parse, and transform logs before sending them to Elasticsearch. It is the “L” part of the ELK stack.
Filebeat: A lightweight log shipper designed to tail log files and forward them directly to Elasticsearch or Logstash.
Fluentd: A flexible log collector and aggregator that supports multiple input sources and can output logs to Elasticsearch among other destinations.

I choose a different option. For Quarkus a project exists that directly pushes my logs to Elasticsearch. You can find the project on Github: https://github.com/codinux-gmbh/quarkus-elasticsearch-logger

I just need to add the below dependency to the build.gradle.kts file.

implementation("net.codinux.log:quarkus-elasticsearch-logger:2.5.0")

The complete configuration can be done in the application.properties of the application:

quarkus.log.elasticsearch.host=http://elasticsearch:9200
quarkus.log.elasticsearch.username=elastic
quarkus.log.elasticsearch.password=password1234
quarkus.log.elasticsearch.index=log-%date{yyyy.MM.dd}

The minimal configuration contains the connection details, credentials for Elasticsearch, and the naming pattern of the index. There are a lot of other configurations possible but this is not the scope of this article.

That’s all that is necessary for preparation. In the next part, I will show the result when bringing all the configured tools together and visualize the result of the application logs.

Visualization

All the configured services can be started together by a simple “docker-compose up” command.

As soon as all services start successfully I can browse http://localhost:5601/ to reach Kibana.

The first step is to navigate to the Data Views menu inside the Stack Management main menu.

I add a new data view with the below configuration.

After saving the data view I can switch to the Discover main menu and already can find my application logs.

That is just the beginning. This is just a raw overview of the logs that are created by the application. The next step is to create a dashboard that visualizes the information in a more expressive format.

A dashboard can be added by going to the Dashboards menu and clicking Create dashboard.

There are a bunch of visualizations available that can be added simply by dragging and dropping the relevant elements together.

You can find a good starting point about what is possible in the official documentation of Kibana.

A final dashboard may look like below (just an example of what is possible):

The visualization is just one part of what is offered by Kibana. There are other use cases too:

Searching for information. I can use the Query DSL to search for specific information in the logs, like errors with a specific error code or exception.

Add alerting with conditions based on log information like all logs with level ERROR or log messages with a specific keyword. It is possible to write complex conditions that can be combined. As soon as the error condition is met a notification can be sent e.g. to Slack. For alerting an additional configuration is necessary (see Alerting).

Conclusion

In today’s article, I looked at how the logs of my Quarkus application can be managed centrally and visualized. The tools that I used for this job are Elasticsearch and Kibana. Both are standard meanwhile when it comes to dealing with application logs.

The setup and integration of both tools are very straightforward and the main work is to configure the docker container in my docker-compose file. Because there is an integration for Quarkus that is pushing the application logs to Elasticsearch it is not necessary to configure an additional container e.g. Logstash.

Summing it up. The time of manually dealing with application logs is over. The setup shown in this article can be used as a starting point for experiencing the power of centralizing log handling with Elasticsearch and Kibana.

You can find the code used for this article on Github: https://github.com/PoisonedYouth/quarkus-playground