How to Convert Kafka Streams Output to JSON Format for Elasticsearch Integration

Описание: Learn how to modify your Kafka Streams application to convert output data into `JSON` format suitable for Elasticsearch integration.
---
This video is based on the question https://stackoverflow.com/q/64656730/ asked by the user 'Suprit' ( https://stackoverflow.com/u/14399952/ ) and on the answer https://stackoverflow.com/a/64672295/ provided by the user 'OneCricketeer' ( https://stackoverflow.com/u/2308683/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: modification in streams app regarding data format conversion to json for a kafka pipeline

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Transforming Output Data in Kafka Streams to JSON Format for Elasticsearch

In the landscape of data processing and streaming applications, the ability to format output appropriately for downstream systems is essential. A common scenario faced by developers working with Apache Kafka is the requirement to output data in the JSON format that is needed for tools like Elasticsearch. This post will discuss a specific use case where we need to convert the output of a Kafka Streams application into the required JSON format.

Understanding the Problem

Imagine you have a Kafka Streams application that is performing a word count operation on sentences published to an input topic. After processing, you notice that the output you receive in the consumer console appears as:

[[See Video to Reveal this Text or Code Snippet]]

This output format is not suitable for sending to Elasticsearch, which specifically requires data in JSON format. The desired output format should look like this:

[[See Video to Reveal this Text or Code Snippet]]

Your goal is to modify the Kafka Streams application so that, post-processing, the output aligns with this JSON structure. Let's explore how to achieve that.

Steps to Convert Output to JSON Format

To create the desired JSON output in your Kafka Streams application, you need to make a couple of changes to your existing code. Here's a breakdown of what you need to do:

1. Mapping the Output to JSON Structure

Before the current output is sent to the topic, you will need to map your key-value pairs to the expected JSON format. You can achieve this by using the .map() function provided by Kafka Streams.

2. Modifying the Produced Serde

You also need to change the serializer used in the Produced.with() method. Instead of using Serdes.Long(), which is the current setting for the count, you will use a Serdes.String() serializer since you are now sending JSON formatted strings.

Example Code Modification

Here’s how you can modify your existing streams application code:

[[See Video to Reveal this Text or Code Snippet]]

Explanation of the Changes

Mapping to JSON: The .map() function creates a new key-value pair where the value is formatted as SQL for easy conversion. The String.format() method constructs the required JSON string with the word and count.

Updated Serde: By specifying Produced.with(Serdes.String(), Serdes.String()), the application prepares to send the JSON strings to downstream consumers, such as Elasticsearch.

Conclusion

Converting your Kafka Streams application's output to JSON format may seem daunting at first, but with the step-by-step adjustments highlighted above, you can achieve it seamlessly. These changes ensure that the output is compatible with Elasticsearch, thus enabling smooth integration.

By implementing the mapping operation and using the correct serializer, your Kafka Stream application can now produce consumer-ready JSON output. Happy streaming!