Troubleshooting Telegraf Sending Incorrect Data to InfluxDB 2: A Guide for New Users

Описание: Discover how to resolve issues with Telegraf sending incorrect metrics to InfluxDB 2 on your Ubuntu VPS setup, using straightforward solutions and custom scripts.
---
This video is based on the question https://stackoverflow.com/q/74710217/ asked by the user 'yannick' ( https://stackoverflow.com/u/20550221/ ) and on the answer https://stackoverflow.com/a/77392288/ provided by the user 'yannick' ( https://stackoverflow.com/u/20550221/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Telegraf is sending incorrect data to influxdb2?

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Troubleshooting Telegraf Sending Incorrect Data to InfluxDB 2: A Guide for New Users

If you've recently set up a monitoring system using Telegraf, InfluxDB 2, and Grafana on your cloud Ubuntu Virtual Private Server (VPS), you may have encountered an issue where the data you're seeing in InfluxDB does not align with what you expect from your VM. In this post, we will explore what could cause these discrepancies and how to resolve them efficiently.

Understanding the Problem

As noted by many users new to this technology stack, you might notice that the metrics reported by Telegraf do not match the real-time statistics of your system. For example, values like load averages, uptime, and memory usage can appear significantly skewed. Common examples include:

Load average might report figures as high as 21.39 while top shows 0.07.

Uptime may be reported as 44 weeks instead of the actual hours your system has been running.

Memory used could be displayed as 87GB while your VM only has 3GB.

The primary reason for these anomalies often lies in how Telegraf interacts with the hypervisor in your VPS environment, leading to it capturing metrics that are not representative of your actual VM.

Solution Overview

To resolve this issue, a straightforward workaround is to create custom metrics using shell scripts, which can be scheduled using cron to send data to Telegraf via the StatsD input. This way, you'll collect accurate metrics that reflect your actual VM performance. Below, we detail out the steps involved in implementing this solution.

Implementing a Custom Metrics Collection Method

1. Write Your Custom Script

Create a shell script that gathers the metrics you want and sends them to Telegraf using StatsD. Below is a basic structure for your script:

[[See Video to Reveal this Text or Code Snippet]]

In this script:

The SendToStatsd function is responsible for sending data to the StatsD server.

We gather the uptime directly from the /proc/uptime file and total swap memory using the free command.

2. Schedule the Script with Cron

Once your script is ready, you’ll want to run it at regular intervals. You can do this using cron jobs. To edit your crontab file, run:

[[See Video to Reveal this Text or Code Snippet]]

Then add an entry that schedules your script. For instance, to run your script every minute, add the following line:

[[See Video to Reveal this Text or Code Snippet]]

3. Adjust Your Telegraf Configuration

Ensure that your Telegraf configuration is set up to listen for StatsD metrics. You can do this by editing your telegraf.conf file. The relevant section should look like this:

[[See Video to Reveal this Text or Code Snippet]]

This configuration allows Telegraf to receive the metrics you're sending from your script.

Conclusion

By harnessing custom scripts in combination with StatsD, you can effectively bypass the challenges posed by hypervisor metrics and ensure that your monitoring setup reflects the true state of your Ubuntu VPS. While it can be daunting at first, following these straightforward steps will help you regain control over your metrics and allow you to utilize the full potential of your monitoring stack.

By addressing these problems with the right approach, you can make the most out of your Telegraf and InfluxDB setup, ensuring integrity in the data you collect and visualize.

Feel free to reach out in the comments if you have questions or share your experiences in setting up a similar monitoring solution!