Step 7: Enabling monitoring

Monitoring your streaming applications

Now that your app is running fine and you know the basics of creating your streaming use case, you can take an additional step which is essential if you decide to take your use case to production.

Enabling monitoring in your consuming or producing applications can be done in various ways. In this Getting Started we will cover one method, by exposing JMX in your application.

Exposing JMX for monitoring purposes is only possible in JVM based applications

Enable monitoring for your Consumer / Producer appplication

Follow the steps below to enable monitoring for your Consumer / Producer application

  1. Enable JMX for your application by adding the following JVM parameters where 'jmxremote.port' is the port you will connect to with a JMX browser application
  2. If you are running your app in a docker container, make sure you expose port 1100, so you can connect to it from the outside

  3. Use a tool like JConsole (bundled with your JDK) or VisualVM to connect to your Consumer/Producer process, on port 9000

  4. In the example screenshot below, you can see some producer metrics exposed for an app called "accountentryservice"

    Browsing Kafka Producer JMX metrics using JConsole
  5. Now that you know that the metrics are exposed correctly, you can decide to use a monitoring tool to (continously) scrape those metrics and perhaps even add some alerting to it

Exposing JMX does not only give you nice metrics, it also allows to control processes within your JVM. If you decide to expose JMX, only do this locally or make sure you protect your JMX interface. We suggest you expose your metrics using a JMX Exporter (see the information below), and not exposing the JMX port altogether.

Gathering metrics using Prometheus

Using a tool like JConsole or VisualVM is fine for the use case above, verifying that JMX works just fine, but for an application in production this certainly is not the best approach. In any environment but local, you would like to have a continous and central monitoring approach for your streaming applications. Tools like Prometheus or Grafana come in very handy here. For the platform components, we have added extensive Grafana dashboards, but for a client, you need to do this yourself.

Prometheus scrapes endpoints to collect the metrics and stores its metrics in a time series database. It scrapes metrics over http, so you need a little trick to make your app expose metrics over http, as opposed to JMX. This trick is called JMX Exporter.

For detailed instruction on how to set up JMX Exporter for your application, please refer to this guide.

Monitoring Axual

For more information on the Axual monitoring and alerting approach, click here.