Motivation

In a previous blog post, I wrote about monitoring Corda nodes with Prometheus, Grafana and ELK (Elasticsearch, Logstash and Kibana) on Docker. In this post, I will show you a how-to monitor your Corda nodes with Grafana Dashboard as the only user interface for metrics and log files.

The ELK stack is removed in favour of Promtail and Grafana Loki for this deployment demonstration.

The architecture deployed provides a clean method of managing your nodes’ operational needs for both JMX metrics and visualised log files. Of course, the “archaic” way would be to tail and grep log files to find what you need. This method becomes troublesome when, for example, you are trying to debug a Corda transaction between multiple nodes.

The end result is the ability to use Grafana Dashboard explorer to provide a split-screen live view of log files for nodes:

And of course, JMX metrics dashboard is included:

Deployment Stack

A few amendments to the stack are required from my previous post, take a look at the diagram below:

• Corda – Corda Open-Source 4.7 will be used in this deployment (latest version at the time of writing this post).
• PostgreSQL databases for each node — I have added this to show how easy it is to switch away from H2.
• Promtail logfile collector — extracts the log files from locations specified in its configuration.
• Grafana Loki — aggregates and parses logfiles.
• Prometheus — an opensource monitoring solution which will be used to parse JMX metrics.
• Grafana Monitoring Dashboard — provides a configuration user interface, collating multiple datasources.

Github

Check out the Github Repository where you will find step-by-step instructions in the README and some handy shell scripts.

Clone the repo:

➜ git clone https://github.com/neal-shah/corda-monitoring-prometheus-grafana-loki

Step by step instructions

Step 1: Prepare the workspace directory

Create the directory structure and download the necessary in this step. 3 jars are required:

• corda-tools-network-bootstrapper-4.7.jar
• corda-finance-contracts-4.7.jar
• corda-finance-workflows-4.7.jar

The two corda-finance-*.jar‘s make up the Corda Finance CorDapp which will be used to test transactions across peer nodes.

Execute the 01_setup-directory.sh shell script:

You should see a new directory called mynetwork created, with a few sub directories and the required jars.

Step 2: Create node configuration files

You will require 3 node configurations:

Execute the 02_create-node-configurations.sh shell script:

Our mynetwork directory now looks like the following:

This will create three .conf files, each representing a single node.

Here’s an example of the partya_node.conf file:

Step 3: Run the Corda Network Bootstrapper

The Corda Network Bootstrapper creates a development network of peer nodes, using dev certificates. You don’t need to worry about registering nodes — the bootstrapper takes care of that for you.

Execute the 03_run-corda-network-bootstrapper.sh shell script:

Bootstrapping complete!

Step 4: Prepare for Docker

There are some common files that are shared between the peer nodes. You can put these in one folder — this will make your Docker-Compose service volumes a bit clearer to read.

Execute the 04_copy-common-files.sh shell script:

This will copy across common files to the ./mynetwork/shared folder.

Step 5: Create the Prometheus configuration files

Execute the 05_create-monitoring-configurations.sh shell script:

This creates a config file in ./mynetwork/prometheus/prometheus.yml:

Check out the ./mynetwork/promtail/ and ./mynetwork/loki/ directories for their respective configuration files.

You define the JMX exporter targets (endpoints) for each node. They are all using port 8080 — don’t worry about port conflicts, Docker will take care of the networking.

Step 6: Create the Docker-Compose file

Finally, you need a docker-compose.yml file which allows you to bring up all the services in just one command.

Execute the 06_create-docker-compose-file.sh shell script:

You can find the docker-compose.yml file in ./mynetwork/docker-compose.yml. Inside the file, you have created services for each node database, along with Prometheus, Grafana, Promtail and Loki:

Start up the services using the following command:

View running containers:

Step 7: Set up Grafana

On your browser, go to http://localhost:3000.

Grafana will ask you for a login — the default admin login is:

• Username: admin
• Password: admin

Change your password as instructed, and then you should see the Grafana homepage.

Click on Add data source.

Step 7.1: Add JMX Metrics Dashboard

Select the Prometheus data source under Time series databases.

Under HTTP, set the URL to http://prometheus:9090. You can use the Prometheus Docker container hostname here as all of the containers run on the same Docker bridge network, so no explicit container IP addresses need to be used for connectivity.

At the bottom of the page, click on Save & Test. You should see a green alert – Data source is working.

Hover over the Dashboards icon, and click Manage.

Click Import, then Upload .json file, and navigate to the clone repository folder. Inside the grafana folder, you will see a json file – Grafana-Corda-Dashboard.json, see here.

On the following screen, click Import.

Boom ?, a dashboard appears!

Step 7.2: Add Loki Data Source

Add the Loki data source under Logging & document databases.

Under HTTP, set the URL to http://loki:9090.

At the bottom of the page, click on Save & Test. You should see a green alert – Data source is working.

Step 8: Explore Grafana Loki

In the Grafana side pane, click on Explore.

At the top you will see a dropdown with data sources you can select — select the Loki data source.

Click Split on the top right menu pane.

On the left-hand side Loki explorer, click on the Log labels dropdown, select node, then select partya. Similarly, on the right-hand side Loki explorer, select partyb.

On both explorers top right menu, click the Live stream your logs button.

You now have both partya and partyb logs live streaming in one UI!

Step 9: Run some Corda Finance flows

On a terminal window, SSH into the PartyA node Crash shell:

When prompted, the password is password.

You should see the following in your terminal:

Let’s execute a CashIssueAndPaymentFlow:

In your Loki explorer split screen, you will see that both partya and partyb will now show you the latest logs of the transaction taking place.

You can use Loki as a data source in your dashboards, adding variables as appropriate and provide search functionality for your users too. Advanced configuration can be handled where multiple data sources can be related by labels or annotations — this provides a wholesome approach to fault finding and issue diagnosis.

Further reading

JMX Monitoring and Prometheus

Corda nodes expose JMX metrics that can be collected to provide data analysis and monitoring functionality.

This is enabled using 2 components:

• On the node side, the Prometheus JMX Exporter is run as a JVM plugin and provides an API interface for the JMX metrics.
• The Prometheus Server collects these metrics using a pre-defined scrape interval, providing both a UI and an API for processed data to be consumed.

To run Corda with the Prometheus JMX Exporter, add the driver to the drivers folder in the node directory, and run the corda.jar with the -javagent option. For example:

The Prometheus JMX Exporter config.yml as specified above can contain config options as specified here – you can simply make this file contain {} to use default options.

The Prometheus Server also requires a configuration file, prometheus.yml, containing a number of config options such as setting target exporters, alerting and rules.

Here is a simple example of a prometheus.yml file targeting a single Corda node which is exporting JMX metrics:

Under the global block, you can set global attributes to all the scrape_configs. Here, you set the scrape_interval to get metrics every 10 seconds. scrape_configs tells Prometheus what services to target and scrape metrics from.

relabel_configs block in the config above allows you to rewrite the target labels – for example, partya:8080 as the instance label will become partya. This just makes things a bit neater when displaying information on a dashboard.

Grafana

Grafana provides a slick UI interface for consuming and aggregating realtime data from our exposed JMX metrics. You can indeed use Graphite rather than Prometheus — there are a number of Graphite vs. Prometheus articles online to determine your choice.

Originally published at https://www.nealshah.dev.

Want to learn more about building awesome blockchain applications on Corda? Be sure to visit https://corda.net, check out our community page to learn how to connect with other Corda developers, and sign up for one of our newsletters for the latest updates.

— Neal Shah is a Solutions Engineer at R3, an enterprise blockchain software firm working with a global ecosystem of more than 350 participants across multiple industries from both the private and public sectors to develop on Corda, its open-source blockchain platform, and Corda Enterprise, a commercial version of Corda for enterprise usage.

Follow Neal on LinkedIn and Instagram.