Upgrading to Broker 5.2.0

Upgrade instructions

Use the following upgrade instructions to upgrade Broker to 5.2.0 (Kafka 2.2.0).

the upgrade is a 2 step process per broker. First, the log.message.format.version is changed and, after a successful restart after the upgrade, the inter.broker.protocol.version can change. Please note that once the inter.broker.protocol.version has been changed, a rollback to a previous version is no longer possible. See also: release notes.

Step 1: verification

First, make sure you have checked out the latest version of Deploy (INSERT_VERSION_HERE). Secondly, update version numbers for the Broker image in your configuration, as follows:

LOG_MESSAGE_FORMAT_VERSION=1.1
INTER_BROKER_PROTOCOL_VERSION=2.0
BROKER_VERSION=5.2.0

Verify that the current broker version indeed is not updated yet (e.g. 0.11.0, 1.1.0), by checking the values of inter.broker.protocol.version and log.message.format.version and the library version:

$ docker logs broker | head -300 | grep inter.broker.protocol.version | head -n1
inter.broker.protocol.version=0.11.0

$ docker logs broker | head -300 | grep log.message.format.version | head -n1
log.message.format.version=0.11.0

$ docker exec broker ls /opt/kafka/libs | grep kafka_
kafka_2.11-1.1.0-javadoc.jar
kafka_2.11-1.1.0-scaladoc.jar
kafka_2.11-1.1.0-sources.jar
kafka_2.11-1.1.0-test-sources.jar
kafka_2.11-1.1.0-test.jar
kafka_2.11-1.1.0.jar

Step 2: upgrade

The following steps depend on whether you have a single or multi cluster setup. Follow the procedure accordingly.

Multi cluster procedure

For multi cluster setups, especially in active/active configurations, we recommend the following procedure (when starting the upgrade with cluster A).

  1. Log in to the cluster API node of cluster A

  2. Deactivate cluster A

$ ./axual.sh -v cluster set active false
  1. Verify that the cluster indeed is switched off by checking the discovery API using curl or your browser. It should not appear in the list of active clusters:

$ curl -sk https://DISCOVERY_API_URL:DISCOVERY_API_PORT/clusters/active | jq
[
  {
    "clusterApiUrl": "https://CLUSTER_1_API_HOST:CLUSTER_21_API_PORT"
  },
  {
    "clusterApiUrl": "https://CLUSTER_2_API_HOST:CLUSTER_2_API_PORT"
  }
]
  1. Use your monitoring tools to see when the load on the deactivated cluster has dropped significantly.

  2. Make sure the configuration on all broker nodes are referring to the new version of the broker image: axual/broker:5.2.0

  3. Do a rolling restart of all the nodes in the cluster, by logging in to the cluster nodes and repeating the following step per node:

$ ./axual.sh -v restart cluster broker
  1. Check if the version has been updated correctly.

$ docker logs broker | head -300 | grep inter.broker.protocol.version | head -n1
inter.broker.protocol.version=1.1

$ docker logs broker | head -300 | grep log.message.format.version | head -n1
log.message.format.version=1.1

$ docker exec broker ls /opt/kafka/libs | grep kafka_
kafka_2.12-2.2.0-javadoc.jar
kafka_2.12-2.2.0-scaladoc.jar
kafka_2.12-2.2.0-sources.jar
kafka_2.12-2.2.0-test-sources.jar
kafka_2.12-2.2.0-test.jar
kafka_2.12-2.2.0.jar

Also, monitor the broker log file to see if any errors occur during the upgrade.

  1. Upgrade the INTER_BROKER_PROTOCOL_VERSION to version 2.2 in your configuration (ATTENTION: Don’t update all cluster versions, leave the other clusters on 1.1 and ONLY set 2.2 on the currently updated cluster.)

  2. Restart the broker again

$ ./axual.sh -v restart cluster broker
  1. See that the version has been updated

$ docker logs broker | head -300 | grep inter.broker.protocol.version | head -n1
inter.broker.protocol.version=2.2
  1. Activate cluster A again

Repeat steps 1 to 11 for all clusters, until you are done.

Monitor the broker log file to see if any errors occur during the upgrade, and keep an eye on the cluster overview dashboard.

Single cluster procedure

When you are running in a single cluster setup, obviously you cannot switch off the entire cluster to do the upgrade. In that case, you can perform a rolling upgrade by repeating steps 5 to 10 above, for all the cluster nodes.