Native High Availability (Native HA) in Docker: Cloud-Native Resilience

Historically, High Availability (HA) in IBM MQ depended on shared storage or external replication (RDQM). In modern microservices and cloud architectures (Kubernetes, OpenShift, Docker), shared storage introduces complexity and latency.

Native HA uses a consensus algorithm (Raft) to replicate data between three independent instances. This lab demonstrates how to stand up a resilient 3-node cluster using Docker Compose, utilizing configuration files (qm.ini) mounted via volumes to ensure 100% reliability in version 9.4+. Additionally, we will demonstrate Automatic Client Reconnect (ACR) using a fourth node as a client.

How does MQ Native HA work?

MQ Native HA was designed to eliminate dependency on shared storage technologies (such as NFS or SAN), which are often complex and slow in cloud environments.

1. The Raft Protocol and Quorum

Native HA is based on the Raft consensus algorithm. In a group of three instances, nodes communicate continuously to elect a Leader (Active). The other two instances become Replicas.

Quorum: For the cluster to accept messages, a majority of nodes (2 out of 3) must be functional and in communication. This prevents "Split-brain", where two nodes could attempt to be active simultaneously, corrupting data.
Automatic Election: If the Leader fails, the Replicas detect the absence of "heartbeats" and initiate a new election. The node with the most up-to-date recovery log is elected as the new Leader.

2. Synchronous Replication

Unlike other solutions, MQ Native HA replicates data at the recovery log level. When an application sends a persistent message: 1. The Leader writes the message to its local log. 2. The Leader sends the log data to the Replicas. 3. Once at least one Replica confirms receipt (guaranteeing the majority), the Leader confirms success to the application.

3. Replica Limitations and Behavior

It is essential to understand that Replicas are not "active read nodes." There are strict operational limitations:

Standby State: Replicas are in a "hot standby" state. They do not process messages or allow applications to connect to them (via channels or locally).
No Application Connectivity: If you try to connect an application to a Replica, you will receive a "Queue Manager Not Available" error (MQRC_Q_MGR_NOT_AVAILABLE).
Limited Administration: Commands like runmqsc have very restricted functionality on Replicas, serving only to query the HA status.

4. Automatic Client Reconnect (ACR)

Automatic Client Reconnect (ACR) is a feature of the IBM MQ client libraries that enables transparent application-side resilience.

How it works:

Transparency: If the connection is interrupted, the MQ client suspends the application operation instead of returning an immediate error.
Automatic Retry: The MQ client consults the configured list of connection names (CONNAME) and attempts to re-establish the session with the other nodes.
Failover in Native HA: In a Native HA cluster, the client will attempt to connect to the Replicas (which will reject the connection) until it finds the new Leader (Active). Once connected, processing resumes automatically, preserving message integrity.

What happens technically in the MQI client:

Reconnect-eligible connection: The client must use client transport and an alternative address list, typically via CONNAME, CCDT, MQSERVER, or mqclient.ini.
Inline reconnect: When the failure is detected, the client attempts to re-establish the connection without forcing the application to execute a new MQCONN or MQCONNX.
Context restoration: If the reconnection succeeds, the connection and object handles are recreated by the client. For many MQI applications, this means processing resumes with minimal impact.
Longer wait during MQI call: If the failure occurs mid-call, the application may observe only an extended wait until the operation completes or fails definitively.
Not all APIs behave the same way: In this lab we will use amqsphac, an IBM MQ sample designed specifically to demonstrate automatic reconnection in high-availability scenarios. The sample uses MQCNO_RECONNECT in MQCONNX and keeps the application loop active during reconnection — which generic samples like amqsputc do not do. MQI-style ACR is not supported by IBM MQ classes for Java; JMS has its own automatic reconnection mechanism.

Configuration Anatomy (`qm.ini`)

The logical Native HA configuration resides in the NativeHALocalInstance and NativeHAInstance stanzas, which are part of the queue manager configuration. In conventional Linux installations, this means editing each instance's qm.ini.

In this container-based lab, we keep those stanzas in dedicated files node1.ini, node2.ini, and node3.ini, mounted as /etc/mqm/nativeha.ini. The important part is the stanza content, not the host filename. The container image consumes that mounted file to initialize the Native HA configuration at startup.

The configuration is divided into two main sections:

NativeHALocalInstance

Defines the identity of the current node.

Name: The unique name of this instance (e.g., node1).

NativeHAInstance

Defines all members participating in the HA group. There must be an entry for each of the 3 nodes.

Name: Unique name of the instance.
ReplicationAddress: The IP address or Hostname and port dedicated exclusively to Raft replication traffic.

What this lab builds

By the end of this lab, you will have:

A cluster of 3 MQ containers communicating via Docker network.
File-based configuration with stanzas equivalent to qm.ini, simulating enterprise deployments.
A dedicated Client node demonstrating automatic reconnection (ACR).
Synchronous log replication without the need for shared storage.

Environment Preparation: Software Installation

Before starting the lab, ensure you have the necessary containerization tools and access to the official MQ image.

You also need one of these two conditions on the Linux host:

access to sudo
or membership in the docker group, to run Docker without sudo

If you have neither of these options, you will not be able to execute this lab as written.

1. Install Docker and Docker Compose

Docker is the base platform for this lab. Docker Compose allows easy orchestration of the 3 MQ nodes.

Windows/macOS: Install Docker Desktop. Docker Compose is already included.
Linux (Ubuntu/Debian):

  # Ensure curl is installed
  sudo apt-get update && sudo apt-get install -y curl
  # Install Docker
  curl -fsSL https://get.docker.com -o get-docker.sh
  sudo sh get-docker.sh
  # Install Docker Compose
  sudo apt-get install -y docker-compose-plugin

Linux (RHEL/CentOS/Fedora):

  # Remove old versions
  sudo dnf remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine
  # Setup the repository
  sudo dnf -y install dnf-plugins-core
  sudo dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
  # Install Docker and Compose
  sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  # Start the service
  sudo systemctl enable --now docker

Confirm the installation:

docker version
docker compose version

Permission Configuration (Linux)

If you receive a "permission denied" error when running Docker commands, your user needs to be added to the docker group:

sudo usermod -aG docker $USER

Note: You must run sudo newgrp docker or logout/login to apply the changes.

If you prefer not to change group membership right away, you can simply run the commands in this lab with sudo, as they already appear in the host-side command blocks.

Confirm before proceeding:

docker version

If this command fails and you also cannot obtain docker group access, stop here and resolve access to the Docker daemon first.

2. Obtain the IBM MQ Image

Simply run the command:

docker pull icr.io/ibm-messaging/mq:latest

1. Create the Configuration Files

MQ Nodes (`.ini`)

We will create three distinct configuration files for each node.

Create the file for node1:

cat > node1.ini <<'EOF'
NativeHALocalInstance:
    Name=node1
NativeHAInstance:
    Name=node1
    ReplicationAddress=node1(5000)
NativeHAInstance:
    Name=node2
    ReplicationAddress=node2(5000)
NativeHAInstance:
    Name=node3
    ReplicationAddress=node3(5000)
EOF

Create the file for node2:

cat > node2.ini <<'EOF'
NativeHALocalInstance:
    Name=node2
NativeHAInstance:
    Name=node1
    ReplicationAddress=node1(5000)
NativeHAInstance:
    Name=node2
    ReplicationAddress=node2(5000)
NativeHAInstance:
    Name=node3
    ReplicationAddress=node3(5000)
EOF

Create the file for node3:

cat > node3.ini <<'EOF'
NativeHALocalInstance:
    Name=node3
NativeHAInstance:
    Name=node1
    ReplicationAddress=node1(5000)
NativeHAInstance:
    Name=node2
    ReplicationAddress=node2(5000)
NativeHAInstance:
    Name=node3
    ReplicationAddress=node3(5000)
EOF

MQ Configuration File (`config.mqsc`)

Create the file with all the security and queue configuration. The IBM MQ Docker image automatically runs .mqsc files mounted in /etc/mqm/ at active node startup — replicas receive the same configuration via Raft replication:

cat > config.mqsc <<'EOF'
DEFINE CHANNEL(DEV.APP.SVRCONN) CHLTYPE(SVRCONN) REPLACE
ALTER CHANNEL(DEV.APP.SVRCONN) CHLTYPE(SVRCONN) MCAUSER('mqm')
SET CHLAUTH(DEV.APP.SVRCONN) TYPE(BLOCKUSER) USERLIST('nobody') ACTION(REPLACE)
ALTER QMGR CHLAUTH(DISABLED)
ALTER QMGR CONNAUTH('')
REFRESH SECURITY(*) TYPE(CONNAUTH)
DEFINE QLOCAL(HA.TEST.QUEUE) DEFPSIST(YES) REPLACE
EOF

2. The Orchestration File (Cluster + Client)

Recreate the file:

rm -f docker-compose.yaml

Then open the file:

vi docker-compose.yaml

And paste exactly this content:

services:
  mq-node1:
    container_name: mq-node1
    hostname: node1
    image: icr.io/ibm-messaging/mq:latest
    environment:
      LICENSE: "accept"
      MQ_QMGR_NAME: "QM_HA"
      MQ_NATIVE_HA: "true"
      MQ_NATIVE_HA_INSTANCE_NAME: "node1"
    volumes:
      - ./node1.ini:/etc/mqm/nativeha.ini
      - ./config.mqsc:/etc/mqm/20-config.mqsc
    ports:
      - "1414:1414"

  mq-node2:
    container_name: mq-node2
    hostname: node2
    image: icr.io/ibm-messaging/mq:latest
    environment:
      LICENSE: "accept"
      MQ_QMGR_NAME: "QM_HA"
      MQ_NATIVE_HA: "true"
      MQ_NATIVE_HA_INSTANCE_NAME: "node2"
    volumes:
      - ./node2.ini:/etc/mqm/nativeha.ini
      - ./config.mqsc:/etc/mqm/20-config.mqsc
    ports:
      - "1415:1414"

  mq-node3:
    container_name: mq-node3
    hostname: node3
    image: icr.io/ibm-messaging/mq:latest
    environment:
      LICENSE: "accept"
      MQ_QMGR_NAME: "QM_HA"
      MQ_NATIVE_HA: "true"
      MQ_NATIVE_HA_INSTANCE_NAME: "node3"
    volumes:
      - ./node3.ini:/etc/mqm/nativeha.ini
      - ./config.mqsc:/etc/mqm/20-config.mqsc
    ports:
      - "1416:1414"

  mq-client:
    container_name: mq-client
    image: icr.io/ibm-messaging/mq:latest
    environment:
      LICENSE: "accept"
    command: sleep infinity

3. Start the Cluster

Start the containers:

docker compose down -v
docker compose up -d

Wait a few seconds and confirm that the cluster is stable and that config.mqsc was applied:

for c in mq-node1 mq-node2 mq-node3; do echo "== $c ==" && docker exec "$c" dspmq -o nativeha -m QM_HA 2>/dev/null || true; done

You should see one node with ROLE(Active) and the other two with ROLE(Replica). If the containers are still initializing, wait a few more seconds and repeat.

Identify the active node and confirm that the queue and channel were created automatically by config.mqsc:

ACTIVE_NODE=$(for c in mq-node1 mq-node2 mq-node3; do docker exec "$c" dspmq -o nativeha -m QM_HA 2>/dev/null | grep -q "ROLE(Active)" && echo "$c" && break; done) && echo "Active node: $ACTIVE_NODE"

docker exec "$ACTIVE_NODE" bash -lc 'echo "DISPLAY QLOCAL(HA.TEST.QUEUE) CURDEPTH" | runmqsc QM_HA'

docker exec "$ACTIVE_NODE" bash -lc 'echo "DISPLAY CHANNEL(DEV.APP.SVRCONN) MCAUSER" | runmqsc QM_HA'

4. Quorum and Synchronous Replication Test

In this first phase, we will prove that the cluster survives and maintains data consistency internally.

Step 1: Put a Message on the Leader

Identify which node is active and store it in a variable:

ACTIVE_NODE=$(
  for c in mq-node1 mq-node2 mq-node3; do
    if docker exec "$c" dspmq -o nativeha -m QM_HA 2>/dev/null | grep -q "ROLE(Active)"; then
      echo "$c"
      break
    fi
  done
)

echo "Active node: $ACTIVE_NODE"

Then put a message on that node:

docker exec -it "$ACTIVE_NODE" /opt/mqm/samp/bin/amqsput HA.TEST.QUEUE QM_HA

Type: "Validating synchronous replication" and press Enter twice.

Step 2: Simulate Failure (Stop Container)

Stop the container that is currently active:

docker stop "$ACTIVE_NODE"

Step 3: Verify Election and Consistency

Check who the new leader is and store it in a variable:

NEW_ACTIVE_NODE=$(
  for c in mq-node1 mq-node2 mq-node3; do
    if docker exec "$c" dspmq -o nativeha -m QM_HA 2>/dev/null | grep -q "ROLE(Active)"; then
      echo "$c"
      break
    fi
  done
)

echo "New active node: $NEW_ACTIVE_NODE"

Retrieve the message there:

docker exec -it "$NEW_ACTIVE_NODE" /opt/mqm/samp/bin/amqsget HA.TEST.QUEUE QM_HA

Success proves that the message was replicated to the new leader's local log before the originally active node went down.

5. ACR (Automatic Client Reconnect) Demonstration

Now that we have validated the backend, we will test application-side resilience with the amqsphac sample, which uses MQCNO_RECONNECT internally and keeps the send loop active during and after failover.

Step 1: Ensure the cluster is operational

docker start mq-node1 && docker start mq-node2 && docker start mq-node3

ACTIVE_NODE=$(for c in mq-node1 mq-node2 mq-node3; do docker exec "$c" dspmq -o nativeha -m QM_HA 2>/dev/null | grep -q "ROLE(Active)" && echo "$c" && break; done) && echo "Active node: $ACTIVE_NODE"

Step 2: Create the sample startup script

Write the script line by line to avoid issues with pasting multi-line commands:

docker exec mq-client bash -c 'echo "#!/bin/bash" > /tmp/acr.sh'
docker exec mq-client bash -c 'echo "export MQSERVER=\"DEV.APP.SVRCONN/TCP/node1(1414),node2(1414),node3(1414)\"" >> /tmp/acr.sh'
docker exec mq-client bash -c 'echo "/opt/mqm/samp/bin/amqsphac HA.TEST.QUEUE QM_HA" >> /tmp/acr.sh'
docker exec mq-client bash -c 'chmod +x /tmp/acr.sh'

Step 3: Start the sample in the background

docker exec -d mq-client bash -c '/tmp/acr.sh'

amqsphac does not produce line-by-line output in non-interactive mode (stdout buffering), so verification is done via queue depth. Confirm that messages are reaching the active node:

sleep 5 && docker exec "$ACTIVE_NODE" bash -lc 'echo "DISPLAY QLOCAL(HA.TEST.QUEUE) CURDEPTH" | runmqsc QM_HA'

CURDEPTH should be continuously growing.

Step 4: Trigger the failure of the active node

docker stop "$ACTIVE_NODE"

Step 5: Verify election of the new leader

NEW_ACTIVE_NODE=$(for c in mq-node1 mq-node2 mq-node3; do docker exec "$c" dspmq -o nativeha -m QM_HA 2>/dev/null | grep -q "ROLE(Active)" && echo "$c" && break; done) && echo "New active node: $NEW_ACTIVE_NODE"

Step 6: Confirm automatic reconnect via queue growth

Wait about 10 seconds for the election to complete and check CURDEPTH on the new leader:

sleep 10 && docker exec "$NEW_ACTIVE_NODE" bash -lc 'echo "DISPLAY QLOCAL(HA.TEST.QUEUE) CURDEPTH" | runmqsc QM_HA'

If CURDEPTH is growing on the new leader, amqsphac reconnected automatically without any intervention — ACR worked.

Step 7: Confirm the messages received

docker exec -it "$NEW_ACTIVE_NODE" /opt/mqm/samp/bin/amqsget HA.TEST.QUEUE QM_HA

Key Conclusions for Docker

Layered Resilience: We first validated data replication and then application continuity.
ACR & Native HA: The perfect combination for fully high-availability architectures. amqsphac uses MQCNO_RECONNECT to demonstrate the transparent reconnection that a real application should implement.
Robust Configuration: Using volumes to mount Native HA configuration stanzas is the most reliable approach.

Cleanup

At the end of the lab, you can remove the containers and files created during the exercise:

docker compose down -v
rm -f node1.ini node2.ini node3.ini config.mqsc docker-compose.yaml

This cleanup does not uninstall Docker, Docker Compose, or the IBM MQ image. It removes only the local artifacts used by this lab.

Menu

Native High Availability (Native HA) in Docker: Cloud-Native Resilience

How does MQ Native HA work?

1. The Raft Protocol and Quorum

2. Synchronous Replication

3. Replica Limitations and Behavior

4. Automatic Client Reconnect (ACR)

Configuration Anatomy (`qm.ini`)

NativeHALocalInstance

NativeHAInstance

What this lab builds

Environment Preparation: Software Installation

1. Install Docker and Docker Compose

Permission Configuration (Linux)

2. Obtain the IBM MQ Image

1. Create the Configuration Files

MQ Nodes (`.ini`)

MQ Configuration File (`config.mqsc`)

2. The Orchestration File (Cluster + Client)

3. Start the Cluster

4. Quorum and Synchronous Replication Test

Step 1: Put a Message on the Leader

Step 2: Simulate Failure (Stop Container)

Step 3: Verify Election and Consistency

5. ACR (Automatic Client Reconnect) Demonstration

Step 1: Ensure the cluster is operational

Step 2: Create the sample startup script

Step 3: Start the sample in the background

Step 4: Trigger the failure of the active node

Step 5: Verify election of the new leader

Step 6: Confirm automatic reconnect via queue growth

Step 7: Confirm the messages received

Key Conclusions for Docker

Cleanup

Useful Documentation

More in this area

Native High Availability (Native HA) in Docker: Cloud-Native Resilience

How does MQ Native HA work?

1. The Raft Protocol and Quorum

2. Synchronous Replication

3. Replica Limitations and Behavior

4. Automatic Client Reconnect (ACR)

Configuration Anatomy (qm.ini)

NativeHALocalInstance

NativeHAInstance

What this lab builds

Environment Preparation: Software Installation

1. Install Docker and Docker Compose

Permission Configuration (Linux)

2. Obtain the IBM MQ Image

1. Create the Configuration Files

MQ Nodes (.ini)

MQ Configuration File (config.mqsc)

2. The Orchestration File (Cluster + Client)

3. Start the Cluster

4. Quorum and Synchronous Replication Test

Step 1: Put a Message on the Leader

Step 2: Simulate Failure (Stop Container)

Step 3: Verify Election and Consistency

5. ACR (Automatic Client Reconnect) Demonstration

Step 1: Ensure the cluster is operational

Step 2: Create the sample startup script

Step 3: Start the sample in the background

Step 4: Trigger the failure of the active node

Step 5: Verify election of the new leader

Step 6: Confirm automatic reconnect via queue growth

Step 7: Confirm the messages received

Key Conclusions for Docker

Cleanup

Useful Documentation

More in this area

IBM MQ Articles

Configuration Anatomy (`qm.ini`)

MQ Nodes (`.ini`)

MQ Configuration File (`config.mqsc`)