IBM MQ streaming queues in practice

IBM MQ streaming queues let a queue manager place a near-identical copy of each message on a second queue. The application that puts to the original queue does not need to change, and the application that gets from the original queue remains unaware that streaming took place.

This article is deliberately practical. It uses a single IBM MQ container, creates a small set of local queues, and then validates the two streaming quality-of-service modes (BESTEF and MUSTDUP).

It also introduces CAPEXPRY, which is important when you want to retain duplicate messages for a limited period instead of letting the destination queue grow indefinitely.

1. Lab goals

By the end of this lab you should be able to:

• Explain the purpose and behavior of IBM MQ streaming queues
• Configure STREAMQ and STRMQOS on local queues
• Understand the difference between BESTEF and MUSTDUP
• Understand why CAPEXPRY matters on the destination queue
• Recognize the main restrictions that apply to streaming queues

2. Required software

For this lab, we will use:

• IBM MQ 9.4.x on Linux
• Docker Engine and Docker Compose plugin
• a Linux account with permission to run Docker commands

If Docker is not installed yet, run:

• Ubuntu / Debian

  sudo apt-get update
  sudo apt-get install -y docker.io docker-compose-plugin
  sudo systemctl enable --now docker

• RHEL / CentOS / Fedora

  sudo dnf -y install dnf-plugins-core
  sudo dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
  sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  sudo systemctl enable --now docker

Verify the installation:

docker version
docker compose version

Pull the IBM MQ image:

docker pull icr.io/ibm-messaging/mq:latest

3. What streaming queues add to an MQ design

Streaming queues are configured in queue properties, not in applications. The attributes to consider are:

The main design decision is usually this:

• If the original business flow must continue even when the duplicate path has problems, use BESTEF
• If the duplicate copy is mandatory, for example for compliance or guaranteed replay capture, use MUSTDUP

4. Important operational behavior

Two details are important to know:

1. The duplicate message is created by the queue manager, not by the putting application 2. When MQ streams a message, the duplicate copy has its expiry reset to unlimited unless you constrain it with CAPEXPRY on the queue receiving the copy

That second point is easy to overlook. If you stream to a history queue and nobody consumes it, that queue will continue growing without control.

5. Main restrictions to keep in mind

There are a few important restrictions in the streaming mechanism:

• It is not possible to create streaming chains such as Q1 -> Q2, Q2 -> Q3
• It is not possible to create loops such as Q1 -> Q2, Q2 -> Q1
• It is not possible to define STREAMQ on a transmission queue
• STREAMQ cannot refer to a model queue

For simplicity, this lab uses only local queues on a single queue manager.

6. Create the lab folder and the docker-compose file

Create a working folder:

mkdir -p ~/mq-streaming-lab
cd ~/mq-streaming-lab

Create the docker-compose.yaml file:

cat > docker-compose.yaml <<'EOF'
services:
  mq-dev:
    container_name: mq-dev
    image: icr.io/ibm-messaging/mq:latest
    environment:
      LICENSE: "accept"
      MQ_QMGR_NAME: "QM1"
    ports:
      - "1414:1414"
EOF

7. Start the queue manager

Start the Docker container:

docker compose up -d

Confirm that the queue manager is active:

docker exec mq-dev dspmq

You should see QM1 in Running state.

8. Create the lab queues

This block creates the following queues:

• APP.IN.NORMAL for a normal streaming example
• APP.IN.BEST for the BESTEF scenario
• APP.IN.MUST for the MUSTDUP scenario
• APP.AUDIT as the normal duplicate queue
• APP.AUDIT.SMALL as a deliberately constrained duplicate queue

docker exec -i mq-dev runmqsc QM1 <<'EOF'
DEFINE QLOCAL(APP.IN.NORMAL) REPLACE
DEFINE QLOCAL(APP.IN.BEST) REPLACE
DEFINE QLOCAL(APP.IN.MUST) REPLACE
DEFINE QLOCAL(APP.AUDIT) REPLACE
DEFINE QLOCAL(APP.AUDIT.SMALL) MAXDEPTH(1) REPLACE
CLEAR QLOCAL(APP.IN.NORMAL)
CLEAR QLOCAL(APP.IN.BEST)
CLEAR QLOCAL(APP.IN.MUST)
CLEAR QLOCAL(APP.AUDIT)
CLEAR QLOCAL(APP.AUDIT.SMALL)
END
EOF

Confirm the definitions:

docker exec -i mq-dev runmqsc QM1 <<'EOF'
DISPLAY QLOCAL(APP.IN.NORMAL) STREAMQ STRMQOS
DISPLAY QLOCAL(APP.IN.BEST) STREAMQ STRMQOS
DISPLAY QLOCAL(APP.IN.MUST) STREAMQ STRMQOS
DISPLAY QLOCAL(APP.AUDIT) CURDEPTH MAXDEPTH CAPEXPRY
DISPLAY QLOCAL(APP.AUDIT.SMALL) CURDEPTH MAXDEPTH CAPEXPRY
END
EOF

9. Basic streaming with BESTEF

Enable streaming from APP.IN.NORMAL to APP.AUDIT:

docker exec -i mq-dev runmqsc QM1 <<'EOF'
ALTER QLOCAL(APP.IN.NORMAL) STREAMQ(APP.AUDIT) STRMQOS(BESTEF)
CLEAR QLOCAL(APP.IN.NORMAL)
CLEAR QLOCAL(APP.AUDIT)
DISPLAY QLOCAL(APP.IN.NORMAL) STREAMQ STRMQOS
END
EOF

Put a message:

docker exec mq-dev bash -lc "printf 'stream-normal-1\n' | /opt/mqm/samp/bin/amqsput APP.IN.NORMAL QM1"

Browse the original queue:

docker exec mq-dev /opt/mqm/samp/bin/amqsbcg APP.IN.NORMAL QM1

Browse the duplicate queue:

docker exec mq-dev /opt/mqm/samp/bin/amqsbcg APP.AUDIT QM1

At this point you should see the same content on both queues.

10. BESTEF behavior when the duplicate queue is already full

We will now create a scenario in which the streaming destination queue cannot receive more messages and the original path must continue to work.

Configure APP.IN.BEST to stream to the small queue:

docker exec -i mq-dev runmqsc QM1 <<'EOF'
ALTER QLOCAL(APP.IN.BEST) STREAMQ(APP.AUDIT.SMALL) STRMQOS(BESTEF)
CLEAR QLOCAL(APP.IN.BEST)
CLEAR QLOCAL(APP.AUDIT.SMALL)
END
EOF

Put one message in APP.AUDIT.SMALL so it reaches MAXDEPTH(1):

docker exec mq-dev bash -lc "printf 'pre-fill-small-queue\n' | /opt/mqm/samp/bin/amqsput APP.AUDIT.SMALL QM1"

Check the queue depth:

docker exec -i mq-dev runmqsc QM1 <<'EOF'
DISPLAY QLOCAL(APP.AUDIT.SMALL) CURDEPTH MAXDEPTH
END
EOF

Now put a message on the original queue configured with BESTEF:

docker exec mq-dev bash -lc "printf 'best-effort-message\n' | /opt/mqm/samp/bin/amqsput APP.IN.BEST QM1"

Check the depth of both queues:

docker exec -i mq-dev runmqsc QM1 <<'EOF'
DISPLAY QLOCAL(APP.IN.BEST) CURDEPTH
DISPLAY QLOCAL(APP.AUDIT.SMALL) CURDEPTH
END
EOF

Interpretation:

• APP.IN.BEST should contain the original message
• APP.AUDIT.SMALL should remain full with only the preloaded message

This is exactly what BESTEF means: the original message path remains available even when the duplicate cannot be delivered.

11. MUSTDUP behavior when the duplicate queue is already full

Now we move to the stricter mode. Configure APP.IN.MUST to stream to the same queue, which has a maximum depth of one message:

docker exec -i mq-dev runmqsc QM1 <<'EOF'
ALTER QLOCAL(APP.IN.MUST) STREAMQ(APP.AUDIT.SMALL) STRMQOS(MUSTDUP)
CLEAR QLOCAL(APP.IN.MUST)
DISPLAY QLOCAL(APP.IN.MUST) STREAMQ STRMQOS
END
EOF

Keeping APP.AUDIT.SMALL full, try to put a new message:

docker exec mq-dev bash -lc "set -o pipefail; printf 'must-duplicate-message\n' | /opt/mqm/samp/bin/amqsput APP.IN.MUST QM1; echo RC:$?"

Check the queues again:

docker exec -i mq-dev runmqsc QM1 <<'EOF'
DISPLAY QLOCAL(APP.IN.MUST) CURDEPTH
DISPLAY QLOCAL(APP.AUDIT.SMALL) CURDEPTH
END
EOF

• The put should fail because the duplicate queue cannot accept the copy
• APP.IN.MUST should remain empty
• APP.AUDIT.SMALL should remain at depth 1

This is the operational difference that matters most in practice:

• BESTEF protects the original application flow
• MUSTDUP protects the guarantee that a duplicate copy exists

12. Apply retention with CAPEXPRY

If you stream to a history queue and nobody consumes it, messages accumulate. A common practice is to use CAPEXPRY to limit the life of the duplicate copy.

Set CAPEXPRY on the destination queue:

docker exec -i mq-dev runmqsc QM1 <<'EOF'
ALTER QLOCAL(APP.AUDIT) CAPEXPRY(300)
DISPLAY QLOCAL(APP.AUDIT) CAPEXPRY
END
EOF

This means that new messages arriving on APP.AUDIT will not be able to live longer than 300 seconds, even if the duplicate copy would otherwise have unlimited expiry.

In a production scenario, this is sometimes preferable to creating a separate application just to clean the history queue.

13. What the lab showed

This lab demonstrated four concrete points:

1. Streaming queues are configured on queue objects, not in the application that puts messages. 2. The queue manager places a near-identical copy of the message on the secondary queue. 3. BESTEF and MUSTDUP are materially different when the duplicate path has a problem. 4. CAPEXPRY is important for controlling retention on the destination queue.

14. When streaming queues are a good option

Streaming queues are especially useful when you need:

• A short-term history of business messages
• A duplicate feed for analysis or observability
• A replay source for recovery tests
• A low-intrusion mechanism to copy messages to another processing flow

They are less suitable when you need complex fan-out or transformation logic. In those cases, publish/subscribe or an additional integration layer will probably be a better choice.

15. Summary

IBM MQ streaming queues are deliberately simple: configure a source queue with STREAMQ, choose BESTEF or MUSTDUP, and let the queue manager duplicate the messages.

The practical design question to keep in mind is not whether streaming works, but what should happen in case of failure:

• If the original business path must remain insulated from failures in the duplicate path, use BESTEF
• If the duplicate copy is mandatory, use MUSTDUP

If needed, use CAPEXPRY to control message retention on the destination queue.

16. Cleanup

This cleanup step removes only the lab artifacts. It does not uninstall Docker or IBM MQ from the host.

docker compose down -v
cd ..
rm -rf ~/mq-streaming-lab

Useful IBM documentation

• Streaming queues
• Streaming queues configuration
• Streaming queue restrictions
• Using streaming queues to store a history of messages
• DEFINE queues reference (STREAMQ, STRMQOS, CAPEXPRY)