Migrating RabbitMQ from AWS EC2 to Google Kubernetes Engine (GKE)

For one of our clients, we recently migrated RabbitMQ which was running on EC2 instance on AWS to a GKE cluster on GCP.

In this article, we will guide you through a step-by-step process for setting up RabbitMQ on an AWS EC2 instance and Google Kubernetes Engine (GKE). We will also demonstrate how to migrate RabbitMQ from an EC2 instance to GKE. This migration process involves taking a backup and restoring it.

Setting Up RabbitMQ on AWS EC2:

Step 1: Prerequisites

Before we begin the RabbitMQ setup, ensure you have completed the following prerequisites:

1. Create a EC2 instance on AWS.

Step 2: Installation of RabbitMQ in a vm

Run the following commands for installation:

## Update package indices
sudo apt-get update -y

## Install Erlang packages
sudo apt-get install -y erlang-base \
                        erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
                        erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
                        erlang-runtime-tools erlang-snmp erlang-ssl \
                        erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl

## Install rabbitmq-server and its dependencies
sudo apt-get install rabbitmq-server -y --fix-missing

For more details you can check official site:

https://www.rabbitmq.com/install-debian.html#manual-installation

Step 3: Use this command to check whether the rabbitmq-server is running

systemctl status rabbitmq-server

#If RabbitMQ is not running, use the command to start rabbitmq-server

systemctl start rabbitmq-server

#Get Cluster Status use the command

rabbitmqctl cluster_status

rabbitmq-server running

Step 4: You need to enable the management plugin:

rabbitmq-plugins enable rabbitmq_management

rabbitmqadmin plugins enabled

Step 5: We are creating rabbitmq-test-user, then we will take a backup of the configurations using the backup file:

rabbitmqctl add_user "<username>" "<password>"

#rabbitmqctl add_user "rabbitmq-test-user" "shubhi"

#list all users using this command

rabbitmqctl list_users

RabbitMQ user created

list all users

Step 6: To backup RabbitMQ configurations, use the command:

rabbitmqadmin export <backup-file-name> 

#rabbitmqadmin export rabbitmq.json

After applying this command, a file was created

Setting Up RabbitMQ on GKE:

Step 1: Prerequisites

Initiating the RabbitMQ setup, please verify that you’ve fulfilled the following prerequisites:

1. Create a Kubernetes Cluster on GKE.
2. After the cluster is created, connect to it using.

Step 2: Create a statefulset for RabbitMQ pods

In Kubernetes, using a StatefulSet for RabbitMQ pods is effectively handling stateful applications. To set it up, you start by creating a YAML file called rabbitmq-statefulset.yaml that defines the configuration of your RabbitMQ Pods. This ensures reliable performance in your Kubernetes cluster.

The YAML file will contain the following content:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rabbitmq-sc
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-balanced
volumeBindingMode: Immediate
reclaimPolicy: Retain
allowVolumeExpansion: true

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  labels:
    app: rabbitmq
spec:
  selector:
    matchLabels:
      app: "rabbitmq"
      log-label: "true"
  # headless service that gives network identity to the RMQ nodes, and enables them to cluster
  serviceName: rabbitmq
  replicas: 1
  template:
    metadata:
      labels:
        app: rabbitmq
        log-label: "true"
    spec:
      containers:
      - name: rabbitmq
        volumeMounts:
          - name: rabbitmq-pv
            mountPath: "/var/lib/rabbitmq/"
        image: rabbitmq:3.12
        ports:
        - name: amqp
          containerPort: 5672
          protocol: TCP
        - name: management
          containerPort: 15672
          protocol: TCP
        - name: prometheus
          containerPort: 15692
          protocol: TCP
        - name: epmd
          containerPort: 4369
          protocol: TCP
        livenessProbe:
          exec:
            command: ["rabbitmq-diagnostics", "status"]
          initialDelaySeconds: 60
          periodSeconds: 60
          timeoutSeconds: 15
        readinessProbe: # probe to know when RMQ is ready to accept traffic
          exec:
            command: ["rabbitmq-diagnostics", "ping"]
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10
  volumeClaimTemplates:
    - metadata:
        name: rabbitmq-pvc
      spec:
        storageClassName: "rabbitmq-sc"
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

By creating this StatefulSet, Kubernetes will ensure that the MongoDB Pods are running and available in a predictable manner.

To create the StatefulSet, run the following command:

kubectl apply -f rabbitmq-statefulset.yaml

This creates a StatefulSet for MongoDB Pods based on the YAML configuration. You can confirm its successful creation by running this command:

kubectl get statefulsets

This will show a list of all the StatefulSets in your Kubernetes cluster, including the one you’ve just set up.

Step 3: Create a Service for the RabbitMQ pods.

When you need to access individual RabbitMQ pods by their hostname within Kubernetes, creating a headless service is valuable.To create this service, you’ll create a YAML file named ‘rabbitmq-service.yaml’ and include the following configuration:

kind: Service
apiVersion: v1
metadata:
  name: rabbitmq
  labels:
    app: rabbitmq
spec:
  selector:
    app: rabbitmq
  ports:
  - name: rabbitmq
    port: 15672

After creating the YAML file, you can establish the headless service by executing this command:

kubectl apply -f mongo-service.yaml

Step 4: To restore RabbitMQ configurations from a backup

Access the RabbitMQ pod using the following command:

kubectl exec -it rabbitmq-0 -- /bin/bash

Once you’ve successfully entered the pod, You need to enable the rabbitmqadmin management plugin

rabbitmq-plugins enable rabbitmq_management

To restore rabbitMQ configuration, use the command at the location of your backup file

rabbitmqadmin import <JSON backup file>

#rabbitmqadmin import rabbitmq.json

RabbitMQ configurations restored

You can see here RabbitMQ configurations restored.

Hope you find it helpful! Keep learning, until next time :)