How to Setup Anthos for VM on Baremetal.

Anthos is Google Cloud’s managed, cloud-native application platform that simplifies the development and management of fleets of Kubernetes applications deployed in Google Cloud, on-premises, in other clouds, or at the edge. Anthos dramatically simplifies modernizing VMs for a cloud-native transformation and allows you to manage VMs through a single pane of glass.

Anthos for VMs supports enterprises that want to standardize on Kubernetes, but have existing workloads running on virtual machines that cannot be easily containerized or migrated to the cloud. Anthos for VMs lets you deploy, manage, and secure workloads running in virtual machines on the Anthos Platform, providing unified management, security policy, and observability across VMs and containers.

Let’s set up Anthos Cluster for VM on Baremetal.

First, we have to set up the Admin workstation for creating the cluster.

Admin Workstation

The admin workstation hosts command-line interface (CLI) tools and configuration files to provision clusters during installation and CLI tools for interacting with provisioned clusters post-installation.

Admin Workstation Configuration:-

Log in to one of the vm and perform the below installation:-

• Docker version 19.03 or later is installed

sudo apt-get update 
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
sudo apt install docker-ce
sudo systemctl status docker

• Add your user to the docker group to run docker as a non-root user

sudo usermod -aG docker $USER

• Download bmctl same version as the cluster version from the Cloud Storage bucket

gsutil cp gs://anthos-baremetal-release/bmctl/1.14.1/linux-amd64/bmctl bmctl
chmod a+x bmctl
sudo mv bmctl /usr/local/sbin/
bmctl version

• Install the kubectl binary

sudo apt-get install kubectl

• Authenticate to gcloud sdk using a service account or IAM user

# Authicating using service account 
gcloud auth activate-service-account <compute@developer.gserviceaccount.com> --key-file=key.json 

# Run this command command to authenticate by a IAM user
gcloud auth application-default login

• To create and manage clusters from your admin workstation, you need Layer 3 connectivity to all cluster node machines & Passwordless root access to all cluster node machines through SSH. SSH access can be either direct or through sudo.

Let’s Now setup the Anthos cluster

Anthos clusters on bare metal support multiple deployment models to meet different availability, isolation, and resource footprint needs. example

• Admin and user cluster deployment
• Hybrid cluster deployment
• Standalone cluster deployment

Based on the use case you can select your deployment model but in this blog, we are going to set up Standalone cluster deployment.

Standalone cluster

This deployment model has a single cluster that serves as a user cluster and as an admin cluster.

Advantages of using this Model:-

• It doesn’t require a separate admin cluster
• It supports the edge profile, which significantly reduces system resource requirements and is recommended for edge devices with high resource constraints.

Nodes Requirements:-

• Control Plane Nodes:
  - One control plane node for non-HA
  - Three or more control plane nodes for HA
• Worker nodes:
  - One or more worker nodes for non-HA
  - Two or more worker nodes for HA

Here we are going to configure HA Cluster so we have created seven nodes on bare-metal three for Control Plane nodes, three for worker nodes & one for admin workstation which we have already configured above.

Configuring Standalone cluster:-

• Get your Google Cloud project ID to use with cluster creation

export CLOUD_PROJECT_ID=$(gcloud config get-value project)

• Create a standalone cluster config file

bmctl create config -c bm-std-al-cluster-test-01 --enable-apis \
    --create-service-accounts --project-id=$CLOUD_PROJECT_ID

The output of the above command will be

This command will enable the required APIs, create the required service account & create a cluster config yaml. You need to update this config file after the update it will look like the one below.

# bmctl configuration variables. Because this section is valid YAML but not a valid Kubernetes
# resource, this section can only be included when using bmctl to
# create the initial admin/hybrid cluster. Afterwards, when creating user clusters by directly
# applying the cluster and node pool resources to the existing cluster, you must remove this
# section.
gcrKeyPath: bmctl-workspace/.sa-keys/<projectid>-anthos-baremetal-gcr.json
sshPrivateKeyPath: <sshkey_path> # like ~/.ssh/privatekey
gkeConnectAgentServiceAccountKeyPath: bmctl-workspace/.sa-keys/<project_id>-anthos-baremetal-connect.json
gkeConnectRegisterServiceAccountKeyPath: bmctl-workspace/.sa-keys/<project_id>-anthos-baremetal-register.json
cloudOperationsServiceAccountKeyPath: bmctl-workspace/.sa-keys/<project_id>-anthos-baremetal-cloud-ops.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-bm-std-al-cluster-test-01
---
# Cluster configuration. Note that some of these fields are immutable once the cluster is created.
# For more info, see https://cloud.google.com/anthos/clusters/docs/bare-metal/1.14/reference/cluster-config-ref#cluster_configuration_fields
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: bm-std-al-cluster-test-01
  namespace: cluster-bm-std-al-cluster-test-01
spec:
  # Cluster type. This can be:
  #   1) admin:  to create an admin cluster. This can later be used to create user clusters.
  #   2) user:   to create a user cluster. Requires an existing admin cluster.
  #   3) hybrid: to create a hybrid cluster that runs admin cluster components and user workloads.
  #   4) standalone: to create a cluster that manages itself, runs user workloads, but does not manage other clusters.
  type: standalone
  # Cluster profile. This can be either 'default' or 'edge'.
  # The edge profile is tailored for deployments on the edge locations
  # and should be used together with the 'standalone' cluster type.
  profile: edge
  # Anthos cluster version.
  anthosBareMetalVersion: 1.14.1
  # GKE connect configuration
  gkeConnect:
    projectID: <project_id>
  # Control plane configuration
  controlPlane:
    nodePoolSpec:
      nodes:
      # Control plane node pools. Typically, this is either a single machine
      # or 3 machines if using a high availability deployment.
      - address: <Machine 1 IP>
      - address: <Machine 1 IP>
      - address: <Machine 1 IP>
  # Cluster networking configuration
  clusterNetwork:
    # Pods specify the IP ranges from which pod networks are allocated.
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    # Services specify the network ranges from which service virtual IPs are allocated.
    # This can be any RFC1918 range that does not conflict with any other IP range
    # in the cluster and node pool resources.
    services:
      cidrBlocks:
      - 10.96.0.0/20
  # Load balancer configuration
  loadBalancer:
    # Load balancer mode can be either 'bundled' or 'manual'.
    # In 'bundled' mode a load balancer will be installed on load balancer nodes during cluster creation.
    # In 'manual' mode the cluster relies on a manually-configured external load balancer.
    mode: bundled
    # Load balancer port configuration
    ports:
      # Specifies the port the load balancer serves the Kubernetes control plane on.
      # In 'manual' mode the external load balancer must be listening on this port.
      controlPlaneLBPort: 443
    # There are two load balancer virtual IP (VIP) addresses: one for the control plane
    # and one for the L7 Ingress service. The VIPs must be in the same subnet as the load balancer nodes.
    # These IP addresses do not correspond to physical network interfaces.
    vips:
      # ControlPlaneVIP specifies the VIP to connect to the Kubernetes API server.
      # This address must not be in the address pools below.
      controlPlaneVIP: 10.0.0.8
      # IngressVIP specifies the VIP shared by all services for ingress traffic.
      # Allowed only in non-admin clusters.
      # This address must be in the address pools below.
      ingressVIP: 10.0.0.2
    # AddressPools is a list of non-overlapping IP ranges for the data plane load balancer.
    # All addresses must be in the same subnet as the load balancer nodes.
    # Address pool configuration is only valid for 'bundled' LB mode in non-admin clusters.
    addressPools:
    - name: pool1
      addresses:
    #   # Each address must be either in the CIDR form (1.2.3.0/24)
    #   # or range form (1.2.3.1-1.2.3.5).
      - 10.0.0.1-10.0.0.4
    # A load balancer node pool can be configured to specify nodes used for load balancing.
    # These nodes are part of the Kubernetes cluster and run regular workloads as well as load balancers.
    # If the node pool config is absent then the control plane nodes are used.
    # Node pool configuration is only valid for 'bundled' LB mode.
    # nodePoolSpec:
    #  nodes:
    #  - address: <Machine 1 IP>
  # Proxy configuration
  # proxy:
  #   url: http://[username:password@]domain
  #   # A list of IPs, hostnames or domains that should not be proxied.
  #   noProxy:
  #   - 127.0.0.1
  #   - localhost
  # Logging and Monitoring
  clusterOperations:
    # Cloud project for logs and metrics.
    projectID: <project_id>
    # Cloud location for logs and metrics.
    location: us-central1
  storage:
    # lvpNodeMounts specifies the config for local PersistentVolumes backed by mounted disks.
    # These disks need to be formatted and mounted by the user, which can be done before or after
    # cluster creation.
    lvpNodeMounts:
      # path specifies the host machine path where mounted disks will be discovered and a local PV
      # will be created for each mount.
      path: /mnt/localpv-disk
      # storageClassName specifies the StorageClass that PVs will be created with. The StorageClass
      # is created during cluster creation.
      storageClassName: local-disks
    # lvpShare specifies the config for local PersistentVolumes backed by subdirectories in a shared filesystem.
    # These subdirectories are automatically created during cluster creation.
    lvpShare:
      # path specifies the host machine path where subdirectories will be created on each host. A local PV
      # will be created for each subdirectory.
      path: /mnt/localpv-share
      # storageClassName specifies the StorageClass that PVs will be created with. The StorageClass
      # is created during cluster creation.
      storageClassName: local-shared
      # numPVUnderSharedPath specifies the number of subdirectories to create under path.
      numPVUnderSharedPath: 5
  # NodeConfig specifies the configuration that applies to all nodes in the cluster.
  nodeConfig:
    # podDensity specifies the pod density configuration.
    podDensity:
      # maxPodsPerNode specifies at most how many pods can be run on a single node.
      maxPodsPerNode: 250
---
# Node pools for worker nodes
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: node-pool-1
  namespace: cluster-bm-std-al-cluster-test-01
spec:
  clusterName: bm-std-al-cluster-test-01
  nodes:
  - address: <Machine 4 IP>
  - address: <Machine 5 IP>
  - address: <Machine 6 IP>

• First, check the preflight check using dry-run if there is any error in the config file it will show

bmctl check preflight -c bm-std-al-cluster-test-01

• When the preflight check passes, create the cluster using the below command

bmctl create cluster -c bm-std-al-cluster-test-01

• if the cluster is successfully created it will generate the kubeconfig file in the bmctl-workspace/cluster_name/cluster_name-kubeconfigdirectory. verify the cluster using kubectlcommand

export clusterid=bm-std-al-cluster-test-01
export KUBECONFIG=$HOME/bmctl-workspace/$clusterid/$clusterid-kubeconfig
kubectl get nodes

The cluster is created successfully

Now our cluster has been created successfully we can deploy our application in this cluster.

Log in to your cluster from the Google Cloud console

Even though both the admin and user clusters are registered to the GKE portal and Anthos portal in Google Cloud Console, you still need to log in to them to view all the details like that of a normal GKE cluster. (Ex: workloads, deployments, services, etc.)

There are multiple ways to log in to these clusters. The suggested way to log in is using your Google Identity(Google Identity Store) and RBAC (Kubernetes Access Control Method).

Let's create cluster roles for your Google users & services accounts so they can access the Anthos clusters. This process uses Connect gateway service of Google Cloud Platform to perform authentication and authorization.

• We can use gcloud cli to create RBAC and apply.

# first export the current context 
export CONTEXT="$(kubectl config current-context)"

# Now create the RBAC using gcloud and apply it 
gcloud container fleet memberships generate-gateway-rbac  \
    --membership=CLUSTER_NAME \
    --role=clusterrole/cluster-admin \
    --users=GOOGLE_ACCOUNT_EMAIL \
    --project=PROJECT_ID \
    --kubeconfig=$KUBECONFIG \
    --context=$CONTEXT\
    --apply

The output of this command is similar to the following

Now you can go to GCP Console either:

• In the GKE Clusters page, click Actions next to the registered cluster, then click Login. or
• In the Anthos Clusters page, select the cluster you want to log in to in the list of clusters, then click Login in the information panel that displays.

After successful login, you can see details about the cluster

Here we want to deploy our vm based workload. so first we have enabled vm runtime in the cluster.

Anthos VM Runtime

Anthos VM Runtime is a part of Anthos clusters on bare metal that lets you run VMs on top of Kubernetes in the same way that you run containers. With Anthos VM Runtime, you can run existing VM-based workloads as you also develop and run new container-based applications. Anthos VM Runtime is like the hypervisor in a more traditional virtualized environment.

• To enable the runtime, use the bmctl tool

bmctl enable vmruntime --kubeconfig <KUBECONFIG_PATH>

• To create and manage VMs, install the virtctl client tool as a kubectl plugin

export GOOGLE_APPLICATION_CREDENTIALS="PATH_TO_SERVICE_ACCOUNT_KEY/bm-gcr.json"
sudo -E bmctl install virtctl

• Verify that the virtctl plugin is installed

kubectl plugin list

If kubectl-virt is listed in the response, the plugin is successfully installed.

Let’s create a vm

• Create a VM in your cluster. The virtctl the plugin is used with the kubectl command

kubectl virt create vm testing-vm \
    --image ubuntu20.04 \
    --configure-initial-password root:testing123

Vm created successfully

• Verify whether the vm is running or not

kubectl get gvm testing-vm

vm running successfully

• Access the vm using the kubectl command

kubectl virt console testing-vm 

# Press enter 
# Give username & Password 

connected to the vm.

The image we have used in the above vm is the default Ubuntu image provided by GCP. let’s create a vm with the custom Windows image.

Note: When we enable vmruntime by default scratch storage is not enabled, we have to enable it.

Scratch Space

In Anthos VM runtime, scratch space refers to temporary storage space that is used for short-term data processing and storage.

• Updating scratch space in vmruntime

kubectl edit vmruntime vmruntime

Add the below part in yaml file:

storage:
    defaultStorageClass: local-shared # any storage class name you can give as per your cluster 

Then you should see in the status something like this

  storage:
    defaultScratchSpaceStorageClass: local-shared
    defaultStorageClass: local-shared

Now scratch space support is enabled.

Let’s Create a windows vm using a custom image:

First, create the custom image in qcow2 format and upload this image to cloud storage. you can keep this in your admin workstation vm also.

• Create a service account in GCP with cloud storage permission and then create a service account key.

# Create the service account using gcloud
gcloud iam service-accounts create SA_NAME \
    --description="DESCRIPTION" \
    --display-name="DISPLAY_NAME"

# Assign role to the service account 
gcloud projects add-iam-policy-binding [PROJECT-ID] --member "serviceAccount:[SA-EMAIL]" --role "roles/storage.admin"

# Create a service account key
gcloud iam service-accounts keys create [FILE-NAME].json --iam-account [SA-EMAIL]

• Create a secret in the Anthos cluster using this service account key so we can access images from the cloud storage bucket.

kubectl create secret generic gcscreds   --from-file=creds-gcp.json=<serviceaccount-json-path>

• Create a disk.yaml for windows vm and keep the below content

apiVersion: vm.cluster.gke.io/v1
kind: VirtualMachineDisk
metadata:
  name: windows-vm-disk
spec:
  size: 100Gi # keep size based on disk image size
  storageClassName: local-shared
  source:
    gcs:
      url: gs://<storage_bucket_name>/<image_name>
      secretRef: gcscreds

# Create the disk using kubectl command
kubectl create -f disk.yaml

# Check the disk is created or not 
kubectl get gdisk

As we create the disk, internally they will create a data volume with the same name here you can see the import process. when the import process is successful, you can create the vm.

# Let's check datavolume is imported successful or not 
kubectl get datavolume

Import successful

• Create a vm.yamlfile and keep the below content

---
apiVersion: vm.cluster.gke.io/v1
kind: VirtualMachine
metadata:
  name: windows-vm
  namespace: default
spec:
  compute:
    cpu:
      vcpus: 2
    memory:
      capacity: 2Gi
  disks:
  - boot: true
    driver: sata
    virtualMachineDiskName: windows-vm-disk
  interfaces:
  - name: eth0
    model: e1000
    default: true
    networkName: pod-network

# Create the vm using kubectl command 
kubectl create -f vm.yaml

# Wait for some time and then check vm is running or not 
kubectl get gvm 

vm is running successfully

• Let’s create a service to access the vm using RDP

apiVersion: v1
kind: Service
metadata:
  name: windows-vm-service
spec:
  selector:
    kubevirt/vm: windows-vm
  ports:
  - name: rdp
    protocol: TCP
    port: 3389
    targetPort: 3389
  type: NodePort

# Create the service 
kubectl create -f <file_name>

Now you can access the vm using RDP. Log in to a windows vm or GUI-based Linux vm in the same network and access it using RDP using the node port.

Thanks for reading!