IOT (Inception Of Things)

https://cdn.intra.42.fr/pdf/pdf/122428/en.subject.pdf

The system administration project Inception of things will use K3S and K3D to enhance our knowledge and provide an excellent introduction to K8S.

The prerequisites to exploit this project to the fullest IMO are knowledge in :

• Basic network knowledge

• K8S concepts -> https://kubernetes.io/docs/concepts/

• Vagrant manipulation -> https://developer.hashicorp.com/vagrant/docs

• A hypervisor that enables nested virtualization, if you want to follow my way to do it, so this prerequisite is not mandatory.

The IOT topic talks about server and worker nodes, terms used interchangeably here with master for server and agent for worker, aligning with traditional Kubernetes (K8S) vocabulary.

As a vagrant box I use CentOS (a derivative of RHEL), it is a popular choice for servers and production environments, theses distributions are highly tested before each release and ensure due to its stability, security and long-term support.

I started this project using 'centos/7' as Vagrant box, i just learned that its EOL dated June 30, 2024 consequently the mirror no longer exists, therefore we upgrade to the vagrant box 'generic/centos8', it's compatible with our configuration and dependency. For those who want to stay on this version, there is a nice ticket: https://serverfault.com/questions/1161816/mirrorlist-centos-org-no-longer-resolve.

PART 1

The first part of the project requires us to set up a cluster with a master node (called K3S server in this context) and an agent node, here is the K3S architecture.

Master Node: The master nodes host the control plane components of Kubernetes, which are responsible for the overall management of the cluster.

Agent Node: The agent nodes execute the cluster workloads, that is to say the containers grouped in pods.

Both of them run the kubelet instance which is a node-level agent that is in charge of executing pod requirements, managing resources, and guaranteeing cluster health. We can see in this diagram that it uses containerd, low-level runtime that uses the kubernetes & docker engine. Containerd is a software responsible for running and managing containers on a host system. It is a resource manager which manages the container processes, image, snapshots, container metadata and its dependencies.

In order to create the expected environment, we will instantiate VMs which will act as cluster nodes with a Vagrantfile.

V_BOX = 'generic/centos8'

VM_PROVIDER = 'virtualbox'

VB_CPU = 2
VB_MEM = 2048

MASTERNODE_MN = 'aschiffeS'
WORKERNODE_MN = 'aschiffeSW'

MASTERNODE_IP = '192.168.56.110'
WORKERNODE_IP = '192.168.56.111'

Vagrant.configure("2") do |config|
  config.vm.box = V_BOX

  config.vm.provider VM_PROVIDER do |v|
    v.cpus = VB_CPU
    v.memory = VB_MEM
  end
  
  # https://developer.hashicorp.com/vagrant/docs/triggers
  config.trigger.before :up do |trigger|
    trigger.run = { inline: "mkdir -p shared" }
  end
  
  # https://github.com/dotless-de/vagrant-vbguest
  config.vbguest.installer_options = { allow_kernel_upgrade: true }

  config.vm.define MASTERNODE_MN do |control|
    control.vm.hostname = MASTERNODE_MN
    control.vm.network "private_network", ip: MASTERNODE_IP
    control.vm.synced_folder "./shared", "/home/vagrant/shared", type: VM_PROVIDER
    control.vm.provision "shell", path: "scripts/s_provisioner.sh", env: { "MASTERNODE_IP" => MASTERNODE_IP }
  end

  config.vm.define WORKERNODE_MN do |control|
    control.vm.hostname = WORKERNODE_MN
    control.vm.network "private_network", ip: WORKERNODE_IP
    control.vm.synced_folder "./shared", "/home/vagrant/shared", type: VM_PROVIDER
    control.vm.provision "shell", path: "scripts/sw_provisioner.sh", env: { "WORKERNODE_IP" => WORKERNODE_IP,
                                                                            "MASTERNODE_IP" => MASTERNODE_IP }
  end
end

I'll let you discover the configuration specifics with the vagrant documentation, as you can see i provide some scripts to the servers in control.vm.provision. I mainly use it to run the necessary configurations on the respective nodes. Let's do a breakdown to see things more clearly.

In the Master Node s_provisioner.sh.

echo "[k42s] firewalld disabled"
systemctl disable firewalld --now

echo "[k42s] install packages"
sudo yum upgrade -y
sudo yum install -y net-tools

echo "[k42s] k3s installation on node"
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --node-ip=$MASTERNODE_IP --flannel-iface=eth1" K3S_KUBECONFIG_MODE="644" sh -s - --token 12345

echo "[k42s] server token are shared via synced_folder technique"
sudo cp /var/lib/rancher/k3s/server/token /home/vagrant/shared

In the Agent Node sw_provisioner.sh

echo "[k42s] firewalld disabled"
systemctl disable firewalld --now

echo "[k42s] install packages"
sudo yum upgrade -y
sudo yum install -y net-tools

echo "[k42s] k3s installation on node"
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent --server https://$MASTERNODE_IP:6443 --token-file /home/vagrant/shared/token --node-ip=$WORKERNODE_IP --flannel-iface=eth1" K3S_KUBECONFIG_MODE="644" sh -s -

In these last scripts we do a very classic configuration recommended by the K3S documentation. What is interesting here is how the agent node connects to the master. We serve a seed to the master node --token 12345 so that it generates a token, this will be copied into a folder shared between the host and my VMs. Like this, the agent node will have access to the token to join the network.

By default, Vagrant will share your project directory (the directory with the Vagrantfile) to /vagrant. But for learning purposes we used https://github.com/dotless-de/vagrant-vbguest for more modularity. In this first part we configured the most trivial cluster to grasp the K3S architecture.

PART 2

This second part asks us to set up 3 web services in the K3S instance, for this we can only use the master node, because it has access to kubelet, we will instantiate one pod per web service. 2 pods will have 1 replica, and the 3rd as much as its place, replicas are crucial for ensuring high availability, load balancing, scalability.

I chose NGINX as a web server for its robustness. Here are the K8S resources that I used to offer a web service. It consists of the following manifests :

• configMap (Stores key-value pairs for configuration data, It will be used to store the HTML page to be rendered, be careful the scope of a configMap cover the cluster namespace)

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-html-app1
data:
  index.html: |
    <!DOCTYPE html>
    <html>
    <head>
        <title>app1</title>
    </head>
    <body>
        <h1>Hello world!</h1>
    </body>
    </html>

• deployment (Manages the desired state of pod replicas)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1
  labels:
    app: app1 # Labels are used to identify and select this Kubernetes object.
spec:
  replicas: 1 # A ReplicaSet's purpose is to maintain a stable set of replica Pods.
  selector:
    matchLabels:
      app: app1 # Selector to match the pods with the label app: app1
  template:
    metadata:
      labels:
        app: app1 # Adds a label to the pod template metadata
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2 # container image
        ports:
        - containerPort: 80 # The port the container listens on inside the pod.
        volumeMounts:
        - name: html-volume
          mountPath: /usr/share/nginx/html # Specifies the path to mount the volume inside the container
      volumes:
      - name: html-volume
        configMap:
          name: nginx-html-app1 # Specifies the ConfigMap to use for the volume
          defaultMode: 0644 # file owner can read and write (6), group members and other users can only read (4).

• ingress (Manages external access to services)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app1-ingress
spec:
  rules:
  - host: app1.com # Host for which the Ingress rules apply
    http:
      paths:
      - path: / # Path to match incoming requests
        pathType: Prefix
        backend:
          service:
            name: app1-service # Name of the Kubernetes Service to forward traffic to
            port:
              number: 80 # Port number of the listening backend service for traffic

• service (Defines a logical set of pods and a policy to access them)

apiVersion: v1
kind: Service
metadata:
  name: app1-service
spec:
  selector:
    app: app1 # Selectors to identify the Pods that belong to this service
  ports:
    - protocol: TCP # Protocol used for communication
      port: 80 # Port number on which the service is exposed
      targetPort: 80 # Port number on the Pods where traffic should be forwarded

Don't forget to add these lines to your /etc/hosts so you can access your web services. The purpose of DNS is to translate a domain name into the appropriate IP address, try to access app1.com without overwriting it.

192.168.56.110 app1.com
192.168.56.110 app2.com
192.168.56.110 app3.com

Now we have 3 web services that can be accessed externally from the cluster and traffic is properly directed to the right service through ingress.

PART 3

This third part focuses on the GitOPS practice, which will be exploited with ArgoCD.

In a traditional project, what happens is that you have a base repository, of your project source code, CI / CD software like jenkins detects changes over time and activates the appropriate pipelines, it will mainly generate an artifact of your project, like a docker image for example and deploy it on a private or public hub, moreover if we follow good practices (Separate the source code and configuration files on different repos), it will update the configuration file which specifies the new image a pull from the hub.

This is exactly where the ArgoCD agent comes into play, it runs within our cluster and is configured to stay in sync with the configuration repo and apply these manifests to keep the deployed application up to date.

In this section, we will use k3d, a lightweight tool for running k3s (Rancher Labs’ minimal Kubernetes distribution) within Docker containers. Essentially, instead of relying on virtual machines (VMs) and hypervisors to create nodes, we will use Docker containers that will function as nodes within our Kubernetes clusters.

The idea here is not to re-simulate the generation of the artifact as explained in my traditional example but the modification of the repo that uses the concept of IaC, you can take a look at https://www.redhat.com/en/topics/automation/what-is-infrastructure-as-code-iac.

Therefore ArgoCD will detect the change on this repo to deploy the right infrastructure.

In order to configure the architecture of this diagram, we will dissect the essential commands together, and understand what they do, in order to reproduce the entire project, you will need the complete code with dependencies available onhttps://github.com/vmmon-th0/Inception_OT/blob/main/p3/scripts/init.sh.

Init Cluster & Argo CD

This will create a cluster, and maps port 8081 on the host to port 30080 on the cluster's load balancer (The port mapping will be used to access the deployed application "WIL APP" on the schema).

function init_cluster {
    local CLUSTER_NAME="iot-cluster"
    k3d cluster create $CLUSTER_NAME --port 8081:30080@loadbalancer
    kubectl config use-context k3d-$CLUSTER_NAME
    echo "The cluster '$CLUSTER_NAME' was successfully initialized and set to kubectl context"
}

Here is our created cluster, we can also observe our nodes.

└[~/Documents/Projects/Inception_OT/p3/scripts]> k3d cluster list
NAME          SERVERS   AGENTS   LOADBALANCER
iot-cluster   1/1       0/0      true
└[~/Documents/Projects/Inception_OT/p3/scripts]> k3d node list
NAME                       ROLE           CLUSTER       STATUS
k3d-iot-cluster-server-0   server         iot-cluster   running
k3d-iot-cluster-serverlb   loadbalancer   iot-cluster   running

Here we will create our argocd service, and accordingly install this technology in this scope.

function init_argoCD {
    local NAMESPACE="argocd"

    kubectl create namespace $NAMESPACE
    kubectl apply -n $NAMESPACE -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
    sleep 20
    kubectl wait --for=condition=ready pod --all --namespace=$NAMESPACE --timeout=600s

    if [ $? -eq 0 ]; then
        kubectl apply -f ../confs/application.yaml
        echo "ArgoCD was successfully deployed."
        kubectl port-forward -n $NAMESPACE svc/argocd-server 8080:443 > /tmp/argocd-port-forwarding.log 2>&1 &
        echo "Port forwarding has been successfully completed, the Argo CD UI is now available from your host machine"
        kubectl -n $NAMESPACE get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d > /tmp/argo-credentials.txt
        echo "Output location of ArgoCD initial admin secret: ./argo-credentials.txt"
    else
        echo "Timeout ! Some pods in the namespace $NAMESPACE are not ready. Please check the pod status."
    fi
}

We will execute ArgoCD application.yaml, you can see that we have configured a source, which will contain the repo containing the IaC. This is the one that will be in sync permanently.

kubectl apply -f ../confs/application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:I
  name: guestbook
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/vmmon-th0/argo_watcher.git
    targetRevision: HEAD
    path: ./
  destination:
    server: https://kubernetes.default.svc
    namespace: dev
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    automated:
      selfHeal: true
      prune: true

Now everything is ready we can test this trivial system. go to localhost:8080, connect with :

login: admin

password: should be generated in /tmp/argo-credentials.txt

Arriving on this beautiful interface you can notice several components that represent their workflow. Check https://argo-cd.readthedocs.io/en/stable/ for more informations.

Argo CD GUI Interface

Let's test what we saw above, let's modify the repo acting as infrastructure as code. By switching from v2 to v1 of the image hosted on docker you will notice that it takes care of the adequate deployment while keeping the old version in cache for optimization purposes.

Conclusion

In short, we have gone over 3 concepts that allow us to better familiarize ourselves with the use cases that Kubernetes can offer us or in which it is involved, it is a very interesting technology.