Kubernetes

Owned by Roger Cruz
Last updated: Apr 01, 2024
27 min read

Intro

My notes on Kubernetes

Documentation

 

Tips and Tidbits

  • Kubernetes 101: Pods, Nodes, Containers, and Clusters

  • A node is the smallest unit of computing hardware in Kubernetes. It is a representation of a single machine in your cluster. 

  • Kubernetes doesn’t run containers directly; instead it wraps one or more containers into a higher-level structure called a pod.

    • Any containers in the same pod will share the same resources and local network.

    • Containers can easily communicate with other containers in the same pod as though they were on the same machine while maintaining a degree of isolation from others.

  • A Pod is a Kubernetes abstraction that represents a group of one or more application containers (such as Docker), and some shared resources for those containers. Those resources include:

    • Shared storage, as Volumes

    • Networking, as a unique cluster IP address

    • Information about how to run each container, such as the container image version or specific ports to use

  • The kubelet on the node hands the work to the runtime (containerd)

 

Learning Resources

 

Installation Resources

 

Pod Startup

Source: KubeCon 2022: So what if I don’t want my Persistent Storage to be yet another Bind Mount. Slides

https://static.sched.com/hosted_files/kccncna2022/e7/DirectStorageKubecon2022.pdf

 

 

 

Monitoring

 

Source: Hybrid Monitoring and Logging on Kubernetes (Cloud Next '18)

In this talk, we will demonstrate how you can use identical monitoring and logging across on-prem and Google Cloud using classic tools to establish a reliable multi-cluster monitoring and logging system, pre-configured and supported by Google. In addition, we will walk through Kubernetes monitoring data and explain how it's generated and collected in the darkest corners of the stack. (stackdriver)

 

Initializing Control-Plane (Master Node)

 

 

  • Get the IP address of the master node

ip addr show

 

  • Use the IP address from above in the following command

sudo kubeadm init --apiserver-advertise-address 10.128.0.4 .... Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 10.128.0.4:6443 --token ymihb0.95lfvw8id381xu4q \ --discovery-token-ca-cert-hash sha256:f2af13b8c467ef96c2ca16ce6e550d3d695dc56b22b0447fecb2536f073f3c2b

 

Joining A Node

 

Go to each node that you want to join to the cluster and issue the join command generated above.

 

kubeadm join 10.128.0.4:6443 --token ymihb0.95lfvw8id381xu4q \ --discovery-token-ca-cert-hash sha256:f2af13b8c467ef96c2ca16ce6e550d3d695dc56b22b0447fecb2536f073f3c2b ... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

 

Configure Networking Plugin On Master Node

 

 

Installing K8 Dashboard

An add-on service.

 

The pod is created in the kube-system namespace

The dashboard is setup to respond on port 31000 by default. To access it, you can use any of the nodes host IP address (10.128.0.4-6 from example below) . Kubernetes will redirect the traffic to the node that is actually running the application (kube-02 in the picture above)

 

 

Setup Visualizer

 

Visualizer runs on node 32000 and like the dashboard, you can access it using any of your nodes' IP address: http://<NODE_IP>:32000/#scale=2.0

More info on this tool can be found here: https://kubernetes-operational-view.readthedocs.io/en/latest/

Kubernetes Resources Specs

Kubernetes resources come in the form of pods, replication sets, services, deployments, etc. There is a long list of resources which you can view with kubectl api-resources. You can define these resources using a spec file written in YAML.

The resource file follows this format:

with details of each section found in Kubernetes API (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/). For example, the spec for creating a pod is given here: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#pod-v1-core

Images

In the example below, we are defining a pod that will use a Docker container with an image vote pulled from schoolofdevops docker registry.

 

The format for specifying an image is: <registry name : port>/<image name : version tag>

When the registry name is not specified, then the Docker public registry is assumed. If a tag is not specified, Docker assumes it is latest, though you should always specify a version so you know what exactly is running in your environment.

More information on Docker images here: https://kubernetes.io/docs/concepts/containers/images

You can specify arguments to the image as detailed here: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/

Adding Volumes To Pods

Kubernetes documenation on volumes can be found here: https://kubernetes.io/docs/concepts/storage/volumes/

  • A Kubernetes volume has an explicit lifetime - the same as the Pod that encloses it. Consequently, a volume outlives any Containers that run within the Pod, and data is preserved across Container restarts.

  • When a Pod ceases to exist, the volume will cease to exist, too.

  • Kubernetes supports many types of volumes, and a Pod can use any number of them simultaneously.

  • To use a volume, a Pod specifies what volumes to provide for the Pod via the .spec.volumes field and where to mount those into Containers via the .spec.containers[*].volumeMounts field

HostPath

A hostPath volume mounts a file or directory from the host node's filesystem into your Pod. It adds storage from the node’s host to the pod. Note that if the pod is placed in a different node, the data will not be accessible. So this is only good for development.

To add storage, you need to provide the volume information (where it is) and then a mount point (how to access it from the pod’s perspective)

Here is an example of how to do it in the container’s spec. The node’s host path, /var/lib/pgdata, will be the location for the storage and it will be mounted in the container’s file system under /var/lib/postgresql/data

 

 

Multi-Container Pods

A pod can run more than one container. For example, you can have an web engine and another helper program running in the same pod

 

You can shell into each container with the -c option

Similarly, you can view each container’s standard output by specifying the container.

 

Each container has its own process and storage space but they both share the same networking configuration and hostname

 

Limiting Resource Consumption

You can optionally specify a minimum and a maximum resource consumption for each container in a pod. The two typical resources controlled through the pod spec are CPU and Memory. A minimum amount is specified as a request and its maximum as a limit. A container is allow to exceed its requested amount if there are resources in the node but it will never exceed the specified limits. Scheduling and resource management is controlled by the kubelet process.

Read more on this topic here: Managing Resources for Containers

Limits and requests for CPU resources are measured in cpu units. One cpu, in Kubernetes, is equivalent to 1 vCPU/Core for cloud providers and 1 hyperthread on bare-metal Intel processors. Fractional values are accepted. 1 vCPU = 1000 millicores so .5 is equal to half a vCPU (500 millicores). Memory is measure in bytes and can be expressed in power-of-twos value (Ki = 1024 bytes) or decimal values (K = 1000 bytes).

Here is an example from the Kubernetes documenetation

 

If you specify a request value that is larger than the specified limit, Kubernetes won’t schedule the pod.

If your pod requests more memory than it is available in the node, then it will remain in the pending state since it can’t be scheduled.

 

Assigning Pods to Nodes

You can constrain a Pod to only be able to run on particular Node(s), or to prefer to run on particular nodes. This can be done via Labels and Selectors.

  • Labels are key/value pairs that are attached to objects.

  • Each key must be unique for a given object.

  •  Valid label keys have two segments: an optional prefix and name, separated by a slash (/). 

    • The name segment is required and must be 63 characters or less.

    • The prefix is optional. If specified, the prefix must be a DNS subdomain

      • If the prefix is omitted, the label Key is presumed to be private to the user.

      • The kubernetes.io/ and k8s.io/ prefixes are reserved for Kubernetes core components

  • Label selectors 

    • A label selector can be made of multiple requirements which are comma-separated.

    • In the case of multiple requirements, all must be satisfied so the comma separator acts as a logical AND (&&) operator.

Debugging

 

Container is in a crash loop. You caan see this in the pod status field: CrashLoopBackOff

 

Use kubectl describe pod db to view the pod’s configuration, including its start up events

 

 

 

  • You can use to look kubectl logs db at the standard output

 

  • Download the Docker image: postgres:9.4

  • Next, run the image and see what errors occur.

So we have the answer. The DB password needs to be specified OR all connections should be trusted (ie no password).

  • Confirm this will fix our problem:

  • Now, we just need to modify our spec.yaml to pass the value as an environment variable to postgres.

Common kubectl Commands

 

Monitoring

Use this Linux command to monitor status every 1 sec.

 

 

kubectl version

  • Show Kubernetes version with output in YAML.

 

kubectl get nodes

  • List of nodes in the Kubernetes cluster.

 

  • List the nodes in the kube-system namespace.

 

kubectl get pods

  • List of pods in the Kubernetes cluster.

  • More info with -o wide. You can also specify output in -o yaml or -o json.

 

  • Show labels

kubectl describe pods

  • Detailed information about a pod

 

kubectl logs

  • Get the standard output for a specified container (-c <container>) in a pod.

kubectl cluster-info

  • Show info on cluster, like the master node.

kubectl get cs

  • Status of node components.

Running Pods

kubectl run

  • Runs a Redis image in a pod.

 

kubectl exec

  • Runs a command in the pod. In this case, it opens a shell in pod vote.

Namespaces

Namespaces allow you to segragate your pods from others.

kubectl config view

  • Show the configuration of the cluster as stored in ~/.kube/config. You can see the contexts here.

  • To get help on this command.

kubectl config get-contexts

  • List available contexxts. Note the namespace field.

kubectl config current-context

  • Show the currrent context.

 

kubectl config set-context

  • Change the currrent context’s namespace. Note the namespace is changed.

kubectl get namespace

  • Get the current namespace (ns). There is a default namespace and a system one.

 

kubectl create namespace

  • Creates a new namespace (ns): instavote

 

ReplicaSets

Replicasets allow you to quickly clone your pods and distribute them to different nodes to maintain a stability.

“A ReplicaSet's purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods”.

 

 

Services

kubectl apply

  • Deploy a service.

kubectl get svc

  • Get a list of Kubernetes services. Of importance here are the selector, endpoints and the type of networking and nodeport #.

This service uses NodePort and it is exposed on TCP port 30000 (defined in the service’s spec file) on the K8 nodes. So if we find any node’s IP address, we can access the app.

 

kubectl edit svc

  • You can use this command to edit a service on the fly. For example, you can change its selector.

If you change the selector to a pod label that doesn’t exist (role=voteee), you will see that the endpoints are not populaed. There is always one endpoint for each pod it finds via a match on the selector fields.

 

Autoscale Metrics using KEDA

 

Custom Resource Definitions (CRD)

Kubernetes provides two ways to add custom resources to your cluster:

  • CRDs are simple and can be created without any programming.

  • API Aggregation requires programming, but allows more control over API behaviors like how data is stored and conversion between API versions.

  • CRDs allow users to create new types of resources without adding another API server. You do not need to understand API Aggregation to use CRDs.