Kubernetes in Action

1. Kubernetes Orchestrates Containers, Not Just Docker

Kubernetes is a container orchestration system made specifically for Docker-based containers.

Beyond Docker. While Docker is a popular container platform, Kubernetes is designed to be container-runtime agnostic. It can orchestrate containers created with Docker, rkt, or other container runtimes that implement the Container Runtime Interface (CRI). This flexibility allows for a broader range of container technologies to be used within a Kubernetes environment.

• Kubernetes uses Linux container technologies for isolation.
• Docker is a platform for packaging, distributing, and running applications.
• rkt is another Linux container engine that emphasizes security and open standards.

Abstraction is key. Kubernetes abstracts away the underlying container technology, allowing developers to focus on application deployment and management rather than the specifics of the container runtime. This abstraction is crucial for portability and interoperability across different environments.

• Kubernetes exposes the whole datacenter as a single deployment platform.
• Developers can deploy apps through Kubernetes without assistance from sysadmins.
• Sysadmins can sleep better by having Kubernetes deal with failed nodes automatically.

Focus on orchestration. Kubernetes' primary role is to orchestrate containers, not to be a container runtime itself. It provides the tools and mechanisms for deploying, scaling, and managing containerized applications, regardless of the underlying container technology. This focus on orchestration is what makes Kubernetes a powerful platform for modern application development.

2. Pods: The Atomic Unit of Kubernetes

A pod is a group of one or more tightly related containers that will always run together on the same worker node and in the same Linux namespace(s).

Pods as logical hosts. Pods are the fundamental building blocks in Kubernetes, representing a group of one or more containers that share resources and are always co-located on the same node. They provide a logical host-like environment for applications, allowing containers to communicate and share data as if they were running on the same machine.

• Pods are like separate logical machines with their own IP, hostname, processes, etc.
• Pods are the basic unit of scaling.
• Pods are ephemeral.

Containers within pods. Containers within a pod share the same network and UTS namespaces, allowing them to communicate through localhost and see the same hostname. They can also share files through volumes. This co-location and resource sharing make pods ideal for running tightly coupled application components.

• Containers in a pod share the same IP address and port space.
• Containers in a pod can communicate through IPC.
• Containers in a pod can share files through volumes.

Organizing containers. Containers should be grouped into pods based on their dependencies and scaling requirements. Tightly coupled processes that need to run together should be placed in the same pod, while independent components should be deployed in separate pods. This approach allows for better resource utilization and more flexible scaling.

• A container shouldn’t run multiple processes.
• A pod shouldn’t contain multiple containers if they don’t need to run on the same machine.

3. Controllers: Ensuring Application Health and Scale

Kubernetes keeps applications healthy by automatically restarting containers.

Controllers as watchdogs. Kubernetes controllers are active components that continuously monitor the state of the cluster and take actions to reconcile the actual state with the desired state. They watch for changes to resources and perform operations to ensure that the system is always running as intended.

• Controllers watch the API server for changes to resources.
• Controllers perform operations to reconcile the actual state with the desired state.
• Controllers don’t communicate with each other directly.

ReplicationControllers and ReplicaSets. These controllers ensure that a specified number of pod replicas are always running. They automatically restart containers that crash and reschedule pods if a node fails. ReplicaSets are the newer version of ReplicationControllers and offer more expressive label selectors.

• ReplicationControllers keep applications healthy by automatically restarting containers.
• ReplicaSets are the newer version of ReplicationControllers.
• ReplicaSets use more expressive label selectors.

DaemonSets and Jobs. DaemonSets ensure that a pod runs on every node in the cluster, while Jobs run pods that perform a single completable task. These controllers provide different ways of managing pods based on specific use cases.

• DaemonSets run a pod on every node.
• Jobs run pods that perform a single completable task.
• CronJobs run jobs periodically or once in the future.

4. Services: Enabling Communication and Discovery

Services represent a static location for a group of one or more pods that all provide the same service.

Services as stable endpoints. Kubernetes Services provide a stable IP address and port through which clients can access a group of pods. This abstraction allows pods to be moved around the cluster without affecting client connectivity.

• Services provide a single static IP address and port for a group of pods.
• Services enable clients to discover and access pods.
• Services load balance connections across multiple pods.

Service types. Kubernetes supports different types of services, including ClusterIP (for internal access), NodePort (for external access through node ports), and LoadBalancer (for external access through a cloud load balancer). Each type provides different ways of exposing services to clients.

• ClusterIP services are only accessible from inside the cluster.
• NodePort services are accessible through a port on all nodes.
• LoadBalancer services are accessible through an external load balancer.

Service discovery. Pods can discover services through environment variables or DNS. This allows applications to find and connect to other services without needing to know their specific IP addresses or locations.

• Services are discoverable through environment variables.
• Services are discoverable through DNS.
• Services can be accessed through their FQDN.

5. Volumes: Managing Data Persistence

Volumes are used to share data between containers.

Volumes as shared storage. Kubernetes volumes provide a way for containers within a pod to share data and access persistent storage. They are defined as part of the pod specification and have the same lifecycle as the pod.

• Volumes are used to share data between containers.
• Volumes are used to access persistent storage.
• Volumes are tied to the lifecycle of a pod.

Volume types. Kubernetes supports various volume types, including emptyDir (for temporary storage), hostPath (for accessing files on the node), gitRepo (for cloning a Git repository), and persistent volumes (for accessing persistent storage). Each type serves a different purpose and provides different levels of data persistence.

• emptyDir volumes are used for temporary storage.
• hostPath volumes are used to access files on the node.
• gitRepo volumes are used to clone a Git repository.
• Persistent volumes are used to access persistent storage.

Persistent storage. Persistent volumes (PVs) and PersistentVolumeClaims (PVCs) decouple pods from the underlying storage technology, allowing developers to request storage without needing to know the specifics of the infrastructure. This abstraction makes applications more portable and easier to manage.

• PersistentVolumes are provisioned by cluster admins.
• PersistentVolumeClaims are used by pods to request storage.
• PersistentVolumes and PersistentVolumeClaims decouple pods from the underlying storage technology.

6. ConfigMaps and Secrets: Externalizing Configuration

ConfigMaps and Secrets are used to pass configuration data and sensitive information like credentials to apps running inside pods.

ConfigMaps for non-sensitive data. ConfigMaps are used to store non-sensitive configuration data, such as application settings, environment variables, and command-line arguments. They allow you to decouple configuration from the application code, making it easier to manage and update.

• ConfigMaps are used to store non-sensitive configuration data.
• ConfigMaps are used to pass command-line arguments to containers.
• ConfigMaps are used to set environment variables for containers.

Secrets for sensitive data. Secrets are used to store sensitive information, such as passwords, API keys, and certificates. They are stored securely and can be mounted into pods as volumes or exposed as environment variables.

• Secrets are used to store sensitive data.
• Secrets are stored securely.
• Secrets can be mounted into pods as volumes.

Decoupling configuration. By using ConfigMaps and Secrets, you can decouple configuration from the application code and container images, making it easier to manage and update your applications. This separation of concerns is crucial for building maintainable and scalable systems.

• ConfigMaps and Secrets decouple configuration from the application code.
• ConfigMaps and Secrets make it easier to manage and update applications.
• ConfigMaps and Secrets make applications more portable.

7. Securing Kubernetes: Authentication, Authorization, and Network Policies

Kubernetes enables you to secure your Kubernetes API server, and by extension the cluster, using authentication and authorization.

Authentication. Kubernetes uses authentication plugins to verify the identity of clients connecting to the API server. These plugins can use various methods, such as client certificates, authentication tokens, or basic HTTP authentication.

• Authentication plugins verify the identity of clients.
• Authentication plugins return the username and groups of the authenticated user.
• ServiceAccounts are used to authenticate pods.

Authorization. Once a client is authenticated, the API server uses authorization plugins to determine whether the client is allowed to perform the requested action on the requested resource. The RBAC (Role-Based Access Control) plugin is the standard authorization plugin in Kubernetes.

• Authorization plugins determine whether a client is allowed to perform an action.
• RBAC is the standard authorization plugin in Kubernetes.
• RBAC uses Roles and RoleBindings to grant permissions.

Network policies. Network policies allow you to control the network traffic between pods, limiting which pods can communicate with each other. This helps to isolate applications and prevent unauthorized access.

• Network policies limit network traffic between pods.
• Network policies use pod selectors and namespace selectors.
• Network policies use ingress and egress rules.

8. Resource Management: Requests, Limits, and QoS

Kubernetes can be configured to automatically scale the number of running replicas of your application.

Resource requests. Resource requests specify the minimum amount of CPU and memory that a container needs to run properly. The Scheduler uses these requests to determine which nodes can accommodate the pod.

• Resource requests affect scheduling.
• Resource requests affect CPU time sharing.
• Resource requests are specified per container.

Resource limits. Resource limits specify the maximum amount of CPU and memory that a container is allowed to consume. If a container exceeds its memory limit, it will be killed by the OOM killer.

• Resource limits prevent containers from using too many resources.
• Resource limits are specified per container.
• Exceeding memory limits results in the container being killed.

QoS classes. Kubernetes categorizes pods into three Quality of Service (QoS) classes: BestEffort, Burstable, and Guaranteed. These classes determine which pods are killed first when memory is low.

• BestEffort pods are killed first.
• Guaranteed pods are killed last.
• Burstable pods are killed in between.

LimitRanges and ResourceQuotas. LimitRanges allow you to set default, min, and max resource requests and limits for pods in a namespace, while ResourceQuotas limit the total amount of resources available in a namespace. These resources help to manage resource consumption and prevent users from overusing cluster resources.

• LimitRanges set default, min, and max resource requests and limits for pods.
• ResourceQuotas limit the total amount of resources available in a namespace.
• ResourceQuotas limit the number of objects that can be created.

9. Autoscaling: Dynamically Adjusting to Demand

Kubernetes can be configured to automatically scale the number of running replicas of your application.

Horizontal pod autoscaling. The Horizontal Pod Autoscaler (HPA) automatically scales the number of pod replicas based on CPU utilization, memory consumption, or other custom metrics. This allows applications to dynamically adjust to changing traffic patterns and resource demands.

• Horizontal pod autoscaling scales the number of pod replicas.
• Horizontal pod autoscaling is based on CPU utilization, memory consumption, or other custom metrics.
• Horizontal pod autoscaling uses the HorizontalPodAutoscaler resource.

Autoscaling process. The autoscaling process involves obtaining pod metrics, calculating the required number of pods, and updating the replicas field of the scaled resource. The HPA controller periodically checks pod metrics and adjusts the number of replicas to meet the target metric value.

• The autoscaling process obtains pod metrics from Heapster.
• The autoscaling process calculates the required number of pods.
• The autoscaling process updates the replicas field of the scaled resource.

Vertical pod autoscaling. While horizontal scaling is the primary method for scaling applications in Kubernetes, vertical pod autoscaling (automatically adjusting resource requests and limits) is also being developed. This feature will allow pods to dynamically adjust their resource consumption based on their needs.

• Vertical pod autoscaling is not yet fully supported.
• Vertical pod autoscaling will automatically configure resource requests.
• Vertical pod autoscaling will modify resource requests while a pod is running.

Cluster autoscaling. The Cluster Autoscaler automatically provisions additional nodes when it detects that pods can't be scheduled due to a lack of resources. It also de-provisions nodes when they are underutilized. This feature allows Kubernetes clusters to dynamically adjust their size based on the needs of the deployed applications.

• Cluster autoscaling scales the number of cluster nodes.
• Cluster autoscaling is based on the needs of the deployed applications.
• Cluster autoscaling is available on cloud providers.

10. Extending Kubernetes: Custom Resources and Service Catalog

Kubernetes can be extended with your own custom objects.

Custom resources. Kubernetes allows you to define your own API objects through CustomResourceDefinitions (CRDs). This enables you to create higher-level abstractions that are specific to your application or domain.

• CustomResourceDefinitions define custom API objects.
• Custom resources can be created, read, updated, and deleted.
• Custom resources can be automated with custom controllers.

Custom controllers. To make custom resources do something, you need to create a custom controller that watches for changes to those resources and performs actions based on those changes. This allows you to automate the deployment and management of your custom resources.

• Custom controllers watch for changes to custom resources.
• Custom controllers perform actions based on changes to custom resources.
• Custom controllers can create other Kubernetes resources.

Service Catalog. The Kubernetes Service Catalog allows you to provision and use external services from within your Kubernetes cluster. It provides a way for users to discover and consume services without needing to know the specifics of the underlying infrastructure.

• The Service Catalog provides a catalog of services.
• The Service Catalog uses service brokers to provision services.
• The Service Catalog uses the OpenServiceBroker API.

11. Developing and Deploying Apps: Best Practices

Applications must expect to be killed and relocated.

Pod lifecycle. Applications running in pods must be designed to handle being killed and relocated at any time. This means they should not rely on a specific IP address or hostname and should be able to gracefully shut down and restart.

• Applications must expect to be killed and relocated.
• Applications must handle the SIGTERM signal.
• Applications must use volumes to persist data across container restarts.

Readiness probes. Readiness probes should be used to signal when a pod is ready to accept connections. This prevents broken client connections when a pod is starting up.

• Readiness probes signal when a pod is ready to accept connections.
• Readiness probes prevent broken client connections when a pod is starting up.
• Readiness probes should check the internals of the app.

Pre-stop hooks. Pre-stop hooks should be used to gracefully shut down a container and prevent broken connections during pod shut-down.

• Pre-stop hooks are executed before a container is terminated.
• Pre-stop hooks prevent broken connections during pod shut-down.
• Pre-stop hooks should wait for a few seconds before terminating.

Manageable container images. Container images should be as small as possible and should only include the necessary files. They should also be properly tagged and use imagePullPolicy wisely.

• Container images should be small.
• Container images should be properly tagged.
• Container images should use imagePullPolicy wisely.

Labels and annotations. Resources should be labeled with multi-dimensional labels and described through annotations. This makes it easier to manage and monitor applications in Kubernetes.

• Resources should be labeled with multi-dimensional labels.
• Resources should be described through annotations.
• Labels and annotations make it easier to manage and monitor applications.

Logging. Applications should log to the standard output and standard error streams. This makes it easier to view logs with the kubectl logs command.

• Applications should log to the standard output and standard error streams.
• Applications should provide information on why the process terminated.
• Centralized logging should be used in production.

Development and testing. Applications should be developed and tested outside of Kubernetes during development. Minikube can be used to test applications in a local Kubernetes environment.

• Applications should be developed outside of Kubernetes during development.
• Minikube can be used to test applications in a local Kubernetes environment.
• Resource manifests should be versioned and auto-deployed.

12. Understanding Kubernetes Internals: How It All Works

Kubernetes is a software system that allows you to easily deploy and manage containerized applications on top of it.

Control Plane components. The Kubernetes Control Plane consists of the etcd distributed storage, the API server, the Scheduler, and the Controller Manager. These components work together to manage the cluster and its resources.

• etcd stores the cluster state.
• The API server provides a REST interface for managing resources.
• The Scheduler assigns pods to nodes.
• The Controller Manager runs controllers that reconcile the actual state with the desired state.

Worker node components. Worker nodes run the Kubelet, the kube-proxy, and the container runtime. These components are responsible for running containers and providing networking for pods.

• The Kubelet manages containers on its node.
• The kube-proxy load-balances network traffic between pods.
• The container runtime runs containers.

Inter-pod networking. Kubernetes provides a flat, NAT-less network for pods, allowing them to communicate with each other regardless of which node they are running on. This is achieved through a Container Network Interface (CNI) plugin.

• Pods communicate through a flat, NAT-less network.
• The network is set up by a CNI plugin.
• Each pod gets its own IP address.

Service implementation. Kubernetes Services are implemented through the kube-proxy, which uses iptables rules to redirect traffic to the pods backing the service. This allows clients to connect to services without needing to know the specific IP addresses of the pods.

• The kube-proxy implements services.
• The kube-proxy uses iptables rules to redirect traffic.
• Services provide load balancing across multiple pods.

High availability. Kubernetes components can be run in multiple instances to ensure high availability. This includes running multiple instances of etcd, the API server, the Scheduler, and the Controller Manager.

• Kubernetes components can be run in multiple instances.
• etcd uses the RAFT consensus algorithm.
• Leader election is used to ensure only one instance of the Scheduler and Controller Manager is active.

Last updated: January 25, 2025