# Kubernetes Penetration Testing & Configuration Review
# **Part 1.1 – The Problem That Changed Infrastructure**
> *"Every revolutionary technology begins as an answer to a question nobody had solved before. Kubernetes was not invented because engineers wanted another orchestration platform. It was invented because the previous generation of infrastructure could no longer keep up."*
* * *
## Before Kubernetes, There Was a Much Simpler Problem
Every software engineer has heard it.
> **"It works on my machine."**
Few sentences have caused more delayed deployments, production outages, and frustrated engineering teams than those five words.
The developer insists the application works perfectly. The QA engineer reproduces crashes consistently. Operations claims nothing changed on the servers. Security discovers production is running an entirely different OpenSSL version than development.
Nobody is lying. Everyone is correct. The application is identical - the **environment isn't**.
Long before Kubernetes, infrastructure engineers weren't trying to orchestrate thousands of containers. They were trying to solve a much simpler but surprisingly difficult problem:
> **How do we make software behave the same everywhere?**
That single question drove nearly two decades of innovation. Each new technology solved one engineering problem. Each solution introduced another. By the end of that journey, Kubernetes became inevitable.
Before we learn how attackers exploit Kubernetes, we first need to understand **why Kubernetes exists at all.**
* * *
## Software Was Never the Problem
Imagine it's the early 2000s. You're the only systems administrator at a growing company. The business has one internal payroll application. Life is simple.
```text
Users
│
▼
+--------------------------+
| Payroll Application |
+--------------------------+
| Linux Operating System |
+--------------------------+
| Physical Server |
+--------------------------+
```
One application. One OS. One physical server. The application performs well. Users are happy. There are no complicated deployment pipelines, no CI/CD, no microservices, and certainly no Kubernetes clusters.
Then the business grows. Human Resources wants an employee portal. Marketing needs a company website. Finance requests an ERP. Developers ask for Git. Operations wants Jenkins. Security wants centralized logging.
The obvious solution? Buy another server. And another. And another. Soon, your server room looks something like this:
```text
+------------+ +------------+ +------------+
| Payroll | | Website | | Database |
| Server | | Server | | Server |
+------------+ +------------+ +------------+
+------------+ +------------+ +------------+
| Git | | Mail | | Jenkins |
| Server | | Server | | Server |
+------------+ +------------+ +------------+
```
At first glance, everything seems reasonable. Every application has its own machine. Every failure is isolated. Nobody worries about resource contention. Then someone opens the monitoring dashboard.
## The Invisible Waste
Most servers weren't busy. They were **waiting**. Typical utilization looked something like this:
| Resource | Typical Utilization |
| --- | --- |
| CPU | 5–15% |
| Memory | 20–30% |
| Disk | Mostly Idle |
| Network | Minimal |
The organization had invested heavily in enterprise hardware. Yet those expensive machines spent most of their lives doing almost nothing. Still, they consumed:
* Electricity
* Cooling
* Rack space
* Maintenance
* Hardware support contracts
* Backup resources
* Monitoring licenses
The problem wasn't insufficient computing power. It was **poor resource utilization**. The hardware industry had advanced faster than the software running on it.
> ### Engineering Insight
>
> **Infrastructure engineers weren't trying to build faster computers.**
>
> **They were trying to use existing computers more efficiently.**
>
> **This distinction would shape the next twenty years of infrastructure evolution.**
* * *
## Hardware Was Becoming Too Good
Moore's Law had delivered increasingly powerful processors. Memory capacities were growing rapidly. Storage was becoming cheaper. Network speeds continued improving. Applications, however, rarely consumed all those resources.
An accounting application didn't need an entire dual-processor server. A web application didn't require 64 GB of RAM. A DNS server certainly didn't need enterprise-grade hardware.
Organizations weren't limited by computing power anymore. They were limited by **how computing resources were allocated**. The challenge had shifted.
Instead of asking:
> *"How do we build bigger servers?"*
Engineers started asking:
> *"How do we safely run multiple applications on the same server?"*
* * *
## The First Major Revolution: Virtualization
The answer came from a simple but revolutionary idea. What if one physical computer could pretend to be many?
Rather than installing a single operating system directly onto hardware, engineers introduced another layer between them. This layer became known as the **hypervisor**.
```text
Physical Server
+---------------------------------------+
Hypervisor
+------------+ +------------+ +------------+
| VM 1 | | VM 2 | | VM 3 |
| Linux | | Windows | | Linux |
+------------+ +------------+ +------------+
Applications inside each VM
```
Instead of purchasing ten physical servers, organizations could consolidate workloads onto a handful of powerful machines. This changed everything. Infrastructure became dramatically more efficient. Applications became easier to migrate. Disaster recovery improved. Provisioning new environments no longer required waiting weeks for hardware procurement.
Virtualization became one of the most important engineering breakthroughs of modern computing.
* * *
## Why Virtual Machines Were So Successful
Virtualization solved several long-standing operational problems simultaneously.
Organizations gained:
* Better hardware utilization.
* Strong isolation between workloads.
* Simplified backup and recovery.
* Faster provisioning.
* Reduced infrastructure costs.
* Improved disaster recovery.
For more than a decade, virtual machines became the default deployment model across enterprise infrastructure. Many organizations still rely heavily on them today. But engineering is a continuous cycle. Solving one problem often reveals another.
* * *
## Every Virtual Machine Carried an Entire Operating System
Although virtual machines shared hardware efficiently, they still duplicated software.
Every VM contained:
* Its own operating system.
* Its own kernel.
* Its own package manager.
* Its own system libraries.
* Its own init system.
* Its own update cycle.
Conceptually, each virtual machine looked like this:
```text
+------------------------+
| Application |
+------------------------+
| Runtime Libraries |
+------------------------+
| Guest Operating System |
+------------------------+
| Guest Kernel |
+------------------------+
| Virtual Hardware |
+------------------------+
```
Now imagine hosting fifty applications. You no longer managed fifty applications.
You managed:
* Fifty operating systems.
* Fifty kernels.
* Fifty patch cycles.
* Fifty security baselines.
* Fifty monitoring agents.
Virtualization had solved hardware utilization. It had not solved software duplication.
* * *
> ### Security Perspective
>
> Virtual machines significantly improved isolation by placing a hypervisor boundary between workloads. A compromise inside one VM generally remained confined to that guest operating system.
>
> However, every additional VM also introduced another operating system that required hardening, patch management, endpoint protection, vulnerability scanning, and compliance monitoring. The operational overhead grew alongside the infrastructure.
* * *
## The "Works on My Machine" Problem Never Went Away
Virtual machines improved deployment consistency. But developers continued facing another challenge. Applications depended on specific:
* Library versions.
* Runtime environments.
* Package dependencies.
* Environment variables.
* Configuration files.
Consider a simple web application. On the developer's laptop:
```plaintext
Python 3.11
OpenSSL 3.x
glibc version A
```
In production:
```plaintext
Python 3.8
Older OpenSSL
Different glibc
```
Same source code. Different environment. Unexpected behavior.
The infamous phrase returned.
> **"It works on my machine."**
The problem wasn't the application. The problem was everything surrounding it.
Developers needed a way to package not only their code, but also the environment required to execute it consistently. The industry had solved hardware efficiency.
Now it needed to solve **application portability**.
* * *
## A Quiet Revolution Inside the Linux Kernel
Interestingly, the next breakthrough didn't begin with Docker. It began years earlier inside the Linux kernel. Linux developers had been building features that, when combined, would eventually redefine application deployment.
Among the most important were:
* **chroot** – restricting a process to a specific portion of the filesystem.
* **Namespaces** – providing isolated views of processes, networking, mounts, hostnames, users, and IPC.
* **Control Groups (cgroups)** – limiting and accounting for CPU, memory, and I/O usage.
* **Linux Capabilities** – decomposing the all-powerful root privilege into finer-grained permissions.
At the time, these appeared to be independent operating system features. Few realized they would eventually become the foundation of modern containerization. Engineers began asking another deceptively simple question:
> **Do applications really need their own operating system?**
That question would lead to one of the most significant transformations in modern infrastructure and ultimately pave the way for Docker, Kubernetes, and the cloud-native ecosystem we know today.
* * *
## Meetup Highlight
Abhishek Pal deliberately began the BreachForce session by introducing the fundamentals—what containers are, why Kubernetes exists, and the security principles that underpin cloud-native environments—before moving into penetration testing techniques and tooling. This progression reinforces an important lesson: meaningful security assessments begin with understanding the architecture and operational goals of the technology being assessed, not with running scanners.
* * *
## Looking Ahead
At this point in history, infrastructure engineers had successfully improved hardware utilization through virtualization, but they were still struggling with software portability and operational complexity.
The next chapter in this journey would challenge a long-held assumption:
> **What if applications didn't need an entire operating system at all?**
In **Part 1.2 – Containers Changed Software Delivery**, we'll explore how Linux namespaces, cgroups, and LXC laid the foundation for containers, how Docker transformed them into a developer-friendly platform, and why solving software packaging created an entirely new class of operational and security challenges that Kubernetes would later address.
* * *
# Part 1.2 – Containers Changed Software Delivery
> *"Virtualization taught computers how to share hardware. Containers taught applications how to share an operating system."*
* * *
## The Assumption Nobody Questioned
By the late 2000s, virtualization had become the backbone of enterprise infrastructure. Organizations had solved one of their biggest operational challenges—hardware utilization. Servers were no longer sitting idle. Provisioning a new environment no longer required purchasing another physical machine. Infrastructure teams could spin up virtual machines in minutes instead of weeks.
It looked like the problem had finally been solved. Except…Developers were still complaining.
* * *
## "Works on My Machine" Was Still Alive
Imagine two developers working on the same project.
Developer A runs: `python app.py` Everything works perfectly. Developer B pulls the exact same repository. Runs the exact same command. The application crashes immediately.
Why?
Because software isn't just source code. Software depends on an entire execution environment. Different machines may have different:
* Python versions
* Java versions
* OpenSSL libraries
* glibc implementations
* Package versions
* Environment variables
* Filesystem layouts
* Kernel features
The code hadn't changed. The environment had. This became known as **dependency hell**. The application itself wasn't broken. The environment surrounding it was.
* * *
## Packaging Software Was Harder Than Writing It
Suppose you build a simple web application. Your laptop contains:
```plaintext
Ubuntu 24.04
Python 3.12
OpenSSL 3.x
Nginx
Gunicorn
Required Python Packages
```
Now the application is deployed to production. Production runs:
```plaintext
Ubuntu 20.04
Python 3.8
Older OpenSSL
Different Package Versions
```
Nobody intentionally introduced a bug. Yet production behaves differently. Developers found themselves documenting increasingly complicated installation guides.
```plaintext
Install Python
Upgrade OpenSSL
Install pip
Install these libraries
Modify this config
Export these variables
Restart service
```
Sometimes these instructions became longer than the application itself.
* * *
### Engineering Insight
> Infrastructure teams realized they weren't deploying software. They were deploying:
>
> * software
>
> * dependencies
>
> * runtimes
>
> * configuration
>
> * operating system expectations
>
>
> The application was only a small part of the deployment. Everything around it mattered equally.
* * *
## Linux Already Had the Building Blocks
Interestingly, the solution didn't originate from Docker. It originated from the Linux kernel. For years, Linux developers had been building mechanisms that isolated processes from one another. Individually they looked unrelated. Collectively they became the foundation of containers.
### chroot — The First Taste of Isolation
Introduced in Unix in 1979, `chroot` allowed a process to believe a specific directory was the root of the filesystem.
Instead of seeing:
```plaintext
/
├── etc
├── home
├── var
├── usr
```
A process could instead see:
```plaintext
/
├── app
├── logs
├── config
```
Everything outside became invisible. Although this wasn't a security boundary by modern standards, it introduced an important concept:
> Different processes don't necessarily need the same view of the operating system.
### Namespaces
The next breakthrough came through **Linux Namespaces**. Namespaces isolate different operating system resources.
For example:
| Namespace | Isolates |
| --- | --- |
| PID | Running Processes |
| NET | Network Interfaces |
| MNT | Mount Points |
| IPC | Inter-Process Communication |
| UTS | Hostname |
| USER | Users & Groups |
| CGROUP | cgroup hierarchy |
Imagine two containers.
Container A executes:
```bash
ps aux
```
Output:
```plaintext
PID 1
PID 20
PID 45
```
Container B executes the same command.
```plaintext
PID 1
PID 6
PID 8
```
Both believe they own PID 1. Neither sees the other's processes.
Reality?
Both are running on the same Linux kernel. Namespaces simply give each process its own perspective of reality.
### Control Groups (cgroups)
Isolation alone wasn't sufficient. Imagine one application consuming every CPU core. Other applications would starve. Linux introduced **Control Groups**, commonly called **cgroups**, to solve resource management.
cgroups allow administrators to control:
* CPU usage
* Memory consumption
* Disk I/O
* Network resources
* Process counts
Instead of trusting applications to behave nicely, the kernel could enforce resource limits.
For example:
```plaintext
Container A
CPU: 2 cores
Memory: 2 GB
Disk: 20 GB
```
Even if the application attempted to consume more resources, the kernel enforced those limits. Today, Kubernetes relies heavily on cgroups to implement resource requests and limits.
### Linux Capabilities
For decades Linux had two privilege levels.
```plaintext
Root
Non-root
```
Nothing in between. This model became problematic. An application might need to bind to port 80 but shouldn't receive unrestricted administrative access.
Linux introduced **Capabilities**, splitting root into smaller privileges.
Examples include:
```plaintext
CAP_NET_ADMIN
CAP_SYS_ADMIN
CAP_SYS_PTRACE
CAP_NET_RAW
```
Instead of granting complete root access, applications could receive only the capabilities they genuinely required. This seemingly small change later became one of Kubernetes' most important security mechanisms.
* * *
> ### Security Perspective
>
> Modern Kubernetes hardening often recommends dropping unnecessary Linux capabilities. This recommendation isn't arbitrary. Every additional capability expands what a compromised container can do after exploitation. During the meetup, Abhishek Pal emphasized running workloads with the least privileges possible and avoiding unnecessary privilege escalation as a core hardening principle.
* * *
## Linux Containers (LXC)
By combining:
* chroot
* Namespaces
* cgroups
* Capabilities
Linux developers created **LXC (Linux Containers)**. For the first time, applications could execute inside isolated environments without requiring an entirely separate operating system.
Conceptually, the architecture changed from this:
```plaintext
Application
Libraries
Guest OS
Kernel
```
to this:
```plaintext
Application
Libraries
Runtime
-------------------
Shared Linux Kernel
```
The implications were enormous. Containers could:
* start in milliseconds
* consume megabytes instead of gigabytes
* achieve significantly higher density than virtual machines
Instead of running ten virtual machines on a server, organizations could run hundreds of containers. Infrastructure efficiency improved dramatically.
* * *
### But LXC Had a Problem
Although technically impressive, LXC wasn't particularly developer-friendly.
Creating containers required understanding:
* Linux namespaces
* cgroups
* filesystem isolation
* networking
* kernel configuration
There was no standardized packaging format. Sharing applications remained cumbersome. Image distribution was inconsistent. For most software developers, Linux containers remained an expert-only technology. The underlying engineering existed. The developer experience did not.
* * *
## Docker Changed Everything
In 2013, Docker entered the industry. Contrary to popular belief, **Docker did not invent containers.** Containers already existed. Docker solved a different problem. It made containers accessible.
As highlighted in Abhishek Pal's training deck, a container can be viewed as **"Code + Configuration + Runtime"** packaged into a portable artifact that behaves consistently regardless of where it is executed. Docker standardized how those artifacts were built, shared, and executed.
Instead of manually configuring Linux namespaces and cgroups, developers simply wrote:
```dockerfile
FROM python:3.12
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python","app.py"]
```
Then executed:
```bash
docker build
docker run
docker push
```
Containers suddenly became:
* reproducible
* portable
* versioned
* easy to distribute
Developers loved Docker. Operations teams loved Docker. Organizations rapidly began containerizing everything. It was one of the fastest infrastructure adoptions in modern computing history.
* * *
## Docker Solved Packaging
But not Operations. This distinction is one of the most important concepts in cloud-native engineering.
Docker answered:
> **"How do I package and run an application consistently?"**
It did **not** answer:
* Where should the application run?
* What happens if the server crashes?
* How do containers discover one another?
* How should traffic be load balanced?
* How do I deploy updates without downtime?
* How do I automatically recover failed workloads?
* How do I scale from 10 containers to 10,000?
Those weren't Docker's goals. They were infrastructure problems. And solving them would require an entirely new way of thinking.
* * *
## Meetup Highlight
One of the recurring themes throughout Abhishek Pal's session was that understanding Kubernetes security begins with understanding the platform itself. Before discussing attack techniques, configuration reviews, or runtime detection, the session established what containers are, why Kubernetes exists, and how modern cloud-native environments are fundamentally different from traditional infrastructure.
* * *
## Looking Ahead
By the mid-2010s, organizations had solved two major engineering problems. Virtualization had solved **hardware utilization**. Docker had solved **application portability**. Then success created a brand-new challenge. Companies weren't running five containers anymore. They were running **thousands**.
Scheduling them. Recovering them. Updating them. Networking them. Securing them.
Docker had never promised to solve those problems. Someone else already had.
Inside Google, engineers had spent nearly a decade building an internal platform capable of orchestrating containers across one of the largest infrastructures on Earth.
They called it **Borg**.
In **Part 1.3 – The Birth of Kubernetes**, we'll explore how Google's experience with Borg shaped Kubernetes, why orchestration became inevitable, and how every new capability introduced an entirely new class of security challenges that continue to define Kubernetes penetration testing today.
* * *
# Part 1.3 – The Birth of Kubernetes
> *"Docker solved how to package software. It never promised to operate an entire data center. That challenge had already been solved years earlier inside Google."*
* * *
## Success Created a Bigger Problem
By 2014, containers had become the hottest topic in infrastructure engineering. Developers loved them. Operations teams loved them. Management loved the cost savings. Every new application was containerized.
Soon, organizations weren't running five containers. They were running thousands. Consider a large streaming platform. Instead of deploying a single application, it now consisted of hundreds of independent services.
* Authentication
* User Profiles
* Recommendation Engine
* Search
* Billing
* Notifications
* Media Processing
* Logging
* Monitoring
* Analytics
Each service ran inside multiple containers. A modern production environment quickly resembled something like this.
```text
Users
│
V
Load Balancer
│
┌────────────┐
│ │ │
Authentication Search Billing
│ │ │
30 Pods 50 Pods 40 Pods
│ │ │
Hundreds of Containers
```
At first glance this looked wonderful. Containers were lightweight. Portable. Fast. Easy to deploy. Then reality struck.
* * *
## The Questions Nobody Could Answer
Infrastructure teams began asking questions Docker was never designed to solve. Imagine one server suddenly crashes. Now ask yourself:
* Which containers were running there?
* Who restarts them?
* Where should they restart?
* Which machine has enough CPU?
* Which machine has enough memory?
* How should traffic be redirected?
* What if another machine is already overloaded?
* How do users avoid downtime?
Now multiply those questions across **10,000 containers**. Docker simply responded:
> "That's not my job."
And it wasn't. Docker was a **container runtime**. Not an orchestration platform.
* * *
### Engineering Insight
> Every technology has boundaries. Docker's responsibility ends the moment a container starts. Everything after that — availability, scheduling, networking, scaling, recovery — is an infrastructure problem.
>
> Those problems required an entirely different class of software.
* * *
## Google Had Already Solved This Problem
Here's something many engineers don't realize. Kubernetes wasn't Google's first attempt at container orchestration. It wasn't even Google's second.
Years before Kubernetes existed, Google had already been operating one of the world's largest computing infrastructures.Every Google Search. Every Gmail request. Every YouTube video. Every Maps query. All required enormous distributed systems. Managing millions of workloads manually was impossible. Google needed automation. The result was an internal platform known as **Borg**.
### Borg: The Invisible Giant
Although Borg was never open sourced, it became one of the most influential systems ever built.
Instead of treating servers as individual machines, Borg viewed the entire data center as one enormous pool of compute resources.
Applications no longer requested:
> "Run this application on Server 42."
Instead they declared:
> "I need two CPU cores, four gigabytes of memory, and three replicas."
The scheduler decided where those workloads should execute. This was a revolutionary shift. Infrastructure stopped becoming imperative. It became declarative. Engineers described **what** they wanted. The platform determined **how** to achieve it.
This philosophy remains at the heart of Kubernetes today.
### Desired State
Imagine writing:
```yaml
replicas: 3
```
This single line says something profound. It doesn't tell Kubernetes to create three Pods. It tells Kubernetes:
> "The correct state of my application is three running Pods."
Notice the wording.
Not:
> Create three Pods.
Instead:
> Ensure three Pods always exist.
Those are fundamentally different instructions.
If one Pod crashes... Kubernetes creates another. If an administrator deletes one...Kubernetes creates another. If a node disappears...Kubernetes schedules another elsewhere.
This continuous reconciliation between **desired state** and **actual state** is one of Kubernetes' defining characteristics.
### From Borg to Omega to Kubernetes
Google continued evolving Borg internally. One of its successors was **Omega**, which experimented with more flexible scheduling and cluster management. The operational lessons learned from both systems eventually inspired Kubernetes, which Google open-sourced in 2014 before donating it to the **Cloud Native Computing Foundation (CNCF)**.
Rather than exposing Google's internal implementation directly, Kubernetes distilled the core orchestration principles into an extensible, open-source platform. Many concepts familiar to Kubernetes users today trace their roots back to Borg:
* Declarative infrastructure
* Scheduling
* Controllers
* Labels
* Self-healing
* Desired state reconciliation
## Kubernetes Didn't Replace Docker
One misconception still appears frequently.
People say:
> Kubernetes replaced Docker.
It didn't. Docker and Kubernetes solve different problems. Think of it this way.
Docker asks:
> "How do I build and run one container?"
Kubernetes asks:
> "How do I operate 100,000 containers reliably?"
Those are completely different engineering questions.
For years, Kubernetes actually used Docker as one of its supported container runtimes before moving to the standardized **Container Runtime Interface (CRI)**.
Docker remained responsible for packaging and running containers. Kubernetes became responsible for orchestrating them.
## Kubernetes Solved Operational Problems
Every major Kubernetes component exists because it solved a specific operational problem.
| Operational Problem | Kubernetes Solution |
| --- | --- |
| Where should a workload run? | Scheduler |
| How do workloads communicate? | Services |
| How do we expose applications? | Ingress |
| How do we recover from failures? | Controllers |
| How do we maintain replicas? | ReplicaSets |
| How do we deploy updates safely? | Deployments |
| Where do configurations live? | ConfigMaps |
| How do applications access secrets? | Secrets |
| How do we persist storage? | Persistent Volumes |
Notice something interesting. None of these components exist because engineers wanted more complexity. Every one of them exists because someone encountered a real operational problem.
* * *
## Meetup Highlight
Abhishek Pal emphasized this same progression during the meetup. Before discussing offensive security, he introduced the foundational Kubernetes resources that administrators interact with daily - **Namespaces, Deployments, ReplicaSets, ConfigMaps, Secrets, Persistent Volumes, Persistent Volume Claims, and Ingress**\-highlighting that these objects are not merely configuration constructs but integral parts of how Kubernetes operates and, consequently, where security assessments must focus.
* * *
## Every Solution Introduced a New Attack Surface
As Kubernetes solved operational challenges, it also expanded the security landscape. Consider a few examples.
### The API Server
Operational purpose: A centralized interface for managing the cluster.
Security implication: A high-value target. Compromising it may allow attackers to control the entire cluster.
### etcd
Operational purpose: Store the cluster's desired state.
Security implication: Contains sensitive configuration, Secrets (if encryption at rest is not enabled), and cluster metadata.
### Service Accounts
Operational purpose: Allow workloads to authenticate to the Kubernetes API.
Security implication: Compromised Service Account tokens can become stepping stones for privilege escalation if RBAC permissions are overly broad.
### ConfigMaps and Secrets
Operational purpose: Separate configuration from application code.
Security implication: Mismanaged secrets, particularly those stored insecurely or exposed through version control, can lead to credential theft and unauthorized access.
### Networking
Operational purpose: Enable communication between workloads.
Security implication: Without NetworkPolicies, lateral movement inside the cluster becomes significantly easier.
* * *
### Engineering Insight
> Every Kubernetes feature exists to reduce operational complexity. Ironically, every feature also expands the attack surface. Security is therefore not about disabling Kubernetes features. It is about understanding the trust relationships those features introduce and ensuring they are configured appropriately.
* * *
## Kubernetes Security Is Different
Traditional penetration testing often focuses on exploiting software vulnerabilities. Kubernetes assessments require a broader perspective. A perfectly patched cluster may still be vulnerable because of:
* Overly permissive RBAC.
* Privileged containers.
* Exposed dashboards.
* Misconfigured Service Accounts.
* Weak NetworkPolicies.
* Plaintext secrets.
* Insecure Infrastructure as Code.
During the BreachForce meetup, Abhishek stressed that assessments should evaluate access controls, network segmentation, application exposure, operating system hardening, and Kubernetes configuration together, often beginning from a **black-box perspective** where the assessor has network access but no privileged Kubernetes credentials.
Understanding Kubernetes therefore means understanding **systems**, not just software.
* * *
## The Journey Begins
We began this chapter with a simple question:
> **Why does Kubernetes exist?**
The answer is surprisingly elegant.
Bare metal solved computing. Virtualization solved hardware utilization. Containers solved software portability. Docker solved packaging. Google solved orchestration. Kubernetes brought those orchestration principles to everyone.
But every engineering solution came with a price. Every API. Every controller. Every Service Account. Every Secret. Every admission controller. Every networking abstraction. Every automation feature. All became potential attack surfaces.
Understanding those surfaces is what separates a Kubernetes administrator from a Kubernetes security professional. And that is precisely where our journey now begins.
* * *
## What's Next?
In **Chapter 2 – Kubernetes Architecture Through a Security Lens**, we'll leave the historical narrative behind and step inside a live Kubernetes cluster.
Rather than memorizing component definitions, we'll answer a more important question:
> **Why does each Kubernetes component exist, how does it work, and what happens if an attacker compromises it?**
We'll dissect the Control Plane, Worker Nodes, API Server, etcd, kubelet, Scheduler, Controller Manager, Pods, Services, Namespaces, ConfigMaps, Secrets, and Persistent Volumes - not as isolated concepts, but as interconnected systems whose trust relationships define the security posture of an entire cluster.
* * *
# Chapter 2 - Kubernetes Architecture Through a Security Lens
> "The easiest way to compromise Kubernetes is not by exploiting software. It is by exploiting trust. Every Kubernetes component trusts another component for a reason. Understanding those trust relationships is the first step toward understanding Kubernetes security."
* * *
## We Finally Have Kubernetes…
Now What? At the end of the previous chapter, we answered a fundamental engineering question. Docker solved application packaging. Google Borg solved orchestration. Kubernetes brought orchestration to everyone.
But saying
> "Kubernetes orchestrates containers"
is a little like saying
> "An airplane flies."
Technically true. Practically useless. To secure Kubernetes, we first need to understand **what Kubernetes actually does every second.**
* * *
# Let's Build Kubernetes Together
Instead of memorizing components, imagine you're one of the engineers designing Kubernetes. Your manager gives you a problem.
> We have hundreds of servers.
Thousands of containers. Millions of users. Make all of this reliable. Where do you begin?
## Problem #1 - How does a user tell Kubernetes what they want?
Suppose an engineer wants to deploy an application. Without Kubernetes they might SSH into a server and execute - `docker run nginx`
That works. Until there are
* 500 servers
* 8 administrators
* multiple cloud regions
* automatic scaling
SSH doesn't scale. There needs to be **one place** where everyone talks to Kubernetes.
## The API Server Is Born
The Kubernetes API Server is exactly that. It is the front door of the cluster. Everything communicates through it. Users, kubectl, CI/CD pipelines, Controllers, Schedulers, Operators, Admission Controllers, Everything.
```text
User
│
kubectl apply
│
V
+-------------------+
| Kubernetes API |
| Server |
+-------------------+
```
Notice something. Nobody talks directly to etcd. Nobody talks directly to the scheduler. Nobody talks directly to worker nodes. Everything begins with the API Server. This architectural decision dramatically simplifies cluster management.
It also creates one of the most valuable attack targets.
* * *
## Engineering Insight
> Google deliberately centralized cluster management. Instead of every component exposing its own management interface, Kubernetes presents **one consistent API**. Every tool in the Kubernetes ecosystem is ultimately just another client of that API. This consistency is one of Kubernetes' greatest strengths.
* * *
## Security Perspective
If attackers obtain sufficient privileges against the API Server, they effectively gain control over the cluster. This explains why API Servers are among the highest-priority targets during Kubernetes assessments.
Misconfigurations commonly include:
* Anonymous authentication
* Weak RBAC
* Exposed API endpoints
* Overly permissive Service Accounts
* Stolen kubeconfig files
During the BreachForce meetup, Abhishek highlighted that exposing the Kubernetes API server to the Internet remains one of the most common initial attack vectors encountered during assessments.
* * *
## Problem #2 - Where Does Kubernetes Remember Anything?
Imagine deploying
```yaml
replicas: 5
```
Tomorrow… every server loses power. Everything shuts down. Electricity returns. How does Kubernetes know you wanted five Pods? Someone needs to remember.
### etcd
The answer is **etcd**. etcd is a distributed key-value database. Think of it as Kubernetes' memory.
```text
API Server
│
V
+--------+
| etcd |
+--------+
```
Everything Kubernetes knows is stored here. Examples include
* Pods
* Nodes
* Deployments
* Secrets
* ConfigMaps
* Service Accounts
* RBAC objects
* Namespaces
If Kubernetes forgets… it cannot recover.
* * *
## Engineering Insight
> Instead of storing state inside running services, Kubernetes stores desired state in etcd. Controllers continuously compare
>
> ```plaintext
> Reality
>
> vs
>
> Desired State
> ```
>
> This idea comes directly from Google's Borg.
* * *
## Security Perspective
`etcd` is one of the most sensitive systems inside the cluster. Compromise etcd…and you may compromise everything.
This is why Kubernetes recommends
* TLS
* Authentication
* Encryption at Rest
* Restricted Network Access
Abhishek also stressed during the meetup that etcd should only trust certificates assigned to API servers and should never be left exposed through insecure configurations.
* * *
## Problem #3 - Who Decides Where Containers Run?
Suppose there are 100 worker nodes.
Some have
* 90% CPU
Others
* 10% CPU
Where should the next Pod go? Random choice? Obviously not.
### Scheduler
The Scheduler solves placement. It asks questions like Does the node have enough memory? Enough CPU? Correct labels? Correct taints? Required affinity? Required storage? If yes… schedule the Pod.
```text
Deployment
│
V
Scheduler
│
┌─────────────────┐
│ │
Worker A Worker B
```
Notice, the Scheduler doesn't create Pods. It chooses **where** Pods should execute.
* * *
## Security Perspective
Although attackers rarely target the Scheduler directly, manipulating scheduling constraints can influence workload placement. Security assessments therefore review node selectors, taints, tolerations, affinity rules, and admission policies to ensure workloads cannot be steered onto inappropriate nodes.
* * *
## Problem #4 - Pods Crash
Applications fail. Containers exit. Nodes disappear. Cloud instances terminate. Who notices? Who fixes it? Should an administrator receive a phone call every five minutes? Google answered - "No."
### Controller Manager
The Controller Manager is Kubernetes' autopilot. It constantly asks
> Is reality still what the user requested?
Example:
```plaintext
Desired: 3 Pods Actual: 2 Pods → Controller: Create another Pod
Desired: 5 Replicas Actual: 7 Replicas → Controller: Delete two
```
This process never stops. Kubernetes is always reconciling reality.
* * *
## Engineering Insight
> This is one of the biggest conceptual differences between traditional infrastructure and Kubernetes. Traditional administration is **imperative**. You tell the system what to do.
>
> Kubernetes is **declarative**. You tell Kubernetes what the end state should look like. Controllers continuously work toward that state.
* * *
### Problem #5 - Where Do Applications Actually Run?
Many beginners answer
> Docker.
That's only partly correct. Applications run inside Containers. Containers run inside Pods. Pods run on Worker Nodes.
### Why Pods?
This confuses nearly everyone initially. Why not just schedule containers? Because Kubernetes needed something slightly more flexible.
A Pod is the smallest deployable unit in Kubernetes. It may contain
* one container
or
* multiple tightly coupled containers
sharing
* networking
* storage
* lifecycle
For example:
```text
+----------------------+
POD
+----------+ +----------+
| App | | Sidecar |
+----------+ +----------+
```
This design enables patterns such as:
* Service Mesh proxies
* Log collectors
* Monitoring agents
without tightly coupling them into a single container image.
* * *
## Meetup Highlight
Abhishek introduced core Kubernetes resources - including Pods, Deployments, ReplicaSets, ConfigMaps, Secrets, Namespaces, Persistent Volumes, and Ingress - before discussing offensive techniques. This progression reinforces that penetration testing begins with understanding the platform's operational model rather than jumping directly into exploitation.
## Why This Matters for Security
Attackers rarely compromise Kubernetes all at once. They compromise a Pod. Then, they discover a Service Account. Then, they query the API Server. Then, they enumerate RBAC. Then, they move laterally. Every attack starts somewhere. Understanding the architecture tells you where that "somewhere" is.
* * *
![](https://cdn.hashnode.com/uploads/covers/65b618fc35b9d2122652b543/ea8615ca-18e3-4003-84c4-bb615d406686.png align="center")
In the next section of Chapter 2, we'll move from the control plane to the worker nodes and explore:
* kubelet
* kube-proxy
* Container Runtime
* Nodes
* Namespaces
* Services
* Ingress
* ConfigMaps
* Secrets
* Persistent Volumes
Most importantly, we'll answer a question that lies at the heart of Kubernetes penetration testing:
> **If you compromise a Pod, what can you actually reach next?**
That question will naturally lead us into the Kubernetes threat model and the offensive techniques demonstrated during Abhishek Pal's session.
* * *
## Worker Nodes – Where Applications Come Alive
> *"If the Control Plane is the brain of Kubernetes, the Worker Nodes are its hands. They perform the actual work. Unfortunately, they're also where attackers usually gain their first foothold."*
## From Desired State to Reality
Let's revisit something we learned earlier. Suppose a developer executes:
```bash
kubectl apply -f payment-service.yaml
```
The API Server accepts the request. The Scheduler decides **where** the workload should execute. The Controller Manager ensures the desired state exists. Everything sounds good.
But...
Who actually starts the container? Someone has to pull the image. Create the network. Mount storage. Launch the process. Monitor it. Restart it if it crashes.
That responsibility belongs to the **Worker Node**.
## Anatomy of a Worker Node
Every Kubernetes cluster consists of one or more worker nodes. Conceptually, each node looks like this:
```text
Worker Node
+-------------------------------------------+
kubelet
kube-proxy
Container Runtime
---------------------------------------------
Pods
---------------------------------------------
Linux Operating System
---------------------------------------------
Physical / Virtual Machine
```
Unlike the Control Plane, which makes decisions, Worker Nodes **execute** those decisions. Think of them as factory workers. Management decides what should happen. Workers actually build the product.
* * *
## Problem #6 - How does the Control Plane talk to Worker Nodes?
Imagine the Scheduler decides:
> Run this Pod on Worker Node 5.
How does Worker Node 5 receive that instruction? Should the API Server SSH into every server? Absolutely not.
That would be:
* slow
* difficult to secure
* impossible to scale
Instead... Google designed something much smarter.
### kubelet
Every Worker Node runs a small agent called the **kubelet**. The kubelet continuously communicates with the API Server. Think of it as the cluster's local representative.
```text
API Server
^
│
Secure HTTPS
│
V
kubelet
│
Starts Containers
```
The kubelet constantly asks:
> "Do you have any work for me?"
The API Server responds:
> "Yes."
> "Run these Pods."
The kubelet then:
* pulls container images
* mounts storage
* configures networking
* starts containers
* reports status back
It never decides **what** should run. It simply executes instructions.
* * *
## Engineering Insight
> Notice the design pattern. The kubelet is intentionally **not** a scheduler. It doesn't choose Pods. It doesn't create Deployments. It doesn't modify cluster state. It simply enforces the desired state assigned to its node. This separation of responsibility keeps Kubernetes modular.
* * *
## Security Perspective
From an attacker's point of view... The kubelet is extremely interesting.
Historically, misconfigured kubelets exposed APIs that allowed attackers to:
* execute commands
* read Pod information
* access logs
* create workloads
Modern Kubernetes disables anonymous access by default, but misconfigurations still appear during security assessments. During the meetup, Abhishek specifically recommended:
* disabling anonymous authentication
* using Webhook authorization
* authenticating with client certificates
to secure kubelets.
* * *
## Problem #7 - Who Actually Runs Containers?
The kubelet knows *what* should happen. But...It doesn't know **how** to start containers. That responsibility belongs to another component.
### Container Runtime
Every Worker Node includes a **Container Runtime**. Historically this was Docker.
Today, common runtimes include:
* containerd
* CRI-O
* Mirantis Container Runtime
The runtime performs low-level operations such as:
* downloading images
* unpacking layers
* creating namespaces
* configuring cgroups
* applying Linux capabilities
* launching processes
Conceptually:
```text
kubelet
│
V
Container Runtime
│
V
Linux Kernel
```
Remember our discussion from Chapter 1? Containers aren't magic.
They're Linux processes using:
* namespaces
* cgroups
* capabilities
The runtime simply assembles those pieces.
* * *
## Security Perspective
Container runtimes are attractive attack targets because they sit directly above the Linux kernel. Vulnerabilities here can enable:
* container escapes
* privilege escalation
* host compromise
One example discussed in the training deck is **CVE-2024-21626**, a vulnerability affecting `runc`, illustrating why keeping container runtimes up to date is just as important as patching Kubernetes itself.
* * *
## Problem #8 - Applications Need to Talk
Imagine two Pods. One hosts a frontend. The other hosts a backend API. How does the frontend find the backend? Can it simply connect to an IP address?
Not really. Pods are ephemeral.
Today:
```text
10.10.2.15
```
Tomorrow:
```text
10.10.8.44
```
IPs change constantly. Applications need something more stable.
### Services
A **Service** provides a stable network identity for a changing set of Pods. Instead of connecting to:
```text
10.10.2.15
```
Applications connect to:
```text
payment-service
```
Kubernetes resolves the correct Pods automatically.
```text
payment-service
│
┌──────────────┐
│ │ │
Pod A Pod B Pod C
```
Pods come and go. The Service remains.
* * *
## Engineering Insight
> Services decouple **identity** from **implementation**. Clients don't care which Pod answers. They only care that *someone* answers. This principle is fundamental to cloud-native architecture.
* * *
## Problem #9 - Users Live Outside the Cluster
Internal Services solve communication inside Kubernetes. But... How does a customer access your application? Users cannot connect directly to Pods. Pods constantly change.Node IPs are not stable interfaces. We need an entry point.
### Ingress
Ingress provides HTTP and HTTPS routing into the cluster. Think of it as Kubernetes' traffic controller.
```text
Internet
│
V
Ingress Controller
│
┌────────────┐
│ │
payment authentication
```
Ingress allows multiple applications to share the same external IP while routing requests based on:
* hostname
* URL path
* TLS configuration
* * *
## Security Perspective
Ingress controllers are often Internet-facing. Misconfigurations here may expose:
* administrative endpoints
* debugging interfaces
* internal services
Because Ingress sits at the boundary between users and the cluster, it is frequently reviewed during penetration tests.
* * *
## Problem #10 - Applications Need Configuration
Hardcoding configuration into images quickly becomes unmanageable. Imagine changing a database hostname. Without external configuration, you'd rebuild the image. That's inefficient.
### ConfigMaps
ConfigMaps separate configuration from application code.
Example:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: payment-config
data:
LOG_LEVEL: INFO
```
Now the same image can run in:
* development
* testing
* staging
* production
using different configuration.
* * *
## Security Perspective
ConfigMaps are **not** designed for secrets. They should never contain:
* passwords
* API keys
* cloud credentials
* tokens
Those belong elsewhere.
* * *
### Secrets
Kubernetes introduces **Secrets** to store sensitive information. However... This is one of the biggest misconceptions among beginners. Secrets are **Base64 encoded by default**. They are **not encrypted** merely because they are Base64 encoded.
This point was explicitly highlighted in Abhishek's training deck and reinforced during the meetup when discussing service account tokens and the ease with which Base64-encoded data can be decoded if an attacker gains access.
Proper protection requires additional controls such as:
* Encryption at rest
* Restricted RBAC
* External secret managers (for example, HashiCorp Vault)
* * *
## Architecture Is About Trust
By now, a pattern should be emerging. Every Kubernetes component exists because another component trusts it.
* The API Server trusts authenticated clients.
* Worker Nodes trust instructions from the API Server.
* kubelet trusts the Container Runtime.
* Pods trust Service Accounts.
* Applications trust Services.
* Users trust Ingress.
Every one of those trust relationships can become an attack path if authentication, authorization, or configuration is weak.
Understanding those relationships is the foundation of Kubernetes penetration testing.
* * *
![](https://cdn.hashnode.com/uploads/covers/65b618fc35b9d2122652b543/1c89d586-f2de-48a4-87ed-1bd89aa75d5d.png align="center")
Now that we understand how a Kubernetes cluster functions internally, it's time to think like an attacker.
In the next chapter, we'll build a **Kubernetes Threat Model** by asking three deceptively simple questions that Abhishek Pal introduced early in the meetup:
1. **Who are the attackers?**
2. **What are they trying to compromise?**
3. **How do they actually do it?**
Answering those questions will shift our perspective from infrastructure engineering to offensive security and lay the groundwork for the penetration testing methodology that follows.
* * *
## Chapter 3 - Understanding the Kubernetes Threat Model
> *"Attackers don't care whether your cluster uses Pods, ReplicaSets, or Deployments. They care about trust. They look for identities to steal, privileges to abuse, and trust relationships to exploit."*
## Every Penetration Test Starts With Three Questions
One of the things I appreciated about Abhishek's session was that he didn't begin with tools. He began with **threat modelling**.
Before discussing Kubescape. Before Falco.Before KICS.
He asked three very simple questions.
> **Who are the attackers?**
> **What are they attacking?**
> **How do they attack it?**
These questions also form the basis of professional security architecture.
Whether you're following Microsoft's Threat Modeling methodology, STRIDE, MITRE ATT&CK, or Google's internal security reviews, every assessment begins by understanding the adversary before discussing defenses. The training deck uses the same progression to introduce Kubernetes security.
* * *
### Who Are The Attackers?
When people hear Kubernetes Penetration Testing they often imagine "Hackers on the Internet." Reality is more interesting. Kubernetes has **multiple attacker personas**.
### External Attackers
The obvious one. Someone on the Internet discovers
* exposed applications
* exposed Ingress
* exposed Dashboard
* exposed API Server
Their goal? Gain an initial foothold. Usually through
* RCE
* SQL Injection
* Command Injection
* Deserialization
* File Upload
* SSRF
Notice something. The Kubernetes cluster itself wasn't initially vulnerable. The application running inside it was.
* * *
## Security Perspective
This is why Kubernetes security and Application Security cannot be separated. A vulnerable application often becomes the doorway into an otherwise well-configured cluster.
* * *
## Internal Attackers
Now imagine the attacker already compromised one application. Maybe `payment-api` . They now have shell access inside a Pod. Game over? Not even close. Actually...
The real attack is only beginning. Professional Kubernetes attackers ask
> What can this Pod access?
They begin enumerating
```bash
env
cat /var/run/secrets/...
mount
ip addr
kubectl
hostname
```
They're looking for
* credentials
* service account tokens
* mounted volumes
* cloud metadata
* RBAC permissions
The kind of lateral movement Abhishek explained during the meetup when discussing post-compromise scenarios inside Kubernetes clusters.
## Developers
Developers aren't malicious. But developers often receive excessive permissions. Examples include `cluster-admin` or `*` because - *"It's just development."*
Months later... Development becomes production. Overly permissive RBAC remains.
* * *
## Engineering Insight
> One of the biggest risks in Kubernetes isn't attackers. It's convenience. Every shortcut eventually becomes somebody else's penetration test.
* * *
## Malicious Administrators
Kubernetes administrators possess enormous power. They can
* create Pods
* mount Secrets
* access logs
* create Service Accounts
* modify RBAC
* read ConfigMaps
* schedule privileged workloads
Fortunately, organizations usually have
* logging
* audit trails
* approval workflows
Unfortunately, misconfigured clusters sometimes don't.
### Compromised CI/CD
One attack vector that receives far less attention than it deserves. Imagine compromising GitHub Actions, Azure DevOps, GitLab CI, Jenkins, ArgoCD.
Suddenly the attacker doesn't need to compromise Kubernetes.
They simply deploy malicious workloads through the organization's own deployment pipeline. This is why Kubernetes security increasingly overlaps with **software supply-chain security**.
### What Are Attackers Actually Trying To Steal?
Many newcomers answer `Containers`\- Not really. Containers are disposable. Attackers want things with **value**.
### Compute Resources
This is surprisingly common. Modern Kubernetes clusters contain enormous compute capacity. Why steal data…when you can steal electricity?
Cryptocurrency mining campaigns have repeatedly targeted Kubernetes clusters because compromised workloads provide scalable compute resources with somebody else paying the cloud bill. The training materials discuss campaigns such as **Kinsing** and **Scarleteel 2.0**, where attackers deployed miners after compromising cloud-native environments.
### Credentials
Credentials are far more valuable than containers.
Examples include
```plaintext
- AWS Access Keys
- Azure Managed Identity Tokens
- GCP Service Accounts
- Kubernetes Service Accounts
- kubeconfig files
- SSH Keys
- Vault Tokens
```
One credential often becomes another. Attack chains rarely stop where they begin.
### Secrets
Every organization stores
* API Keys
* JWT Signing Keys
* Database Passwords
* OAuth Secrets
* Certificates
The question is **where?** If the answer is `secrets`the next question becomes
> Who can read it?
### Cluster State
Remember etcd? Everything lives there. If attackers compromise etcd
they effectively understand the entire cluster - Workloads, Secrets, RBAC, Nodes, Configuration, Network,
This is why etcd is often called **the crown jewels** of Kubernetes.
### Data
Containers are temporary. Volumes are not. Persistent Volumes may contain
* customer data
* payment information
* healthcare records
* intellectual property
Destroying a Pod doesn't destroy the data.
## How Do Attackers Actually Get In?
**This is where Kubernetes becomes fascinating.**
Very few attacks begin with
> Exploit Kubernetes.
Instead they begin somewhere completely different.
### Initial Access
Examples include
* vulnerable web application
* compromised credentials
* phishing
* exposed dashboard
* vulnerable CI/CD
* cloud credential leakage
Exactly as Abhishek summarized in the deck, the most common initial attack vectors include exposed API servers, weak RBAC, plaintext secrets, privileged containers, unpatched software, compromised credentials, and vulnerable applications.
### Enumeration
Once inside attackers ask
> Where am I?
Examples
```bash
hostname
cat /etc/os-release
id
ip route
```
Inside Kubernetes additional questions become
```bash
cat /var/run/secrets/kubernetes.io/serviceaccount/token
cat /var/run/secrets/kubernetes.io/serviceaccount/namespace
```
Now they know
* namespace
* identity
* API endpoint
### Privilege Discovery
Can I access
```bash
kubectl auth can-i --list
```
Do I have
* create pods
* read secrets
* list nodes
* impersonate users
The answer determines the next attack.
### Lateral Movement
Professional attackers rarely stop after compromising one Pod. They pivot. Sometimes through RBAC. Sometimes hostPath. Sometimes Service Accounts. Sometimes cloud credentials. Sometimes the Kubernetes Dashboard. Exactly the lateral movement concepts discussed during the meetup.
### Kubernetes Is A Trust Graph
One way to think about Kubernetes is not as a cluster. Think of it as a graph of trust relationships.
```text
User
│
V
kubeconfig
│
V
API Server
│
┌────────────┐
│ │
Service Accounts Worker Nodes
│ │
V V
Secrets Container Runtime
│ │
└──────|─────┘
▼
Pods
```
Every arrow represents trust. Every trust relationship represents an opportunity.
Professional Kubernetes penetration testing is largely the process of discovering which trust relationships are stronger than they should be.
* * *
## Meetup Highlight
One of the recurring messages throughout the BreachForce session was that **misconfiguration remains the leading cause of Kubernetes compromise**. Rather than focusing solely on software vulnerabilities, Abhishek emphasized issues such as exposed API servers, overly permissive RBAC assignments (especially `cluster-admin`), plaintext secrets, privileged containers, insecure dashboards, and weak identity management. The offensive demonstrations consistently showed how attackers chain these weaknesses together instead of relying on a single critical vulnerability.
* * *
![](https://cdn.hashnode.com/uploads/covers/65b618fc35b9d2122652b543/f0855495-6086-4b5e-ac0b-ac1bc846acfc.png align="center")
Now that we understand **who attacks Kubernetes**, **what they target**, and **how they think**, we're finally ready to conduct a real assessment.
In **Chapter 4 – Kubernetes Penetration Testing Methodology**, we'll walk through a professional engagement from start to finish:
* Scoping and prerequisites
* Black-box vs. white-box assessments
* Reconnaissance
* Enumeration
* API discovery
* RBAC review
* Service Account analysis
* Dashboard assessment
* Manual verification
* Configuration review
* Reporting and risk prioritization
This chapter will closely follow the practical assessment methodology demonstrated during Abhishek Pal's BreachForce session while expanding it into a consultant-grade penetration testing workflow.
* * *
## Chapter 4 - Kubernetes Penetration Testing Methodology
> *"A penetration test is not a collection of tools. It is a structured investigation into how trust can be abused. In Kubernetes, success depends less on discovering vulnerabilities and more on understanding identities, permissions, and relationships."*
### What Does a Kubernetes Penetration Test Actually Look Like?
The first phase of every engagement involves asking questions - not about vulnerabilities, but about **scope**, **access**, and **objectives**.
During the BreachForce meetup, Abhishek Pal emphasized that understanding the assessment scope is critical before any technical testing begins. A Kubernetes security review may include evaluating access controls, cluster segregation, application exposure, network policies, operating system hardening, and Infrastructure as Code (IaC), depending on the engagement's objectives.
### Step 1 – Defining the Scope
Unlike a traditional web application assessment, Kubernetes environments introduce multiple layers that may or may not fall within scope. A professional engagement begins by identifying **what is being assessed**. Typical assessment targets include:
* Kubernetes Control Plane
* Worker Nodes
* Applications deployed inside the cluster
* Container images
* Infrastructure as Code (YAML manifests, Helm charts, Terraform)
* CI/CD pipelines
* Kubernetes Dashboard
* Cloud integrations (AWS, Azure, GCP)
* Container registries
* Secrets management systems
* Network segmentation
* Runtime security
Every engagement is different. Some organizations only want a configuration review. Others request a full penetration test that combines offensive techniques with hardening recommendations.
* * *
### Engineering Insight
> Kubernetes is not a product.
>
> It is an ecosystem.
>
> A meaningful assessment rarely focuses on only one component.
>
> Instead, it evaluates how the different components interact—and whether those interactions create opportunities for abuse.
* * *
### Step 2 – Understanding the Assessment Model
Not every Kubernetes engagement starts from the same position. During the meetup, Abhishek described two common approaches.
### Black-Box Assessment
In a black-box engagement, the assessor begins with minimal knowledge.
Typically, they have:
* No cluster documentation
* No kubeconfig
* No administrative credentials
* No architecture diagrams
The assumption is that the tester has gained access to the organization's network—or an Internet-facing application—but must discover everything else.
This closely resembles a real-world attacker. The objective is to answer questions such as:
* Can Kubernetes components be discovered?
* Is the API Server exposed?
* Can workloads be enumerated?
* Are identities leaked?
* Is lateral movement possible?
### White-Box Assessment
In a white-box engagement, the organization intentionally provides information such as:
* Architecture diagrams
* Cluster topology
* YAML manifests
* Read-only kubeconfig
* RBAC documentation
This approach allows assessors to spend less time discovering the environment and more time evaluating security controls.
### Meetup Highlight
Abhishek explained that most professional engagements are performed using a **black-box methodology**, where testers assume an attacker already has a foothold inside the network but **does not possess privileged Kubernetes credentials**. This mirrors the way many real-world intrusions unfold.
### Step 3 – Assessment Prerequisites
One question frequently asked by organizations is:
> "What do you need from us before the assessment begins?"
The answer depends on the engagement. For configuration reviews, assessors often request:
* Read-only `kubeconfig`
* Access to Infrastructure as Code repositories
* Network connectivity to the cluster
* Cluster version information
For black-box assessments, testers may require only network access.
During the meetup, Abhishek recommended that organizations provide a **virtual machine within the same subnet as the Kubernetes environment**, along with a **read-only kubeconfig** that allows listing resources without risking accidental modifications. If production access is not possible, a separate testing environment should be provided.
### Step 4 – Reconnaissance
Once scope is established, the technical assessment begins. Reconnaissance aims to answer one question:
> **What exists?**
Unlike traditional infrastructure, Kubernetes exposes multiple discovery points.
Examples include:
```plaintext
Internet-Facing Assets
- Web applications
- APIs
- Ingress controllers
- Load balancers
Kubernetes Services
- Kubernetes Dashboard
- API Server
- kubelet endpoints
- Metrics endpoints
Internal Infrastructure
- Cluster DNS
- Internal Services
- Node IPs
- Pod networks
```
The objective is not exploitation.
It is **visibility**.
* * *
### Step 5 – Enumerating the Cluster
If the assessor gains Kubernetes credentials — or compromises a workload — the next phase is enumeration, asking questions such as: which namespaces exist? Which Pods are running? Which Service Accounts are present? Which Secrets are accessible? What RBAC permissions are available? Which nodes host workloads? Which APIs are enabled?
Example commands:
```shell
kubectl get namespaces
kubectl get pods -A
kubectl get services -A
kubectl get secrets -A
kubectl get nodes
```
Enumeration provides the map. Without a map, exploitation is blind.
### Step 6 – Identity First, Exploits Later
One of the biggest differences between Kubernetes and traditional penetration testing is the importance of identity. Attackers do not immediately search for kernel exploits — they ask: *"Who am I?"*
Common checks include:
```shell
id
hostname
env
cat /var/run/secrets/kubernetes.io/serviceaccount/token
cat /var/run/secrets/kubernetes.io/serviceaccount/namespace
```
These simple commands reveal Service Account tokens, namespace, cluster identity, environment variables, and potential cloud credentials. As demonstrated during the meetup, mounted Service Account tokens can often be used to authenticate to the Kubernetes API, making RBAC permissions the determining factor in what an attacker can access next.
### Step 7 – Reviewing RBAC
Once an identity is obtained, the next logical question is: *what am I allowed to do?* Rather than guessing, Kubernetes provides a straightforward mechanism for checking permissions:
bash
```bash
kubectl auth can-i --list
```
Assessors look for permissions such as `get`, `list`, `watch`, `create`, `update`, `patch`, and `delete`. Excessive privileges — especially broad access to Secrets, Pods, or ClusterRoles — can significantly increase the impact of a compromise.
> C**ommon Misconfiguration** - One of the most frequently encountered issues during Kubernetes assessments is assigning the `cluster-admin` role where a much narrower role would suffice. Convenience during development often becomes a critical finding during production assessments.
### Step 8 – Configuration Review
Professional Kubernetes engagements extend beyond runtime testing. Configuration reviews examine Infrastructure as Code before workloads are even deployed. Typical review areas include `securityContext`, `hostNetwork`, `hostPID`, `hostIPC`, `privileged: true`, Linux capabilities, `automountServiceAccountToken`, image sources, NetworkPolicies, and resource limits.
As Abhishek emphasized during the meetup, Kubernetes manifests are effectively Infrastructure as Code, and misconfigurations within these files frequently become production vulnerabilities. Reviewing YAML definitions is therefore a critical part of any assessment.
* * *
## Security Is About Context
A key takeaway from the session—and one echoed throughout this series—is that **individual findings rarely tell the whole story**.
For example:
* A Service Account token may not be dangerous if RBAC is properly restricted.
* A privileged container becomes far more significant if it also mounts sensitive host paths.
* An exposed Dashboard is far more severe if authentication and RBAC are weak.
Professional assessments therefore focus on **attack chains**, not isolated issues.
* * *
![](https://cdn.hashnode.com/uploads/covers/65b618fc35b9d2122652b543/a39a7487-a359-4902-ae9e-f70d8fc3829d.png align="center")
In the next chapter, we'll move beyond methodology and begin exploiting common Kubernetes weaknesses.
We'll explore:
* Service Account abuse
* RBAC misconfigurations
* Secret extraction
* Privileged containers
* Container escapes
* Linux capabilities
* HostPath abuse
* Lateral movement across the cluster
This is where the offensive techniques demonstrated during the BreachForce meetup begin to come together, showing how seemingly minor misconfigurations can be chained into meaningful compromises.
* * *
## Chapter 5 - Exploitation : From Initial Foothold to Cluster Compromise
> *"Modern Kubernetes attacks rarely rely on a single critical vulnerability. They are chains of seemingly harmless misconfigurations that, when combined, allow attackers to move from one compromised container to complete control of a cluster—or even the underlying cloud account."*
### Exploitation Begins Long Before Kubernetes
One of the biggest misconceptions surrounding Kubernetes security is that attackers begin by attacking Kubernetes. They don't. They attack **applications**.
A vulnerable web application running inside Kubernetes is still just a vulnerable web application. Consider the following scenario.
```text
Internet
│
V
Vulnerable Web Application
│
V
Remote Code Execution
│
V
Shell Inside a Kubernetes Pod
```
At this point, many defenders breathe a sigh of relief.
> "The attacker is only inside one container."
From an attacker's perspective, however, this is merely the beginning.
### The First Question Every Attacker Asks
Professional attackers rarely execute complex exploits immediately. Instead, they stop and ask a deceptively simple question: *"Where am I?"* The objective is to understand the environment before taking any action.
Typical commands include:
```bash
hostname
id
whoami
pwd
env
mount
ip addr
cat /etc/os-release
```
These commands reveal valuable information about the operating system, user privileges, mounted filesystems, environment variables, network configuration, and runtime environment. Inside Kubernetes, they often reveal something even more valuable.
### Service Account Tokens
Every Pod has an identity, usually represented by a Service Account. Historically, Kubernetes automatically mounted Service Account tokens into Pods under `/var/run/secrets/kubernetes.io/serviceaccount/`. An attacker immediately checks:
```bash
ls /var/run/secrets/kubernetes.io/serviceaccount/
cat token
cat namespace
cat ca.crt
```
Finding these files tells the attacker three things: their Kubernetes identity, which namespace they're running in, and how to authenticate to the API Server.
During the BreachForce meetup, Abhishek demonstrated how mounted Service Account tokens become one of the first artefacts attackers inspect after compromising a Pod, because they often provide a direct path to interacting with the Kubernetes API.
> **Engineering Insight** Service Accounts were created to solve an operational problem. Applications often need to communicate with Kubernetes — creating Jobs, watching Pods, reading ConfigMaps, updating custom resources. Without Service Accounts, every application would require manually managed credentials, and automation would become nearly impossible. Like many Kubernetes features, Service Accounts solved a real engineering problem — they also introduced a new trust relationship.
> **Security Perspective** A Service Account token by itself is not a vulnerability. Its impact depends entirely on RBAC. A token with no permissions is almost useless. A token bound to `cluster-admin` can result in complete cluster compromise.
### Talking to the API Server
If the Pod contains a Service Account token, an attacker next asks: *"Can I talk to Kubernetes?"* Often, the API endpoint is already available as environment variables:
```shell
echo $KUBERNETES_SERVICE_HOST
echo $KUBERNETES_SERVICE_PORT
```
Or via DNS:
```text
https://kubernetes.default.svc
```
Using the mounted token, requests can be sent directly to the Kubernetes API. If permissions allow, the attacker can begin enumerating resources without ever installing kubectl. This is an important lesson for defenders: **blocking kubectl does not prevent interaction with the Kubernetes API. The API itself is the security boundary.**
### RBAC — The Real Target
Many newcomers think RBAC is simply another configuration file. Attackers think differently. RBAC answers one question: *"What can this identity do?"*
Once a token is obtained, attackers enumerate permissions. Using kubectl, a common check is:
```bash
kubectl auth can-i --list
```
From raw API access, attackers attempt operations such as listing Pods, reading Secrets, creating Pods, listing Nodes, viewing ConfigMaps, and creating Service Accounts. The goal isn't simply to collect information — the goal is to discover paths to higher privilege.
> **Common Misconfiguration** One of the most common findings in Kubernetes assessments is overly broad RBAC assignments — Development Service Accounts bound to `cluster-admin`, wildcard (`*`) permissions, unnecessary access to Secrets, and ClusterRoles reused across unrelated workloads. These shortcuts often exist because they simplify deployment during development. They become dangerous once the application reaches production.
### Secrets — The Crown Jewels
Once attackers understand their permissions, they begin searching for secrets — database passwords, cloud credentials, API keys, TLS private keys, OAuth secrets, internal service credentials. If RBAC allows:
```bash
kubectl get secrets
kubectl describe secret
```
the attacker may recover credentials that extend far beyond Kubernetes itself. This is one reason why least privilege is so heavily emphasized in Kubernetes hardening guidance.
> **A Common Misunderstanding** Many engineers believe Kubernetes Secrets are encrypted. By default, they are Base64 encoded, not encrypted. Encoding improves transport compatibility — it does not provide confidentiality. As highlighted during the meetup, proper protection requires controls such as encryption at rest, strong RBAC, and, where appropriate, integration with dedicated secret management systems.
### Privileged Containers
Not every attack involves credentials. Sometimes the workload itself has been granted excessive Linux privileges. Consider a Pod running with:
```yaml
securityContext:
privileged: true
```
From an engineering perspective, privileged containers were introduced for workloads that genuinely require deep interaction with the host — for example, certain storage drivers, networking plugins, or system-level agents. From a security perspective, they dramatically expand what a compromised container can do — a privileged container may interact with host devices, load kernel modules, or access resources that are normally isolated. Professional assessments therefore treat privileged workloads as high-priority review items, validating whether the privilege is truly necessary or merely convenient.
### HostPath — When Isolation Breaks Down
Another frequent review area is the use of hostPath volumes:
```yaml
volumes:
- name: host
hostPath:
path: /
```
`hostPath` allows a Pod to mount directories from the underlying node. This is sometimes required for infrastructure components, but it also weakens the isolation boundary between the container and the host. During an assessment, reviewers ask: what directories are mounted? Is write access required? Can sensitive files be reached? Is this workload running with elevated privileges? The risk depends on context, which is why manual review is essential.
### Attack Chains, Not Individual Findings
One of the most important lessons from the BreachForce session was that real-world compromises are rarely the result of a single catastrophic vulnerability. Instead, attackers combine multiple weaknesses. A typical chain might look like this:
```text
Internet
│
V
Application RCE
│
V
Shell in Pod
│
v
Mounted Service Account Token
│
V
Excessive RBAC Permissions
│
V
Read Kubernetes Secrets
│
V
Recover Cloud Credentials
│
V
Compromise Cloud Infrastructure
```
None of these steps may be critical in isolation. Together, they can lead to complete environmental compromise. This emphasis on chaining weaknesses together rather than relying on a single exploit was a recurring theme throughout Abhishek Pal's demonstrations and discussion.
> **Engineering Insight** Security reviews often focus on individual findings because reports are easier to structure that way. Attackers don't think in findings — they think in paths. The shortest path from a compromised application to sensitive data is the path they'll choose. Understanding and breaking those paths is one of the primary goals of Kubernetes security.
* * *
## Looking Ahead
In the next chapter, we'll shift from runtime exploitation to **Infrastructure as Code (IaC)**.
Rather than attacking a running cluster, we'll examine the Kubernetes manifests that define it.
We'll learn why many vulnerabilities never originate in the cluster itself—they originate in YAML files committed months earlier to source control.
This chapter will closely follow the configuration review techniques demonstrated during the BreachForce meetup, examining `securityContext`, `hostNetwork`, `hostPID`, Linux capabilities, `automountServiceAccountToken`, image sources, and other manifest-level decisions that determine the security posture of a cluster before a single Pod is ever created.
* * *
## Chapter 6 - Configuration Review : Finding Vulnerabilities Before They Reach Production
> *"Every Kubernetes cluster is built from YAML. Long before an attacker compromises a Pod, a developer has already decided what that Pod is allowed to do. Configuration reviews aim to discover insecure decisions before attackers do."*
### Security Starts Before the Cluster Exists
Imagine two organizations. Company A spends thousands of dollars every year on:
* Runtime Detection
* SIEM
* EDR
* Falco
* Incident Response
Company B spends time reviewing Kubernetes manifests before deployment.
Ironically, Company B often experiences fewer security incidents. Why? Because many Kubernetes vulnerabilities don't originate inside running Pods. They originate months earlier...inside Git.
A single YAML file committed during development may eventually become a production vulnerability. This is why Infrastructure as Code (IaC) has become one of the most important aspects of Kubernetes security.
As emphasized throughout the BreachForce meetup, reviewing Kubernetes manifests is just as important as testing a live cluster because insecure defaults frequently originate in deployment definitions rather than runtime behavior.
## Infrastructure Became Code
Years ago, provisioning infrastructure meant logging into servers manually.
```bash
ssh production
apt install nginx
vim nginx.conf
service nginx restart
```
Every administrator configured servers slightly differently. Configuration drift became inevitable. Infrastructure as Code changed that. Instead of manually configuring servers, engineers described infrastructure declaratively.
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service
```
The YAML file became the **single source of truth**. If the YAML is secure...the deployment begins securely. If the YAML is insecure... every deployment inherits that insecurity.
### Engineering Insight
> Infrastructure as Code isn't merely an automation technique. It's a security control. It enables:
>
> * Code Review
>
> * Version Control
>
> * Peer Review
>
> * Automated Security Scanning
>
> * Change Tracking
>
> * Rollbacks
>
>
> Security shifts **left**, before workloads ever reach production.
### Reading YAML Like a Pentester
Beginners often read manifests from top to bottom. Professional reviewers ask a different question.
> **What permissions does this workload receive?**
Every assessment begins with identity. Then privilege. Then isolation. Then networking. Then storage.
### securityContext
The `securityContext` is one of the first sections reviewers inspect because it defines how a workload interacts with the Linux kernel.
Example:
```yaml
securityContext:
runAsUser: 1000
runAsNonRoot: true
allowPrivilegeEscalation: false
```
This configuration communicates several important security decisions.
* The application should not run as root.
* It should execute using a dedicated user.
* It should not gain additional privileges after startup.
### Why Does This Exist?
Linux has always allowed processes to execute under different users. Kubernetes extends this capability into containers. Instead of trusting developers to choose secure defaults, cluster administrators can enforce them declaratively.
### Security Perspective
Running containers as `root` isn't automatically a vulnerability. Many legitimate workloads still require elevated privileges.
However, if an application running as root is compromised, the attacker begins with significantly greater control inside the container. The first review question therefore becomes:
> **Does this workload actually need root?**
### Privileged Containers
One flag immediately catches every reviewer's attention.
```yaml
privileged: true
```
This setting effectively removes many of the isolation guarantees containers normally provide.
A privileged container may:
* Access host devices
* Load kernel modules
* Modify network configuration
* Interact directly with the host operating system
There are valid reasons for privileged workloads. Examples include:
* CNI plugins
* Storage drivers
* Low-level monitoring agents
However, most business applications should never require this level of access.
### Common Misconfiguration
During internal development, developers sometimes enable:
```yaml
privileged: true
```
to "make things work." The configuration remains. The application reaches production. Nobody revisits the decision. Months later, it appears in a penetration testing report.
### Linux Capabilities
Remember Chapter 1. Linux split root privileges into smaller capabilities. Kubernetes allows those capabilities to be managed explicitly. Example:
```yaml
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
```
This follows the **Principle of Least Privilege**. Instead of granting unrestricted power, only the capability required for the application is added.
* * *
### Engineering Insight
Capabilities illustrate one of Kubernetes' core design philosophies. Rather than asking
> Root or Non-root?
Kubernetes asks
> Exactly which privilege does this workload require?
Granularity improves security.
* * *
## hostNetwork
Another important review point. Example:
```yaml
hostNetwork: true
```
Normally, Pods receive isolated network namespaces. With `hostNetwork` enabled, the Pod shares the host's network stack. Why would someone enable this?
Some infrastructure software genuinely requires direct access to host networking. Examples include:
* CNI plugins
* Network monitoring agents
* DNS services
### Security Perspective
For ordinary applications,`hostNetwork` increases exposure. The Pod may interact with services that would normally remain isolated. Reviewers therefore ask:
* Is host networking necessary?
* What services become reachable?
* Does the application require direct host communication?
### hostPID
Another powerful option. `hostPID: true` Normally, containers only see their own processes. With `hostPID`, the container can observe processes running on the host. For troubleshooting, this may be useful.
For attackers, it's equally interesting. Process visibility often reveals:
* Monitoring agents
* Security software
* Runtime characteristics
* Sensitive process arguments
### hostIPC
Similarly, `hostIPC: true` shares the host's Inter-Process Communication namespace. Most business applications never require this. Whenever reviewers encounter `hostIPC`, they ask:
> **What operational requirement justifies weakening isolation?**
If no clear answer exists, the configuration deserves further investigation.
### automountServiceAccountToken
One of the most overlooked configuration options. By default, Pods often receive a mounted Service Account token automatically.
```yaml
automountServiceAccountToken: false
```
disables this behavior.
### Why Does This Matter?
Many applications never communicate with the Kubernetes API. If they don't require API access, they don't require credentials. Removing unnecessary Service Account tokens reduces the attack surface. This recommendation aligns closely with the least-privilege principles emphasized during the BreachForce meetup.
#### Image Sources
Another review question: where does this image come from?
```yaml
image: nginx:latest
```
This immediately raises several questions: why `latest`? Which registry? Who maintains it? Is it signed? Has it been scanned?
> **Engineering Insight** Containers inherit the security posture of their images. If the image contains vulnerable software, every Pod created from it inherits those vulnerabilities. Secure Kubernetes begins with trusted images.
#### ConfigMaps vs Secrets
A surprisingly common finding — developers place passwords inside ConfigMaps:
```yaml
data:
DATABASE_PASSWORD: Password123
```
ConfigMaps are intended for configuration, not credentials. Sensitive values belong in Kubernetes Secrets — or, better yet, in dedicated secret management systems integrated with the cluster.
#### Resource Limits
```yaml
resources:
limits:
cpu: "500m"
memory: "512Mi"
```
These limits are not merely operational settings — they are security controls. Without resource limits, a compromised workload may consume excessive CPU or memory, contributing to denial-of-service conditions against other workloads.
### Configuration Reviews Are About Intent
One of the biggest lessons in reviewing Kubernetes manifests is this: most findings aren't bugs, they're decisions. Some are excellent decisions. Some are shortcuts. Some are temporary workarounds that became permanent. The reviewer's job is not simply to identify insecure settings — it's to understand why they exist and whether that justification still holds.
> **Meetup Highlight** During the BreachForce session, Abhishek stressed that configuration reviews are not about blindly applying CIS recommendations. Reviewers should understand the operational purpose of each configuration, distinguish legitimate infrastructure requirements from unnecessary privilege, and balance security with functionality. The goal is to identify decisions that increase risk — not merely settings that differ from a benchmark.
* * *
## Coming Up Next
So far we've relied primarily on manual analysis. In practice, consultants combine manual reviews with specialized security tools. In the next chapter, we'll explore the Kubernetes security ecosystem, including:
* **Kubescape**
* **KICS**
* **kube-bench**
* **Trivy**
* **Grype**
* **Falco**
* **kube-hunter**
* **kubeaudit**
Rather than simply listing features, we'll answer four practical questions for each tool:
1. **What problem does it solve?**
2. **How does it work?**
3. **Where does it fit in an assessment?**
4. **What are its limitations?**
This approach mirrors real-world consulting engagements, where no single tool is sufficient and effective assessments combine automation with human judgment.
* * *
## Chapter 7 - Security Automation: Scaling Kubernetes Security Beyond Manual Reviews
> No security engineer wants to manually review 10,000 YAML files, inspect every container image, or compare an entire cluster against hundreds of CIS recommendations. Automation exists to reduce repetitive work — not to replace human judgement.
### Manual Reviews Don't Scale
Imagine joining a Fortune 500 company. You receive access to their Kubernetes environment and immediately discover 350 Namespaces, 7,000 Pods, 1,200 Deployments, 900 ConfigMaps, 600 Secrets, 300 Helm Charts, and hundreds of Git repositories. Your manager asks: *"Can you review everything before Friday?"*
Impossible. Even reading the YAML files would take days. Reviewing every deployment manually would take weeks. Infrastructure had become too large for humans alone.
> **Engineering Insight** This is exactly the same engineering problem we encountered throughout this series. Humans solved deployment by writing Infrastructure as Code. Infrastructure eventually became too large. The next logical step was automation. Security followed the same evolutionary path.
### What Security Tools Actually Do
One misconception is that security tools "find vulnerabilities." Most Kubernetes security tools don't exploit anything — instead, they answer questions: Does this container image contain vulnerable software? Does this manifest violate security best practices? Does this cluster comply with CIS? Is this Pod privileged? Is this workload suspicious at runtime? Different tools answer different questions; no single tool answers all of them.
### Static vs Runtime Security
A useful way to classify Kubernetes security tools is by **when** they operate.
```text
Kubernetes Security
┌──────────────┴───────┐
Before Deployment After Deployment
Static Analysis Runtime Analysis
IaC Scanners Runtime Detection
Image Scanners Behavioral Monitoring
Compliance Threat Detection
```
This distinction matters because finding a problem in Git is very different from finding one in production.
### Shift Left Security
Modern security increasingly follows the principle of Shift Left. Rather than waiting until software reaches production, organizations move security checks earlier in the development lifecycle:
```plaintext
Developer → Git Commit → CI Pipeline → Security Scanning → Deployment
```
Finding an issue before deployment is almost always cheaper than responding to an incident after deployment.
### KICS (Keeping Infrastructure as Code Secure)
The first tool discussed during the meetup was **KICS**. Unlike runtime security tools, KICS analyzes Infrastructure as Code before deployment. Supported technologies include Kubernetes YAML, Terraform, Dockerfiles, Helm Charts, and CloudFormation.
KICS asks questions such as: is `hostNetwork` enabled? Is `hostPID` enabled? Are containers privileged? Are resource limits missing? Are images using `latest`? Is `runAsRoot` configured? Notice something — KICS isn't scanning Kubernetes, it's scanning your code. This aligns perfectly with the "shift left" philosophy discussed earlier.
> **Security Perspective** KICS helps prevent insecure configurations from ever reaching production. However, it cannot determine whether a privileged container is justified by a legitimate operational requirement. Human review is still required — automation identifies patterns, engineers evaluate context.
### Trivy
Most people know Trivy as a vulnerability scanner, but it does considerably more. Trivy can scan container images, Kubernetes clusters, filesystems, Git repositories, SBOMs, and configuration files. Suppose a container image includes an outdated version of OpenSSL — Trivy detects the vulnerable package before deployment, significantly reducing the attack surface
> **Engineering Insight** Container images should be treated like software dependencies. If an image contains vulnerable libraries, every Pod created from that image inherits those vulnerabilities. Secure deployments begin with secure images.
### Grype
Grype occupies a similar space, focusing on Software Bill of Materials (SBOM) analysis and vulnerability identification. Organizations often use Trivy and Grype together because each has different vulnerability databases and detection capabilities. Security is rarely about choosing one tool — it's about layering capabilities.
### kube-bench
Configuration vulnerabilities differ from software vulnerabilities. Suppose every package is fully patched — the cluster may still violate the CIS Kubernetes Benchmark. **kube-bench** compares Kubernetes configuration against the CIS recommendations, examining API Server configuration, kubelet configuration, etcd configuration, Control Plane security, and node security. Instead of asking
> "Is software vulnerable?",
kube-bench asks
> "Is Kubernetes configured securely?"
### Kubescape
One of the highlights of Abhishek's session was **Kubescape**, an open-source Kubernetes security platform that combines posture management, configuration analysis, vulnerability scanning, and compliance checks.
Kubescape can evaluate clusters against multiple frameworks, including:
* CIS Kubernetes Benchmark
* NSA/CISA Kubernetes Hardening Guidance
* MITRE ATT&CK for Containers
It provides a broad view of the cluster's security posture rather than focusing on a single class of issue.
### Engineering Insight
> Think of Kubescape as a platform rather than a single scanner. Instead of answering one question, it correlates multiple security signals to help prioritize remediation.
### Falco
Everything we've discussed so far happens **before** or **during** deployment. Falco operates after deployment. It asks:
> "What is happening inside my cluster right now?"
Examples include:
* Unexpected shell execution inside a container
* Reading sensitive files
* Privilege escalation attempts
* Access to the Docker socket
* Suspicious network connections
Falco uses kernel events (and increasingly eBPF on modern systems) to detect behavior that deviates from expected activity.
### Security Perspective
> Falco does not prevent attacks. It improves **visibility**. Detection remains a critical layer because preventive controls are never perfect.
### kube-hunter
Where kube-bench checks compliance, **kube-hunter** thinks like an attacker. It actively searches for weaknesses such as:
* Exposed dashboards
* Open kubelet APIs
* Anonymous authentication
* Weak TLS configurations
* Misconfigured API Servers
Its focus is on identifying externally observable weaknesses rather than validating internal compliance.
### kubeaudit
Another useful configuration review tool is **kubeaudit**. It examines Kubernetes resources for common security issues, including:
* Privileged containers
* Missing resource limits
* Service Account token usage
* Insecure capabilities
* Image tags
* SecurityContext configuration
Many of the findings discussed in the previous chapter can be detected automatically by kubeaudit.
### No Tool Replaces Understanding
At this point, you might wonder:
> Why not simply run every tool?
Because tools operate on rules. Attackers operate on context. Consider this Deployment:
```yaml
securityContext:
privileged: true
```
Every scanner reports: **High Risk**. But suppose the workload is a Container Network Interface (CNI) plugin that genuinely requires elevated privileges to manage network interfaces. The finding is technically correct.
The risk assessment requires human judgement. This is why professional consultants never rely exclusively on automated output.
> **Meetup Highlight** Throughout the BreachForce session, Abhishek emphasized that tools should assist — not replace — manual assessment. Automated scanners quickly identify common misconfigurations and compliance gaps, but understanding architecture, validating findings, and determining business impact remain responsibilities of the assessor. That balance between automation and expertise is what distinguishes a mature Kubernetes security assessment from a simple scan.
### Defense in Depth Through Automation
An effective Kubernetes security program layers multiple forms of automation:
| Stage | Goal | Example Tools |
| --- | --- | --- |
| Development | Secure IaC | KICS |
| Build | Scan Images | Trivy, Grype |
| Deploy | Validate Policies | Kubescape |
| Compliance | Benchmark Configuration | kube-bench |
| Runtime | Detect Suspicious Behavior | Falco |
| Offensive Validation | Identify Exposed Services | kube-hunter |
| Continuous Review | Audit Cluster Resources | kubeaudit |
No single tool is sufficient. Together, they provide broad visibility across the software supply chain and the running cluster.
* * *
## Looking Ahead
Automation helps identify weaknesses. The next logical question is:
> **How do we eliminate them?**
In **Chapter 8 – Hardening Kubernetes Clusters**, we'll move from identifying problems to preventing them.
We'll explore:
* Principle of Least Privilege
* RBAC design
* Pod Security Admission
* OPA Gatekeeper
* Kyverno
* NetworkPolicies
* API Server hardening
* kubelet hardening
* etcd security
* Image signing
* Supply-chain security
* Runtime protection
* Secrets management
Rather than presenting these as isolated recommendations, we'll examine the engineering problem each control solves, the attack it prevents, and how organizations can implement it without disrupting operations.
* * *
## Chapter 8 - Hardening Kubernetes Clusters : Building Security by Design
> *"Hardening is not the process of adding security. It is the process of removing unnecessary trust."*
### Security Is an Engineering Decision
Many organizations think Kubernetes hardening is a checklist. Disable this. Enable that. Install this tool. Run this benchmark. Reality is different.
Every Kubernetes hardening recommendation exists because someone experienced a failure. Every security control is the solution to an engineering problem. Understanding that problem makes the recommendation obvious.
### Principle of Least Privilege
Let's begin with the single most important security principle - **Least Privilege.** Imagine giving every employee inside a company:
* Master building keys
* Payroll access
* HR database access
* CEO email access
Ridiculous. Yet many Kubernetes clusters do exactly that. Applications receive permissions they never need.
### Engineering Problem
> Applications sometimes need to interact with Kubernetes. Examples include
>
> * Watching Pods
>
> * Reading ConfigMaps
>
> * Creating Jobs
>
>
> How do we give applications enough permission to work...without giving them control of the cluster?
## Kubernetes Solution
**RBAC -** Role-Based Access Control. Instead of saying
> This application is trusted.
RBAC asks
> What exactly does this application need?
Example:
```yaml
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get","list"]
```
Notice what's missing.
* delete
* patch
* update
* create
Those permissions simply aren't required.
### Security Perspective
Many real-world Kubernetes compromises don't begin with software vulnerabilities. They begin with:
```plaintext
cluster-admin
```
assigned because
> "It was easier."
During the meetup, Abhishek repeatedly emphasized avoiding broad privileges, especially unnecessary `cluster-admin` assignments, because they transform otherwise limited compromises into full cluster takeovers.
### Pod Security
Remember Chapter 6.
We reviewed
```yaml
privileged: true
```
Now let's understand why Kubernetes introduced Pod Security.
### Engineering Problem
Developers can write anything inside YAML. Should Kubernetes simply trust them? Imagine someone deploying
```yaml
hostPID: true
hostNetwork: true
privileged: true
```
Technically valid. Operationally dangerous.
### Pod Security Standards
Modern Kubernetes introduces three security profiles.
**Privileged**
Almost unrestricted. Designed for infrastructure software. Examples:
* CNI plugins
* CSI drivers
* Storage systems
**Baseline**
Allows common workloads. Blocks dangerous configurations. Good default for many organizations.
**Restricted**
Strictest profile. Requires
* Non-root users
* Limited capabilities
* Restricted host access
* Safer defaults
Most business applications should strive toward this profile.
* * *
### Engineering Insight
> Notice the philosophy. Instead of trusting every deployment, Kubernetes validates workloads before they enter the cluster. Security moves earlier in the lifecycle.
### Pod Security Admission
Older Kubernetes versions used **PodSecurityPolicy (PSP).** Although powerful, PSP became difficult to configure. It was deprecated and eventually removed. Modern Kubernetes now uses **Pod Security Admission (PSA)** which enforces the Privileged, Baseline, and Restricted standards at the namespace level.
This is an important distinction because older articles still reference PSP even though new clusters rely on PSA.
## Network Policies
Now imagine two Pods.
```plaintext
Payment API
Recommendation Engine
```
Should they communicate? Maybe. Maybe not. Without explicit restrictions, many Kubernetes networking implementations allow Pods to communicate freely.
This "allow by default" model can make lateral movement easier after an initial compromise.
* * *
### Engineering Problem
Applications need networking. Attackers love networking. How do we permit only legitimate communication?
* * *
### Kubernetes Solution
NetworkPolicy. Example:
```yaml
kind: NetworkPolicy
```
Rather than saying
> Anyone may connect.
We define exactly who may connect. This introduces micro-segmentation inside the cluster.
### Security Perspective
Suppose attackers compromise `Frontend`. Without Network Policies they may attempt connections to
```plaintext
Database
Redis
Internal APIs
Monitoring
```
With properly designed policies, those paths may simply not exist. Lateral movement becomes significantly more difficult.
### Secrets Management
Earlier we learned Secrets are Base64 encoded. Not encrypted. Now let's discuss how organizations should actually manage credentials.
### Engineering Problem
> Applications need passwords. But storing passwords inside Git repositories is obviously unacceptable. Where should credentials live?
## Kubernetes Solution
Kubernetes Secrets. Better than ConfigMaps. But still only part of the answer. Modern organizations increasingly integrate dedicated secret management platforms such as:
* HashiCorp Vault
* External Secrets Operator
* Cloud-native secret managers (AWS Secrets Manager, Azure Key Vault, Google Secret Manager)
This keeps long-lived credentials outside application manifests while allowing workloads to retrieve them securely at runtime.
### API Server Hardening
Throughout this series, one component has appeared repeatedly. The API Server. Because everything communicates through it.
### Hardening Priorities
Organizations should ensure:
* TLS enabled
* Anonymous authentication disabled
* Strong RBAC
* Audit logging enabled
* Admission controllers configured
* Regular version updates
Remember the API Server isn't simply another service. It is the control plane's front door.
### kubelet Hardening
Worker Nodes deserve equal attention. Recommended practices include:
* Disable anonymous authentication
* Enable Webhook authorization
* Authenticate using client certificates
* Restrict kubelet API exposure
* Monitor kubelet logs
These recommendations align with the hardening guidance discussed by Abhishek during the meetup.
### etcd Hardening
Earlier we called etcd the cluster's memory. Now let's secure it. Best practices include:
* Mutual TLS
* Encryption at rest
* Restricted network access
* Regular backups
* Certificate rotation
If attackers obtain unrestricted access to etcd, cluster recovery becomes far more difficult.
### Supply Chain Security
Security doesn't begin inside Kubernetes. It begins before images are built. Questions reviewers should ask include:
* Who built this image?
* Which base image was used?
* Is the image signed?
* Has it been scanned?
* Can its provenance be verified?
Technologies such as **Sigstore Cosign** and **Software Bills of Materials (SBOMs)** help organizations establish trust in container images before deployment.
### Engineering Insight
Modern attackers increasingly target the software supply chain because compromising one trusted artifact can affect every deployment derived from it. Protecting the build pipeline is therefore just as important as protecting the cluster.
### Runtime Protection
Preventive controls are essential. But no preventive control is perfect. Organizations should therefore complement prevention with runtime visibility.
Examples include:
* Falco
* eBPF-based monitoring
* Audit logs
* SIEM integration
* Behavioral analytics
The goal is not merely to stop attacks. It is to detect and respond when prevention fails.
### Defense in Depth
No individual control secures Kubernetes. Instead, security emerges from multiple independent layers.
```text
Kubernetes Security
┌───────────────────────────────┐
│ Secure Software Development │
├───────────────────────────────┤
│ Image Scanning & Signing │
├───────────────────────────────┤
│ Infrastructure as Code Review │
├───────────────────────────────┤
│ Admission Policies │
├───────────────────────────────┤
│ RBAC │
├───────────────────────────────┤
│ Pod Security │
├───────────────────────────────┤
│ Network Policies │
├───────────────────────────────┤
│ Runtime Detection │
├───────────────────────────────┤
│ Monitoring & Incident Response│
└───────────────────────────────┘
```
Each layer assumes another layer may eventually fail. That philosophy is known as **Defense in Depth**, and it remains one of the foundational principles of Kubernetes security.
### Meetup Highlight
> One of the strongest messages from Abhishek's session was that **hardening is not a single activity performed after deployment**. It is a continuous process that spans Infrastructure as Code, RBAC, Pod Security, network segmentation, runtime monitoring, and operational discipline. Organizations that rely solely on vulnerability scanning often overlook the misconfigurations that attackers exploit most frequently.
* * *
## Chapter 9 - Kubernetes in the Real World : Lessons from Modern Cloud Attacks
> *"The most dangerous Kubernetes attacks don't begin with Kubernetes. They begin with a small mistake that nobody considered important."*
### Security Isn't About CVEs
Ask someone:
> "How do attackers compromise Kubernetes?"
The usual answers are
* Zero-days
* Container escapes
* Kernel exploits
Reality is surprisingly different. Most cloud compromises begin with a misconfiguration. Not because engineers are careless. Because modern cloud infrastructure has become incredibly complex. Attackers simply exploit complexity better than defenders manage it.
* * *
### Case Study 1 — Kinsing
One of the most active malware campaigns targeting cloud-native infrastructure in recent years is **Kinsing**. Unlike traditional malware, Kinsing doesn't care about stealing documents. It wants one thing. **Compute.** Because compute generates money.
### Initial Access
The attack often begins with an exposed application.
```text
Internet
↓
Vulnerable Web Application
↓
Remote Code Execution
```
Nothing Kubernetes-specific yet.
### Post Exploitation
Once inside the container, the attacker begins asking questions. Exactly like we learned in Chapter 5.
```bash
id
hostname
env
mount
cat serviceaccount/token
```
### Privilege Discovery
If RBAC is weak, the attacker may
* enumerate the cluster
* discover Secrets
* retrieve credentials
* identify nodes
### Objective
Deploy cryptocurrency miners. Consume CPU. Remain hidden. Generate revenue. Notice something interesting. The cluster itself wasn't the objective. The cluster was merely the infrastructure.
### Engineering Insight
Kubernetes clusters represent enormous pools of compute. To attackers, every compromised cluster is someone else's cloud bill.
* * *
## Case Study 2 — Scarleteel 2.0
One of the most fascinating cloud attacks discussed during the meetup was **Scarleteel 2.0**.
Unlike Kinsing, Scarleteel focused on cloud identities.
### Attack Chain
```text
Internet
↓
Application Vulnerability
↓
Container
↓
AWS Credentials
↓
S3 Enumeration
↓
Privilege Escalation
↓
Persistence
↓
Cryptomining
```
Notice what happened. The Kubernetes cluster wasn't the final target. AWS was. This illustrates something extremely important. Modern Kubernetes assessments cannot stop at Kubernetes. Cloud identity becomes part of the attack surface. The training deck uses Scarleteel 2.0 to demonstrate how a compromise can extend beyond the cluster into the surrounding cloud environment when credentials are improperly managed.
## The Modern Attack Chain
Professional attackers don't think in CVEs. They think in graphs.
```text
Internet
↓
Web Application
↓
Container
↓
Service Account
↓
RBAC
↓
Secrets
↓
Cloud Credentials
↓
Cloud Account
```
Every arrow represents trust.
### Engineering Insight
Cloud-native attacks rarely require sophisticated exploitation. They require understanding how systems trust one another.
## What We Learned Throughout This Series
### Bare Metal
**Problem**: Poor utilization. **Solution**: Virtualization.
### Virtual Machines
**Problem:** Heavy operating systems. **Solution**: Containers.
### Containers
**Problem**: Packaging. **Solution**: Docker.
### Docker
**Problem**: Operations. **Solution**: Kubernetes.
### Kubernetes
**Problem**: Distributed infrastructure. **Solution**: Automation.
### Automation
**Problem**: Scale. **Solution**: Controllers.
### Controllers
**Problem**: Trust. **Solution**: RBAC.
### RBAC
**Problem**: Identity. **Solution**: Least Privilege.
### Least Privilege
**Problem**: Attack Paths. **Solution**: Defense in Depth.
Notice something. The entire history of infrastructure is one long sequence of
```shell
Problem
↓
Solution
↓
New Problem
```
Every engineering decision introduced a new attack surface.
## Kubernetes Security Isn't Container Security
This is probably the biggest takeaway from the entire series. Kubernetes security isn't about containers.
It's about
* Identity
* Authorization
* Trust
* Supply Chain
* Networking
* Infrastructure as Code
* Cloud Platforms
* Runtime Behavior
Containers simply happen to be where applications execute.
## Future Trends
Cloud-native security continues to evolve. Several emerging technologies are reshaping Kubernetes security:
### eBPF
Moving beyond traditional kernel modules for deep runtime visibility.
### Confidential Containers
Protecting workloads even from the underlying infrastructure.
### SPIFFE & SPIRE
Establishing strong workload identities for Zero Trust architectures.
### Sigstore & Cosign
Verifying the authenticity and integrity of container images.
### AI-Assisted Security
Using machine learning to identify anomalous cluster behavior, prioritize alerts, and accelerate investigations—while recognizing that human expertise remains essential for validating findings and understanding business context.
* * *
## Meetup Highlight
Throughout the BreachForce session, Abhishek consistently reinforced a central idea:
> **Misconfigurations are often more dangerous than vulnerabilities.**
The demonstrations showed that seemingly minor issues—excessive RBAC permissions, privileged containers, exposed APIs, weak identity management, or improperly handled secrets—can be chained together into realistic attack paths. Preventing those chains requires understanding both Kubernetes' engineering design and its security implications.
* * *
# Final Thoughts
When we began this journey, Kubernetes looked like another DevOps platform.
By now, it should be clear that it is much more than that. Kubernetes is a distributed operating system for modern applications.
Securing it demands more than vulnerability scanning or compliance checklists. It requires understanding how the platform was engineered, how its components communicate, where trust is established, and how attackers exploit those relationships.
For penetration testers, Kubernetes presents a shift from exploiting individual servers to evaluating complex systems of identity, automation, and policy.
For defenders, the objective is not to eliminate every risk—that is rarely possible—but to reduce unnecessary trust, enforce least privilege, build secure defaults into Infrastructure as Code, and continuously validate that those controls remain effective as the environment evolves.
The BreachForce meetup led by **Abhishek Pal** reminded us that successful Kubernetes security is not defined by the number of tools deployed, but by the depth of understanding behind the people using them. That lesson extends beyond Kubernetes itself and reflects a broader truth in cybersecurity:
> **Technology changes rapidly. Engineering principles endure.**
* * *
## References
The complete published version should conclude with a curated references section, including:
* [Official Kubernetes Documentation](https://kubernetes.io/docs/home/)
* [CNCF Kubernetes Security Documentation](https://www.cncf.io/blog/2021/03/22/kubernetes-security/)
* [NIST SP 800-190: *Application Container Security Guide*](https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-190.pdf)
* [CIS Kubernetes Benchmark](https://www.cisecurity.org/benchmark/kubernetes)
* [NSA/CISA Kubernetes Hardening Guidance](https://media.defense.gov/2022/Aug/29/2003066362/-1/-1/0/CTR_KUBERNETES_HARDENING_GUIDANCE_1.2_20220829.PDF)
* [MITRE ATT&CK for Containers](https://attack.mitre.org/matrices/enterprise/containers/)
* [Google's *Borg, Omega, and Kubernetes* research papers](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44843.pdf)
* [Kubernetes Enhancement Proposals (KEPs)](https://www.kubernetes.dev/resources/keps/) relevant to Pod Security Admission, CRI, and related topics
* Abhishek Pal's BreachForce training deck.