By Prithak Sharma
πππ Introducing Netassert V2! πππ
NetAssert is a command line tool that enables you to check the network connectivity between Kubernetes objects such as Pods, Deployments, DaemonSets, and StatefulSets, as well as test their connectivity to remote hosts or IP addresses. This enables Network Policy and firewall testing to ensure that yesterday’s requirements are consistent with tomorrow’s changes.
NetAssert v2 is written in Go and is a complete rewrite of the original NetAssert utility. Under the hood NetAssert V2 utilises the ephemeral container support in Kubernetes to verify network connectivity between various Kubernetes resources. Currently, it only supports TCP and UDP protocols.
NetAssert v2 has its own test specification and uses YAML format for defining tests, reading them from either a single file or from a directory of tests.
This article is a technical deep-dive into the tool and its usage.
NetAssert V2 has three main components:
NetAssert v2 employs a scanner container that can perform a TCP connectivity test without needing any privileges. However, for UDP scanning, a sniffer ephemeral container is injected into the target Pod, which requires “NET_RAW” capabilities to capture data from a network interface.
While conducting UDP testing, NetAssert v2 deploys two ephemeral container images, namely the scanner and sniffer, which are injected as ephemeral containers into source and destination Pods. For any single test, NetAssert v2 injects the above container images as ephemeral containers and configures them using environment variables. The list of environment variables that are used can be found here and here and can be considered as the API contract between the NetAssert engine and the container images. Netassert also provides the flexibility to override the sniffer and scanner images from the command line during a run. Therefore, one can also bring their own container image(s) as long as they support the same environment variables.
NetAssert v2 tests are defined as YAML documents. Each YAML file should contain at least one test. A NetAssert v2 test should adhere to the following specification:
NetAssert
test. Each test has the following keys:name: test # the name of the connection
type: k8s # the type of connection, only "k8s" is supported at this time
protocol: tcp # the protocol used for the connection, which must be "tcp" or "udp"
targetPort: 8080 # the target port used by the connection
timeoutSeconds: 67 # the timeout, in seconds, for the connection
attempts: 3 # the number of connection attempts for the test
exitCode: 0 # the expected exit code from the ephemeral/debug container(s)
src: # the source Kubernetes resource
k8sResource:
kind: deployment # the kind of the Kubernetes resource, which can be deployment, statefulset, daemonset or pod
name: busybox # the name of the Kubernetes resource
namespace: busybox # the namespace of the Kubernetes resource
dst: # the destination Kubernetes resource or host, **which can have one of the the following keys** i.e both `k8sResource` and `host` **are not supported at the same time** :
host: # type host or node or machine
name: 0.0.0.0 # the name or IP address of the host/node. (Note: Only allowed when protocol is "tcp" or "udp", but not both at the same time)
k8sResource:
kind: deployment # the kind of the Kubernetes resource, which can be deployment, statefulset, daemonset or pod
name: echoserver # the name of the Kubernetes resource
namespace: echoserver # the namespace of the Kubernetes resource
A valid sample NetAssert v2 test based on the above specification looks like the following:
---
- name: busybox-deploy-to-echoserver-deploy
type: k8s
protocol: tcp
targetPort: 8080
timeoutSeconds: 67
attempts: 3
exitCode: 0
src:
k8sResource:
kind: deployment
name: busybox
namespace: busybox
dst:
k8sResource:
kind: deployment
name: echoserver
namespace: echoserver
#######
#######
- name: busybox-deploy-to-core-dns
type: k8s
protocol: udp
targetPort: 53
timeoutSeconds: 67
attempts: 3
exitCode: 0
src:
k8sResource:
kind: deployment
name: busybox
namespace: busybox
dst:
k8sResource:
kind: deployment
name: coredns
namespace: kube-system
######
######
- name: busybox-deploy-to-web-statefulset
type: k8s
protocol: tcp
targetPort: 80
timeoutSeconds: 67
attempts: 3
exitCode: 0
src:
k8sResource: # this is type endpoint
kind: deployment
name: busybox
namespace: busybox
dst:
k8sResource: ## this is type endpoint
kind: statefulset
name: web
namespace: web
###
###
- name: fluentd-daemonset-to-web-statefulset
type: k8s
protocol: tcp
targetPort: 80
timeoutSeconds: 67
attempts: 3
exitCode: 0
src:
k8sResource: # this is type endpoint
kind: daemonset
name: fluentd
namespace: fluentd
dst:
k8sResource: ## this is type endpoint
kind: statefulset
name: web
namespace: web
###
####
- name: busybox-deploy-to-control-plane-dot-io
type: k8s
protocol: tcp
targetPort: 80
timeoutSeconds: 67
attempts: 3
exitCode: 0
src:
k8sResource: # type endpoint
kind: deployment
name: busybox
namespace: busybox
dst:
host: # type host or node or machine
name: control-plane.io
###
###
- name: test-from-pod1-to-pod2
type: k8s
protocol: tcp
targetPort: 80
timeoutSeconds: 67
attempts: 3
exitCode: 0
src:
k8sResource: ##
kind: pod
name: pod1
namespace: pod1
dst:
k8sResource:
kind: pod
name: pod2
namespace: pod2
###
###
- name: busybox-deploy-to-fake-host
type: k8s
protocol: tcp
targetPort: 333
timeoutSeconds: 67
attempts: 3
exitCode: 1
src:
k8sResource: # type endpoint
kind: deployment
name: busybox
namespace: busybox
dst:
host: # type host or node or machine
name: 0.0.0.0
All the tests are read from an YAML file or a directory (step 1) and the results are written following the TAP format (step 5 for UDP and step 4 for TCP). The tests are performed in two different manners depending on whether a TCP or UDP protocol is defined in the test spec.
NetAssert V2 has been tested with the following flavors of Kubernetes:
K8s Distribution | Version | CNI | Working |
---|---|---|---|
AWS EKS | 1.25 | AWS VPC CNI | Yes |
AWS EKS | 1.24 | AWS VPC CNI | Yes |
AWS EKS | 1.25 | Calico Version 3.25 | Yes |
AWS EKS | 1.24 | Calico version 3.25 | Yes |
GCP GKE | 1.24 | GCP VPC CNI | Yes |
GCP GKE | 1.24 | GCP Cilium 1.11 (Dataplane v2) | Yes |
NetAssert v2 requires following Kubernetes RBAC privileges to work:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: netassert
rules:
- apiGroups:
- ""
- "apps"
resources:
- deployments
- statefulsets
- daemonsets
- pods
verbs:
- get
##
- apiGroups:
- ""
- "apps"
resources:
- replicasets
- pods
verbs:
- list
##
- apiGroups:
- ""
resources:
- pods
- pods/ephemeralcontainers
verbs:
- watch
- patch
The above role can then be bound to a “principal” either through a RoleBinding or a ClusterRoleBinding, depending on whether the scope of the role is supposed to be namespaced or not.
Requires ephemeral container support in the Kubernetes version
When performing UDP scanning, the sniffer container needs the CAP_NET_RAW Linux capability so that it can read packets from the network interface. As a result, admission controllers or other security mechanisms must be modified to allow the sniffer image to run with this capability. Currently, the sniffer ephemeral container looks like the following:
ephemeralContainers:
- env:
- name: TIMEOUT_SECONDS
value: "72"
- name: IFACE
value: eth0
- name: SNAPLEN
value: "1024"
- name: SEARCH_STRING
value: f679595c-dac3-11ed-adb8-70321792e6f9
- name: PROTOCOL
value: udp
- name: MATCHES
value: "1"
image: docker.io/controlplane/netassertv2-packet-sniffer:latest
imagePullPolicy: Always
name: netassertv2-sniffer-i7cfugzj6
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_RAW
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
Although they do not consume any resources, ephemeral containers that are injected as part of the test(s) by NetAssert will remain in the Pod specification
Service mesh testing is currently not supported