Netassert v2: Network Security Testing

🎉🎉🎉 Introducing Netassert V2! 🎉🎉🎉

NetAssert is a command line tool that enables you to check the network connectivity between Kubernetes objects such as Pods, Deployments, DaemonSets, and StatefulSets, as well as test their connectivity to remote hosts or IP addresses. This enables Network Policy and firewall testing to ensure that yesterday’s requirements are consistent with tomorrow’s changes.

NetAssert v2 is written in Go and is a complete rewrite of the original NetAssert utility. Under the hood NetAssert V2 utilises the ephemeral container support in Kubernetes to verify network connectivity between various Kubernetes resources. Currently, it only supports TCP and UDP protocols.

NetAssert v2 has its own test specification and uses YAML format for defining tests, reading them from either a single file or from a directory of tests.

This article is a technical deep-dive into the tool and its usage.

Components

NetAssert V2 has three main components:

NetAssert: This is responsible for orchestrating the tests and is also known as Netassert-Engine or simply engine.
NetAssertv2-packet-sniffer: This is the sniffer component that is utilised only during a UDP test and is injected to the destination/target Pod as an ephemeral container.
NetAssertv2-l4-client: This is the scanner component that is injected as the scanner ephemeral container onto the source Pod and is utilised during both TCP and UDP tests.

NetAssert v2 employs a scanner container that can perform a TCP connectivity test without needing any privileges. However, for UDP scanning, a sniffer ephemeral container is injected into the target Pod, which requires “NET_RAW” capabilities to capture data from a network interface.

While conducting UDP testing, NetAssert v2 deploys two ephemeral container images, namely the scanner and sniffer, which are injected as ephemeral containers into source and destination Pods. For any single test, NetAssert v2 injects the above container images as ephemeral containers and configures them using environment variables. The list of environment variables that are used can be found here and here and can be considered as the API contract between the NetAssert engine and the container images. Netassert also provides the flexibility to override the sniffer and scanner images from the command line during a run. Therefore, one can also bring their own container image(s) as long as they support the same environment variables.

Test Specification

NetAssert v2 tests are defined as YAML documents. Each YAML file should contain at least one test. A NetAssert v2 test should adhere to the following specification:

A YAML document is a list of NetAssert test. Each test has the following keys:

name: test # the name of the connection
type: k8s # the type of connection, only "k8s" is supported at this time
protocol: tcp # the protocol used for the connection, which must be "tcp" or "udp"
targetPort: 8080 # the target port used by the connection
timeoutSeconds: 67 # the timeout, in seconds, for the connection
attempts: 3 # the number of connection attempts for the test
exitCode: 0 # the expected exit code from the ephemeral/debug container(s)
src: # the source Kubernetes resource
  k8sResource:
    kind: deployment # the kind of the Kubernetes resource, which can be deployment, statefulset, daemonset or pod
    name: busybox # the name of the Kubernetes resource
    namespace: busybox # the namespace of the Kubernetes resource
dst: # the destination Kubernetes resource or host, **which can have one of the the following keys** i.e both `k8sResource` and `host` **are not supported at the same time** :
  host: # type host or node or machine
    name: 0.0.0.0 # the name or IP address of the host/node. (Note: Only allowed when protocol is "tcp" or "udp", but not both at the same time)
  k8sResource:
    kind: deployment # the kind of the Kubernetes resource, which can be deployment, statefulset, daemonset or pod
    name: echoserver # the name of the Kubernetes resource
    namespace: echoserver # the namespace of the Kubernetes resource

A valid sample NetAssert v2 test based on the above specification looks like the following:

Expand

---
- name: busybox-deploy-to-echoserver-deploy
  type: k8s
  protocol: tcp
  targetPort: 8080
  timeoutSeconds: 67
  attempts: 3
  exitCode: 0
  src:
    k8sResource:
      kind: deployment
      name: busybox
      namespace: busybox
  dst:
    k8sResource:
      kind: deployment
      name: echoserver
      namespace: echoserver
#######
#######
- name: busybox-deploy-to-core-dns
  type: k8s
  protocol: udp
  targetPort: 53
  timeoutSeconds: 67
  attempts: 3
  exitCode: 0
  src:
    k8sResource:
      kind: deployment
      name: busybox
      namespace: busybox
  dst:
    k8sResource:
      kind: deployment
      name: coredns
      namespace: kube-system
######
######
- name: busybox-deploy-to-web-statefulset
  type: k8s
  protocol: tcp
  targetPort: 80
  timeoutSeconds: 67
  attempts: 3
  exitCode: 0
  src:
    k8sResource: # this is type endpoint
      kind: deployment
      name: busybox
      namespace: busybox
  dst:
    k8sResource: ## this is type endpoint
      kind: statefulset
      name: web
      namespace: web
###
###
- name: fluentd-daemonset-to-web-statefulset
  type: k8s
  protocol: tcp
  targetPort: 80
  timeoutSeconds: 67
  attempts: 3
  exitCode: 0
  src:
    k8sResource: # this is type endpoint
      kind: daemonset
      name: fluentd
      namespace: fluentd
  dst:
    k8sResource: ## this is type endpoint
      kind: statefulset
      name: web
      namespace: web
###
####
- name: busybox-deploy-to-control-plane-dot-io
  type: k8s
  protocol: tcp
  targetPort: 80
  timeoutSeconds: 67
  attempts: 3
  exitCode: 0
  src:
    k8sResource: # type endpoint
      kind: deployment
      name: busybox
      namespace: busybox
  dst:
    host: # type host or node or machine
      name: control-plane.io
###
###
- name: test-from-pod1-to-pod2
  type: k8s
  protocol: tcp
  targetPort: 80
  timeoutSeconds: 67
  attempts: 3
  exitCode: 0
  src:
    k8sResource: ##
      kind: pod
      name: pod1
      namespace: pod1
  dst:
    k8sResource:
      kind: pod
      name: pod2
      namespace: pod2
###
###
- name: busybox-deploy-to-fake-host
  type: k8s
  protocol: tcp
  targetPort: 333
  timeoutSeconds: 67
  attempts: 3
  exitCode: 1
  src:
    k8sResource: # type endpoint
      kind: deployment
      name: busybox
      namespace: busybox
  dst:
    host: # type host or node or machine
      name: 0.0.0.0

Detailed steps/flow of tests

All the tests are read from an YAML file or a directory (step 1) and the results are written following the TAP format (step 5 for UDP and step 4 for TCP). The tests are performed in two different manners depending on whether a TCP or UDP protocol is defined in the test spec.

UDP Test Steps

Validate the test spec and ensure that the src and dst fields are correct, for UDP tests both of them must be of type k8sResource.
Find a running Pod called dstPod in the object defined by the dst.k8sResource field. Ensure that the Pod is in running state and has an IP address allocated by the CNI.
Find a running Pod called srcPod in the object defined by the src.k8sResource field. Ensure that the Pod is in running state and has an IP address allocated by the CNI.
Generate a random UUID string, which will be used by both ephemeral containers.
Inject the netassert-l4-client as an ephemeral container in the srcPod (step 2) and set the port and protocol according to the test specifications. Provide also the target host equal to the previously found dstPod IP address, and the random UUID that was generated in the previous step as the message to be sent over the UDP connection. At the same time, inject the netassertv2-packet-sniffer (step 3) as an ephemeral container in the dstPod using the protocol, search string, number of matches and timeout defined in the test specifications. The SEARCH_STRING environment variable is equal to the UUID that was generated in the previous step which is expected to be found in the data sent by the scanner when the connections are successful.
Poll that status of the ephemeral containers (step 4) and
1. Ensure that the netassertv2-packet-sniffer ephemeral sniffer container’s exit status matches the one defined in the test specification.
2. Ensure that the netassert-l4-client, exits with exit status of zero. This should always be the case as UDP is not a connection oriented protocol.

TCP Test Steps

Validate the test spec and ensure that the src field is of type k8sResource.
Find a running Pod called srcPod in the object defined by the src.k8sResource field. Ensure that the Pod is in running state and has an IPAddress.
Check if the dst field has k8sResource defined as a child object; If so then find a running Pod defined by the dst.K8sResource.
Inject the netassert-l4-client as an ephemeral container in the srcPod (step 2). Configure the netassert-l4-client similarly to the UDP test case. If the dst field is set to host then use the host name field as the scanner target host.
Poll that status of the ephemeral container (step 3) and ensure that the exit code of that container matches the exitCode field defined in the test specification.

Compatibility

NetAssert V2 has been tested with the following flavors of Kubernetes:

K8s Distribution	Version	CNI	Working
AWS EKS	1.25	AWS VPC CNI	Yes
AWS EKS	1.24	AWS VPC CNI	Yes
AWS EKS	1.25	Calico Version 3.25	Yes
AWS EKS	1.24	Calico version 3.25	Yes
GCP GKE	1.24	GCP VPC CNI	Yes
GCP GKE	1.24	GCP Cilium 1.11 (Dataplane v2)	Yes

RBAC Privileges

NetAssert v2 requires following Kubernetes RBAC privileges to work:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: netassert
rules:
  - apiGroups:
      - ""
      - "apps"
    resources:
      - deployments
      - statefulsets
      - daemonsets
      - pods
    verbs:
      - get
  ##
  - apiGroups:
      - ""
      - "apps"
    resources:
      - replicasets
      - pods
    verbs:
      - list
  ##
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/ephemeralcontainers
    verbs:
      - watch
      - patch

The above role can then be bound to a “principal” either through a RoleBinding or a ClusterRoleBinding, depending on whether the scope of the role is supposed to be namespaced or not.

Limitations

Requires ephemeral container support in the Kubernetes version

When performing UDP scanning, the sniffer container needs the CAP_NET_RAW Linux capability so that it can read packets from the network interface. As a result, admission controllers or other security mechanisms must be modified to allow the sniffer image to run with this capability. Currently, the sniffer ephemeral container looks like the following:

ephemeralContainers:
  - env:
      - name: TIMEOUT_SECONDS
        value: "72"
      - name: IFACE
        value: eth0
      - name: SNAPLEN
        value: "1024"
      - name: SEARCH_STRING
        value: f679595c-dac3-11ed-adb8-70321792e6f9
      - name: PROTOCOL
        value: udp
      - name: MATCHES
        value: "1"
    image: docker.io/controlplane/netassertv2-packet-sniffer:latest
    imagePullPolicy: Always
    name: netassertv2-sniffer-i7cfugzj6
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
          - NET_RAW
      runAsNonRoot: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File

Although they do not consume any resources, ephemeral containers that are injected as part of the test(s) by NetAssert will remain in the Pod specification
Service mesh testing is currently not supported