STIG and NSA/CISA Hardening

By default, Tanzu Kubernetes Grid (TKG) workload clusters deployed by standalone management clusters are hardened to the levels shown in STIG Results and Exceptions and CIS Results and Exceptions.

In TKG v2.5, Ubuntu 22.04 and Photon 5 OS nodes are hardened to CIS and STIG standards without requiring additional steps.

• Photon 5 nodes are CIS- and STIG-hardened.
• Ubuntu 22.04 nodes are CIS-hardened and hardened for STIG per Ubuntu 20.04 controls because STIG specifications have not yet been released for Ubuntu 22.04, as of the TKG v2.5.0 release date.

This topic explains how to further harden clusters that are not based on Ubuntu 22.04 or Photon 5 OS.

The methods depend on whether the cluster is class-based or plan-based, as described as described in Workload Cluster Types.

TKG releases are continuously validated against the Defense Information Systems Agency (DISA) Kubernetes Security Technical Implementation Guide (STIG) and NSA/CISA Kubernetes Hardening Guide.

Additional Kubernetes Hardening in Class-Based Workload Clusters

To increase the STIG- and CIS-compliance of Kubernetes in class-based workload clusters, configure them as described in the sections below.

For more granular exception handling, you can follow the workarounds listed in the exceptions tables in STIG Results and Exceptions and CIS Results and Exceptions.

STIG Hardening

To harden Kubernetes in class-based workload clusters to STIG standards, do either of the following before creating the cluster:

• Include the variable settings below in the cluster configuration file

• Set the variables in your local environment, for example with export

  ETCD_EXTRA_ARGS: "auto-tls=false;peer-auto-tls=false;cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384"
  KUBE_CONTROLLER_MANAGER_EXTRA_ARGS: "tls-min-version=VersionTLS12;profiling=false;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384"
  WORKER_KUBELET_EXTRA_ARGS: "streaming-connection-idle-timeout=5m;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384;protect-kernel-defaults=true"
  APISERVER_EXTRA_ARGS: "tls-min-version=VersionTLS12;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384"
  KUBE_SCHEDULER_EXTRA_ARGS: "tls-min-version=VersionTLS12;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384"
  CONTROLPLANE_KUBELET_EXTRA_ARGS: "streaming-connection-idle-timeout=5m;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384;protect-kernel-defaults=true"
  ENABLE_AUDIT_LOGGING: true
  
  

NSA/CISA Hardening

To harden Kubernetes in class-based workload clusters to CIS standards, do the following before creating the cluster:

1. Review the Event Rate limit configuration below. If you want to change any settings, save the code to event-rate-config.yaml and change the settings as desired:

   apiVersion: eventratelimit.admission.k8s.io/v1alpha1
   kind: Configuration
   limits:
   - type: Namespace
     qps: 50
     burst: 100
     cacheSize: 2000
   - type: User
     qps: 10
     burst: 50
   

2. If you created event-rate-config.yaml with custom settings, base64-encode the file by running the following and recording the output string:

   • Linux: base64 -w 0 event-rate-config.yaml
   • Mac: base64 -b 0 event-rate-config.yaml

3. Do either of the following before creating the cluster:

   • Include the variable settings below in the cluster configuration file

   • Set the variables in your local environment, for example with export

     ETCD_EXTRA_ARGS: "auto-tls=false;peer-auto-tls=false;cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
     KUBE_CONTROLLER_MANAGER_EXTRA_ARGS: "profiling=false;terminated-pod-gc-threshold=500;tls-min-version=VersionTLS12;profiling=false;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
     WORKER_KUBELET_EXTRA_ARGS: "read-only-port=0;authorization-mode=Webhook;client-ca-file=/etc/kubernetes/pki/ca.crt;event-qps=0;make-iptables-util-chains=true;streaming-connection-idle-timeout=5m;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256;protect-kernel-defaults=true"
     APISERVER_EXTRA_ARGS: "enable-admission-plugins=AlwaysPullImages,NodeRestriction;profiling=false;service-account-lookup=true;tls-min-version=VersionTLS12;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
     KUBE_SCHEDULER_EXTRA_ARGS: "profiling=false;tls-min-version=VersionTLS12;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
     CONTROLPLANE_KUBELET_EXTRA_ARGS: "read-only-port=0;authorization-mode=Webhook;client-ca-file=/etc/kubernetes/pki/ca.crt;event-qps=0;make-iptables-util-chains=true;streaming-connection-idle-timeout=5m;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256;protect-kernel-defaults=true"
     APISERVER_EVENT_RATE_LIMIT_CONF_BASE64: "<EVENT-RATE-CONFIG>"
     ENABLE_AUDIT_LOGGING: true
     

   Where <EVENT-RATE-CONFIG> is the base64-encoded value from the last step, or the following if you did not change the Event Rate limit configuration:

   • YXBpVmVyc2lvbjogZXZlbnRyYXRlbGltaXQuYWRtaXNzaW9uLms4cy5pby92MWFscGhhMQpraW5kOiBDb25maWd1cmF0aW9uCmxpbWl0czoKLSB0eXBlOiBOYW1lc3BhY2UKICBxcHM6IDUwCiAgYnVyc3Q6IDEwMAogIGNhY2hlU2l6ZTogMjAwMAotIHR5cGU6IFVzZXIKICBxcHM6IDEwCiAgYnVyc3Q6IDUwCg==

Additional OS Hardening in Class-Based Workload Clusters

To harden Ubuntu OS v20.04 in class-based workload clusters to STIG or CIS standards, create custom hardened VM images for the clusters by running Image Builder with the ansible_user_vars settings for STIG or CIS hardening, as described in Build a Linux Image

Hardening Plan-Based Workload Clusters

Legacy, non class-based TKG workload clusters deployed by standalone management clusters can be hardened by using ytt overlays. For information on how to customize plan-based TKG clusters using ytt, see Legacy Cluster Configuration with ytt.

You can create legacy clusters by setting allow-legacy-cluster to true in your CLI configuration as describe in Features in Tanzu CLI Architecture and Configuration.

STIG Hardening

To further harden plan-based TKG clusters, VMware provides a STIG hardening ytt overlay.

The following snippet is an ytt overlay to set tls-min-version (STIG: V-242378) on the api-server.

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")
#@overlay/match missing_ok=True,by=overlay.subset({"kind":"KubeadmControlPlane"})
---
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
        #@overlay/match missing_ok=True
        tls-min-version: VersionTLS12

NSA/CISA Hardening

To further harden plan-based TKG clusters for NSA/CISA, VMware provides the following Antrea object specifications:

• NSA/CISA hardening: Antrea ClusterNetworkPolicies:

  The following Antrea ClusterNetworkPolicies specification for Network policies control sets a default policy for all Pods to deny all ingress and egress traffic and ensure that any unselected Pods are isolated.

  apiVersion: security.antrea.tanzu.vmware.com/v1alpha1
  kind: ClusterNetworkPolicy
  metadata:
    name: default-deny
  spec:
    priority: 150
    tier: baseline
    appliedTo:
      - namespaceSelector: {}
    ingress:
      - action: Drop              # For all Pods in every namespace, drop and log all ingress traffic from anywhere
        name: drop-all-ingress
        enableLogging: true
    egress:
      - action: Drop              # For all Pods in every namesapces, drop and log all egress traffic towards anywhere
        name: drop-all-egress
        enableLogging: true
  

• NSA/CISA hardening: Antrea network policy:

  The following Antrea network policy allows tanzu-capabilities-manager egress to kube-apiserver ports 443 and 6443.

  apiVersion: security.antrea.tanzu.vmware.com/v1alpha1
  kind: NetworkPolicy
  metadata:
    name: tanzu-cm-apiserver
    namespace: tkg-system
  spec:
    priority: 5
    tier: securityops
    appliedTo:
      - podSelector:
          matchLabels:
          app: tanzu-capabilities-manager
    egress:
      - action: Allow
        to:
        - podSelector:
            matchLabels:
            component: kube-apiserver
          namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
        ports:
        - port: 443
          protocol: TCP
        - port: 6443
          protocol: TCP
        name: AllowToKubeAPI
  

• NSA/CISA hardening: OPA template and constraints:

  The following example of uses OPA gatekeeper to restrict allowed images repositories.

  • OPA template:

    apiVersion: templates.gatekeeper.sh/v1beta1
    kind: ConstraintTemplate
    metadata:
      name: k8sallowedrepos
      annotations:
        description: Requires container images to begin with a repo string from a specified
          list.
    spec:
      crd:
        spec:
          names:
            kind: K8sAllowedRepos
          validation:
            # Schema for the `parameters` field
            openAPIV3Schema:
              type: object
              properties:
                repos:
                  type: array
                  items:
                    type: string
      targets:
        - target: admission.k8s.gatekeeper.sh
          rego:|
            package k8sallowedrepos
    
            violation[{"msg": msg}] {
              container := input.review.object.spec.containers[_]
              satisfied := [good| repo = input.parameters.repos[_] ; good = startswith(container.image, repo)]
              not any(satisfied)
              msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
            }
    
            violation[{"msg": msg}] {
              container := input.review.object.spec.initContainers[_]
              satisfied := [good| repo = input.parameters.repos[_] ; good = startswith(container.image, repo)]
              not any(satisfied)
              msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
            }
    

  • OPA constraints:

    apiVersion: constraints.gatekeeper.sh/v1beta1
    kind: K8sAllowedRepos
    metadata:
      name: repo-is-openpolicyagent
    spec:
      match:
        kinds:
          - apiGroups: [""]
            kinds: ["Pod"]
      parameters:
        repos:
          - "<ALLOWED_IMAGE_REPO>"
    

• NSA/CISA hardening: OPA mutations:

  The following example uses OPA mutation to set allowPrivilegeEscalation to false if it is missing in the pod spec.

  apiVersion: mutations.gatekeeper.sh/v1alpha1
  kind: Assign
  metadata:
    name: allow-privilege-escalation
  spec:
      match:
        scope: Namespaced
        kinds:
          - apiGroups: ["*"]
            kinds: ["Pod"]
        excludedNamespaces:
        - kube-system
      applyTo:
      - groups: [""]
        kinds: ["Pod"]
        versions: ["v1"]
      location: "spec.containers[name:*].securityContext.allowPrivilegeEscalation"
      parameters:
        pathTests:
        - subPath: "spec.containers[name:*].securityContext.allowPrivilegeEscalation"
          condition: MustNotExist
        assign:
          value: false
  

Antrea CNI is used in this guide for network hardening as it provides fine-grained control of network policies using tiers and the ability to apply a cluster-wide security policy using ClusterNetworkPolicy.

Open Policy Agent (OPA) is used instead of pod security policies as they were deprecated in Kubernetes 1.21.

ytt overlays, network policies and OPA policies are created in a way to make it easy for cluster admins to opt out of hardening controls for certain workloads. We suggest not completely opting out of the hardening practices and instead isolating workloads in namespaces where these hardening controls are not applied. Opting out of hardening controls also depends on the risk appetite of the TKG deployment.

NSA/CISA Kubernetes Hardening Guidance