Configuration File to Cluster Class Variable Translations

This topic explains how variables in workload cluster configuration files translate to variable settings within a class-based Cluster objects and their ClusterBootstrap and other subordinate objects in Tanzu Kubernetes Grid (TKG). For an example Cluster object and its subordinate objects, see Example Cluster Object and Its Subordinate Objects below.

Overview: Configuration File and Cluster Class Variables

Configuration file variables and class-based cluster variables differ:

  • Configuration file variables are:

  • Class-based cluster variable settings are nested YAML settings within Cluster or ClusterBootstrap object specs:

    • Cluster object settings are:

      • Most of the cluster settings available, and can be changed arbitrarily.
      • Defined by the ClusterClass object set as the cluster’s spec.topology.class value.
      • Set in the controlPlane, workers or variables blocks under spec.topology.

      • Example setting:

        variables:
- name: imageRepository
  value:
    host: stg-project.vmware.com
    • ClusterBootstrap settings are a few one-time settings for container networking and other low-level infrastructure that cannot be changed in an existing cluster.

When you run the tanzu cluster create command with a cluster configuration file as described in Create a Class-Based Cluster, the Tanzu CLI performs this translation.

Node Configuration

The table below lists variables for configuring the control plane and worker nodes, and the operating system that the node instances run. For information about the variables in the Configuration File Variables column, see Node Configuration in Configuration File Variable Reference.

In the Class-based Cluster Object Structure column, all name / value pair settings are located in the Cluster object definition under spec.topology.variables. The example code in this column applies to Cluster objects on vSphere; object structure on AWS and Azure may differ.

Configuration File Variables Class-based Cluster Object Structure 
 OS_NAME: ubuntu OS_VERSION: 20.04 OS_ARCH: amd64
 
 metadata: annotations: osInfo: ubuntu,20.04,amd64 
 CONTROL_PLANE_MACHINE_COUNT: 3 
 controlPlane: … replicas: 3 
 CONTROLPLANE_SIZE: large SIZE: large
Values translate to machine settings based on infrastructure.
SIZE applies to both control plane and workers.		 
 - name: controlPlane value: machine: diskGiB: 40 memoryMiB: 16384 numCPUs: 4 
 CONTROL_PLANE_NODE_SEARCH_DOMAINS: corp.local, example.com 
 - name: controlPlane value: … network: … searchDomains: - corp.local - example.com 
 CONTROL_PLANE_NODE_LABELS: ‘key1=value1,key2=value2’ 
 - name: controlPlane value: … nodeLabels: - key: key1 value: value1 - key: key2 value: value2 
 WORKER_MACHINE_COUNT: 6 
 workers: machineDeployments: - class: tkg-worker … replicas: 6 
 WORKER_SIZE: extra-large SIZE: extra-large
Values translate to machine settings based on infrastructure.
SIZE applies to both control plane and workers.		 
 - name: worker value: … machine: diskGiB: 80 memoryMiB: 32768 numCPUs: 8 
 WORKER_NODE_SEARCH_DOMAINS: corp.local, example.com
 
 - name: worker value: … network: searchDomains: - corp.local - example.com 
 CUSTOM_TDNF_REPOSITORY_CERTIFICATE: “YPdeNjLW[…]”
 
 - name: customTDNFRepository value: certificate: YPdeNjLW[…] 
 WORKER_ROLLOUT_STRATEGY: RollingUpdate
 
 workers: machineDeployments: - class: tkg-worker failureDomain: “1” metadata: annotations: run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu,os-version=2004 name: md-0 replicas: 1 strategy: type: RollingUpdate

Pod Security Admission (PSA) Controller

The table below lists variables for configuring Pod Security Standards for Pod Security Admission Controller. For information about the variables in the Configuration File Variables column, see Pod Security Standards for Pod Security Admission Controller in Configuration File Variable Reference.

In the Class-based Cluster Object Structure column, all name / value pair settings are located in the Cluster object definition under spec.topology.variables.

Configuration File Variables Class-based Cluster Object Structure 
 POD_SECURITY_STANDARD_DEACTIVATED: false POD_SECURITY_STANDARD_AUDIT: privileged POD_SECURITY_STANDARD_WARN: privileged POD_SECURITY_STANDARD_ENFORCE: baseline 
 - name: podSecurityStandard value: deactivated: false audit: “privileged” enforce: “privileged” warn: “baseline” auditVersion: “v1.26” enforceVersion: “v1.26” warnVersion: “v1.26” exemptions: namespaces: [“kube-system”, “tkg-system”]

Cluster Autoscaler

When ENABLE_AUTOSCALER is true, the Tanzu CLI creates a Deployment object for Cluster Autoscaler and adds Cluster Autoscaler annotations to the Cluster object. For information about the variables in the Configuration File Variables column below, see Cluster Autoscaler in Configuration File Variable Reference.

Configuration File Variables Class-based Cluster and Autoscaler Deployment Object Structure 
 AUTOSCALER_SCALE_DOWN_DELAY_AFTER_ADD: 10m AUTOSCALER_SCALE_DOWN_DELAY_AFTER_DELETE: 10s AUTOSCALER_SCALE_DOWN_DELAY_AFTER_FAILURE: 3m AUTOSCALER_SCALE_DOWN_UNNEEDED_TIME: 10m AUTOSCALER_MAX_NODE_PROVISION_TIME: 15m AUTOSCALER_MAX_NODES_TOTAL: 0
In Cluster Autoscaler Deployment object:
 spec: containers: - args: … - –scale-down-delay-after-add=10m - –scale-down-delay-after-delete=10s - –scale-down-delay-after-failure=3m - –scale-down-unneeded-time=10m - –max-node-provision-time=15m - –max-nodes-total=0
 
 AUTOSCALER_MAX_SIZE_0: 12 AUTOSCALER_MIN_SIZE_0: 8 AUTOSCALER_MAX_SIZE_1: 10 AUTOSCALER_MIN_SIZE_1: 6 AUTOSCALER_MAX_SIZE_2: 8 AUTOSCALER_MIN_SIZE_2: 4
If AUTOSCALER_MAX_SIZE_ or AUTOSCALER_MIN_SIZE_ are not set, the annotations settings take the values of WORKER_MACHINE_COUNT_*.		 In the Cluster object, under spec.topology.workers:
 machineDeployments: - class: tkg-worker metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: “12” cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: “8” … name: md-0 - class: tkg-worker metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: “10” cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: “6” … name: md-1 …

Proxy and Private Image Registry

This section lists variables for using proxies and private image registries, for example in an internet-restricted deployment.

In the Class-based Cluster Object Structure column, all name / value pair settings are located in the Cluster object definition under spec.topology.variables.

Configuration File Variables Class-based Cluster Object Structure Notes 
 TKG_CUSTOM_IMAGE_REPOSITORY: example.com/yourproject TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY: true
 
 - name: imageRepository value: host: example.com/yourproject tlsCertificateValidation: enabled: false
 tldCertificateValidation.enabled reverses the boolean setting for TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY and is not written if its value is true. 
 TKG_HTTP_PROXY_ENABLED: true TKG_HTTP_PROXY: http://proxy.example.com:80 TKG_HTTPS_PROXY: https://proxy.example.com:3128 TKG_NO_PROXY: .noproxy.example.com,noproxy.example.com,192.168.0.0/24
 
 - name: proxy value: httpProxy: http://proxy.example.com:80 httpsProxy: https://proxy.example.com:3128 noProxy: - .noproxy.example.com - noproxy.example.com - 192.168.0.0/24 - […]
 Internally, the Tanzu CLI appends to the noProxy list values that are not set in a configuration file, as described in Proxy Configuration. 
 TKG_PROXY_CA_CERT: “LS0tLSBL[…]” TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE: MIIEpgIH[…]"
 
 - name: trust value: - name: proxy data: LS0tLSBL[…] - name: imageRepository data: MIIEpgIH[…]
 Values are base64-encoded CA certificates.
TKG_PROXY_CA_CERT takes precedence over TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE, and trust imageRespository value is not written if trust proxy value is set.		 
 ADDITIONAL_IMAGE_REGISTRY_1: “example.com/otherregistry-1” ADDITIONAL_IMAGE_REGISTRY_1_SKIP_TLS_VERIFY: false ADDITIONAL_IMAGE_REGISTRY_1_CA_CERTIFICATE: “LS0tLSBL[…]” ADDITIONAL_IMAGE_REGISTRY_2: “example.com/otherregistry-2” ADDITIONAL_IMAGE_REGISTRY_2_SKIP_TLS_VERIFY: true ADDITIONAL_IMAGE_REGISTRY_3: “example.com/otherregistry-3” ADDITIONAL_IMAGE_REGISTRY_3_SKIP_TLS_VERIFY: false ADDITIONAL_IMAGE_REGISTRY_3_CA_CERTIFICATE: “MIIEpgIH[…]”
 
 - name: additionalImageRegistries value: - caCert: LS0tLSBL[…] host: example.com/otherregistry-1 skipTlsVerify: false - host: example.com/otherregistry-2 skipTlsVerify: true - caCert: MIIEpgIH[…] host: example.com/otherregistry-3 skipTlsVerify: false

Common Variables

The following table lists variables that are common to all target platforms.

For information about the variables in the Configuration File Variables column, see Common Variables for All Target Platforms in Configuration File Variable Reference.

In the Class-based Cluster Object Structure column, all name / value pair settings are located in the Cluster object definition under spec.topology.variables.

Configuration File Variables Class-based Cluster Object Structure 
 CLUSTER_NAME: my-cluster NAMESPACE: default
 
 metadata: name: my-cluster namespace: default
 
 CLUSTER_PLAN: dev
 
 metadata: annotations: tkg/plan: dev
 
 INFRASTRUCTURE_PROVIDER: vsphere
 
 topology: class: tkg-vsphere-default-v1.0.0 
 CLUSTER_API_SERVER_PORT: 6443
 
 - name: apiServerPort value: 6443
 
 CNI: antrea
 
 - name: cni value: antrea
 
 ENABLE_AUDIT_LOGGING: true
 
 - name: auditLogging value: enabled: true
 
 CLUSTER_CIDR: 100.96.0.0/11 SERVICE_CIDR: 100.64.0.0/13
 
 spec: clusterNetwork: pods: cidrBlocks: - 100.96.0.0/11 services: cidrBlocks: - 100.64.0.0/13
 
 CONTROLPLANE_CERTIFICATE_ROTATION_ENABLED: true CONTROLPLANE_CERTIFICATE_ROTATION_DAYS_BEFORE: 65
 
 - name: controlPlaneCertificateRotation value: activate: true daysBefore: 65

Antrea CNI

This section lists variables for configuring a cluster’s Antrea container network interface (CNI). The class-based cluster variables reside in the AntreaConfig object that is referenced under the ClusterBootstrap object’s spec.cni.refName property.

Class-based clusters support many Antrea configuration options that do not have corresponding TKG configuration file variables. For all Antrea configuration options that are settable for class-based Cluster objects, see Feature Gates and other topics in the Antrea documentation.

In the AntreaConfig Object Structure column, all settings are located in the AntreaConfig object definition, under spec.antrea.config.

Configuration File Variables AntreaConfig Object Structure
 
 ANTREA_DISABLE_UDP_TUNNEL_OFFLOAD: false ANTREA_ENABLE_USAGE_REPORTING: false ANTREA_KUBE_APISERVER_OVERRIDE: “https://192.168.77.100:6443” ANTREA_NO_SNAT: false ANTREA_TRAFFIC_ENCAP_MODE: “encap” ANTREA_TRANSPORT_INTERFACE: “eth0” ANTREA_TRANSPORT_INTERFACE_CIDRS: “10.0.0.2/24”
 
 disableUdpTunnelOffload: false enableUsageReporting: false kubeAPIServerOverride: https://192.168.77.100:6443 noSNAT: false trafficEncapMode: encap transportInterface: eth0 transportInterfaceCIDRs: - 10.0.0.2/24
 
 ANTREA_EGRESS: true ANTREA_IPAM: false ANTREA_MULTICAST: false ANTREA_NETWORKPOLICY_STATS: true ANTREA_NODEPORTLOCAL: true ANTREA_SERVICE_EXTERNALIP: false ANTREA_POLICY: true ANTREA_TRACEFLOW: true
 
 featureGates: Egress: true AntreaIPAM: false Multicast: false NetworkPolicyStats: true NodePortLocal: true ServiceExternalIP: false AntreaPolicy: true AntreaTraceflow: true
 
 ANTREA_PROXY: true ANTREA_PROXY_ALL: false ANTREA_PROXY_LOAD_BALANCER_IPS: true ANTREA_PROXY_NODEPORT_ADDRS: “100.70.70.12” ANTREA_PROXY_SKIP_SERVICES: 10.11.1.2,kube-system/kube-dns
 
 antreaProxy: nodePortAddresses: - 100.70.70.12 proxyAll: false proxyLoadBalancerIPs: true skipServices: - 10.11.1.2 - kube-system/kube-dns 
 ANTREA_FLOWEXPORTER: false ANTREA_FLOWEXPORTER_ACTIVE_TIMEOUT: “60s” ANTREA_FLOWEXPORTER_COLLECTOR_ADDRESS: “flow-aggregator.svc:4739:tls” ANTREA_FLOWEXPORTER_POLL_INTERVAL: “5s” ANTREA_FLOWEXPORTER_IDLE_TIMEOUT: “15s”
 
 flowExporter: activeFlowTimeout: 60s collectorAddress: flow-aggregator.svc:4739:tls idleFlowTimeout: 15s pollInterval: 5s
 
 ANTREA_NODEPORTLOCAL_ENABLED: true ANTREA_NODEPORTLOCAL_PORTRANGE: 61000-62000
 
 nodePortLocal: enabled: true portRange: 61000-62000
 
 ANTREA_EGRESS_EXCEPT_CIDRS: “10.0.0.0/6”
 
 egress: exceptCIDRs: - 10.0.0.0/6 
 ANTREA_MULTICAST_INTERFACES: “eth0”
 
 multicastInterfaces: - eth0

vSphere

This section lists variables for deploying workload clusters to vSphere. For information about the variables in the Configuration File Variables column below, see vSphere in Configuration File Variable Reference. Settings listed in the Class-based Object Structure column are in the objects of the kind listed.

Configuration File Variables Object Structure 
 TKG_IP_FAMILY: ipv4 VSPHERE_CONTROL_PLANE_ENDPOINT: 10.237.177.161 VSPHERE_REGION: my-region VSPHERE_ZONE: my-zone
 
 kind: VSphereCPIConfig spec: vsphereCPI: ipFamily: ipv4 vmNetwork: excludeExternalSubnetCidr: 10.237.177.161/32 excludeInternalSubnetCidr: 10.237.177.161/32 zone: my-zone


kind: Cluster metadata: annotations: tkg.tanzu.vmware.com/cluster-controlplane-endpoint: 10.237.177.161 spec: topology: variables: - name: apiServerEndpoint value: 10.237.177.161
 
VSPHERE_MTU

 
 kind: Cluster spec: topology: variables: - name: controlPlane value: … network: mtu: 1500 - name: worker value: … network: mtu: 1500
 
 VSPHERE_DATACENTER: /dc0 VSPHERE_DATASTORE: /dc0/datastore/sharedVmfs-0 VSPHERE_FOLDER: /dc0/vm/folder0 VSPHERE_NETWORK: /dc0/network/VM Network VSPHERE_RESOURCE_POOL: /dc0/host/cluster0/Resources/rp0 VSPHERE_SERVER: 10.237.179 VSPHERE_STORAGE_POLICY_ID: my-local-sp VSPHERE_TEMPLATE: /dc0/vm/ubuntu-2004-kube-v1.27.5+vmware.2-tkg.1 VSPHERE_TLS_THUMBPRINT: B7:15:(…):1D:2F
 
 kind: ClusterBootstrap spec: additionalPackages: - refName: tkg-storageclass* valuesFrom: inline: VSPHERE_STORAGE_POLICY_ID: my-local-sp


kind: Cluster spec: topology: variables: - name: vcenter value: datacenter: /dc0 datastore: /dc0/datastore/sharedVmfs-0 folder: /dc0/vm/folder0 network: /dc0/network/VM Network resourcePool: /dc0/host/cluster0/Resources/rp0 server: 10.237.179.190 storagePolicyID: my-local-sp template: /dc0/vm/ubuntu-2004-kube-v1.27.5+vmware.2-tkg.1 tlsThumbprint: B7:15:(…):1D:2F
 
 VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa NzaC1yc2EA[…]==
 
 kind: Cluster spec: topology: variables: - name: user value: sshAuthorizedKeys: - ssh-rsa NzaC1yc2EA[…]== 
 VSPHERE_CONTROL_PLANE_DISK_GIB: “30” VSPHERE_CONTROL_PLANE_MEM_MIB: “2048” VSPHERE_CONTROL_PLANE_NUM_CPUS: “2” VSPHERE_WORKER_DISK_GIB: “50” VSPHERE_WORKER_MEM_MIB: “4096” VSPHERE_WORKER_NUM_CPUS: “4” 
 kind: Cluster spec: topology: variables: - name: controlPlane value: machine: diskGiB: 40 memoryMiB: 8192 numCPUs: 2 - name: worker value: machine: diskGiB: 40 memoryMiB: 8192 numCPUs: 2 
 NTP_SERVERS: time.google.com 
 kind: Cluster spec: topology: variables: - name: ntpServers value: - time.google.com 
 VSPHERE_AZ_0: rack1 VSPHERE_AZ_1: rack2 VSPHERE_AZ_2: rack3 
 kind: Cluster spec: topology: workers: machineDeployments: - class: tkg-worker failureDomain: rack1 name: md-0 - class: tkg-worker failureDomain: rack2 name: md-1 - class: tkg-worker failureDomain: rack3 name: md-2 
 CONTROL_PLANE_NODE_NAMESERVERS: “10.10.10.10,10.10.10.11” WORKER_NODE_NAMESERVERS: “10.10.10.10,10.10.10.11”
 
 - name: controlplane value: network: nameservers: - 10.10.10.10 - 10.10.10.11 - name: worker value: network: nameservers: - 10.10.10.10 - 10.10.10.11

AWS

This section lists variables for deploying workload clusters to AWS. For information about the variables in the Configuration File Variables column below, see AWS in Configuration File Variable Reference.

In the Class-based Cluster Object Structure column, all name / value pair settings are located in the Cluster object definition under spec.topology.variables.

Configuration File Variables Class-based Cluster Object Structure 
 AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_PROFILE
N/A
These settings are not stored in the Cluster object.		 
 AWS_REGION: us-east-1 
 - name: region value: us-east-1 
 AWS_SSH_KEY_NAME: aws-tkg-clusteradmin 
 - name: sshKeyName value: aws-tkg-clusteradmin 
 AWS_LOAD_BALANCER_SCHEME_INTERNAL: false 
 - name: loadBalancerSchemeInternal value: false
 
 BASTION_HOST_ENABLED: true 
 - name: bastion value: enabled: true 
 AWS_NODE_AZ: us-east-1a AWS_PRIVATE_NODE_CIDR: 10.0.0.0/24 AWS_PRIVATE_SUBNET_ID: subnet-0a7d376dde53c77ed AWS_PUBLIC_NODE_CIDR: 10.0.1.0/24 AWS_PUBLIC_SUBNET_ID: subnet-0794d50f57e9801b6 AWS_NODE_AZ_1: us-west-2b AWS_PRIVATE_NODE_CIDR_1: 10.0.2.0/24 AWS_PRIVATE_SUBNET_ID_1: subnet-0c338780824d1c59d AWS_PUBLIC_NODE_CIDR_1: 10.0.3.0/24 AWS_PUBLIC_SUBNET_ID_1: subnet-0addabd635d02ba97 AWS_NODE_AZ_2: ap-southeast-3 AWS_PRIVATE_NODE_CIDR_2: 10.0.4.0/24 AWS_PRIVATE_SUBNET_ID_2: subnet-00b9638e419a6187b AWS_PUBLIC_NODE_CIDR_2: 10.0.5.0/24 AWS_PUBLIC_SUBNET_ID_2: subnet-0ed174ef16a2f43aa 
 - name: network value: subnets: - az: us-east-1a private: cidr: 10.0.0.0/24 id: subnet-0a7d376dde53c77ed public: cidr: 10.0.1.0/24 id: subnet-0794d50f57e9801b6 - az: us-west-2b private: cidr: 10.0.2.0/24 id: subnet-0c338780824d1c59d public: cidr: 10.0.3.0/24 id: subnet-0addabd635d02ba97 - az: ap-southeast-3 private: cidr: 10.0.4.0/24 id: subnet-00b9638e419a6187b public: cidr: 10.0.5.0/24 id: subnet-0ed174ef16a2f43aa 
 AWS_VPC_CIDR: 10.0.0.0/16 
 - name: network value: … vpc: cidr: 10.0.0.0/16 
 AWS_VPC_ID: vpc-0ce8bdfea218 
 - name: network value: … vpc: existingID: vpc-0ce8bdfea218 
 AWS_SECURITY_GROUP_BASTION: sg-1 AWS_SECURITY_GROUP_APISERVER_LB: sg-2 AWS_SECURITY_GROUP_LB: sg-3 AWS_SECURITY_GROUP_CONTROLPLANE: sg-4 AWS_SECURITY_GROUP_NODE: sg-5 
 - name: network value: … securityGroupOverrides: bastion: sg-1 apiServerLB: sg-2 lb: sg-3 controlPlane: sg-4 node: sg-5 
 AWS_IDENTITY_REF_NAME: my-aws-id AWS_IDENTITY_REF_KIND: AWSClusterRoleIdentity 
 - name: identityRef value: name: my-aws-id kind: AWSClusterRoleIdentity 
 NODE_MACHINE_TYPE: m5.large AWS_NODE_OS_DISK_SIZE_GIB: 80
In a multi-AZ deployment, you can also set NODE_MACHINE_TYPE_1 and NODE_MACHINE_TYPE_2		 In the Cluster object, under spec.topology.workers:
 machineDeployments: - class: tkg-worker name: md-0 value: instanceType: m5.large rootVolume: sizeGiB: 80 
 CONTROL_PLANE_MACHINE_TYPE: t3.large AWS_CONTROL_PLANE_OS_DISK_SIZE_GIB: 80 
 - name: controlPlane value: instanceType: t3.large rootVolume: sizeGiB: 80

Microsoft Azure

This section lists variables for deploying workload clusters to Microsoft Azure. For information about the variables in the Configuration File Variables column below, see Microsoft Azure in Configuration File Variable Reference.

In the Class-based Cluster Object Structure column, all name / value pair settings are located in the Cluster object definition under spec.topology.variables.

Configuration File Variables Class-based Cluster Object Structure 
 AZURE_CLIENT_ID AZURE_CLIENT_SECRET AZURE_TENANT_ID
 N/A
These settings are not exposed in the Cluster object.		 
AZURE_LOCATION: eastus2
 
 - name: location value: eastus2
 
AZURE_RESOURCE_GROUP: my-azure-rg
 
 - name: resourceGroup value: my-azure-rg
 
AZURE_SUBSCRIPTION_ID: c789uce3-aaaa-bbbb-cccc-a51b6b0gb405
 
 - name: subscriptionID value: c789uce3-aaaa-bbbb-cccc-a51b6b0gb405
 
AZURE_ENVIRONMENT: AzurePublicCloud
 
 - name: environment value: AzurePublicCloud
 
AZURE_SSH_PUBLIC_KEY_B64: “c3NoLXJzYSBB […] vdGFsLmlv”
 
 - name: sshPublicKey value: c3NoLXJzYSBB […] vdGFsLmlv 
AZURE_FRONTEND_PRIVATE_IP: 10.0.0.100
Set if AZURE_ENABLE_PRIVATE_CLUSTER is true.		 
 - name: frontendPrivateIP value: 10.0.0.100 
AZURE_CUSTOM_TAGS: “foo=bar, plan=prod”
 
 - name: customTags value: “foo=bar, plan=prod” 
AZURE_ENABLE_ACCELERATED_NETWORKING: true
 
 - name: acceleratedNetworking value: enabled: true 
AZURE_ENABLE_PRIVATE_CLUSTER: false
 
 - name: privateCluster value: enabled: false 
 AZURE_VNET_CIDR: 10.0.0.0/16 AZURE_VNET_NAME: my-azure-vnet AZURE_VNET_RESOURCE_GROUP: my-azure-vnet-rg
 
 - name: network value: vnet: cidrBlocks: - 10.0.0.0/16 name: my-azure-vnet resourceGroup: my-azure-vnet-rg
 
 AZURE_IDENTITY_NAME: my-azure-id AZURE_IDENTITY_NAMESPACE: default 
 - name: identityRef value: name: my-azure-id namespace: default  
 AZURE_CONTROL_PLANE_DATA_DISK_SIZE_GIB: 256 AZURE_CONTROL_PLANE_OS_DISK_SIZE_GIB: 128 AZURE_CONTROL_PLANE_OS_DISK_STORAGE_ACCOUNT_TYPE: Premium_LRS AZURE_ENABLE_CONTROL_PLANE_OUTBOUND_LB: true AZURE_CONTROL_PLANE_OUTBOUND_LB_FRONTEND_IP_COUNT: 1 AZURE_CONTROL_PLANE_SUBNET_CIDR: 10.0.0.0/24 AZURE_CONTROL_PLANE_SUBNET_NAME: my-azure-cp-subnet AZURE_CONTROL_PLANE_MACHINE_TYPE: Standard_D2s_v3 
 - name: controlPlane value: dataDisks: - sizeGiB: 256 osDisk: sizeGiB: 128 storageAccountType: Premium_LRS outboundLB: enabled: true frontendIPCount: 1 subnet: cidr: 10.0.0.0/24 name: my-azure-cp-subnet vmSize: Standard_D2s_v3 
 AZURE_ENABLE_NODE_DATA_DISK: true AZURE_NODE_DATA_DISK_SIZE_GIB: 256 AZURE_NODE_OS_DISK_SIZE_GIB: 128 AZURE_NODE_OS_DISK_STORAGE_ACCOUNT_TYPE: Premium_LRS AZURE_ENABLE_NODE_OUTBOUND_LB: true AZURE_NODE_OUTBOUND_LB_FRONTEND_IP_COUNT: 1 AZURE_NODE_OUTBOUND_LB_IDLE_TIMEOUT_IN_MINUTES: 4 AZURE_NODE_SUBNET_CIDR: 10.0.1.0/24 AZURE_NODE_SUBNET_NAME: my-azure-worker-subnet AZURE_NODE_MACHINE_TYPE: Standard_D2s_v3 
 - name: worker value: dataDisks: - sizeGiB: 256 osDisk: sizeGiB: 128 storageAccountType: Premium_LRS outboundLB: enabled: true frontendIPCount: 1 idleTimeoutInMinutes: 4 subnet: cidr: 10.0.1.0/24 name: my-azure-worker-subnet vmSize: Standard_D2s_v3

NSX Advanced Load Balancer

This section lists variables for configuring NSX Advanced Load Balancer (ALB) in TKG.

Configuration File Variables Class-based Cluster Object Structure 
 AVI_CONTROL_PLANE_HA_PROVIDER
 
 topology: variables: - name: aviAPIServerHAProvider value: true

Example Cluster Object and Its Subordinate Objects

When you pass a cluster configuration file to the --file flag of tanzu cluster create, the command converts your cluster configuration file into a cluster spec file. See below for an example Cluster object and its subordinate objects generated by tanzu cluster create from the following configuration file:

CLUSTER_NAME: example-cluster
CLUSTER_PLAN: dev
NAMESPACE: default
CNI: antrea
VSPHERE_NETWORK: /dc0/network/VM Network
VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa AAAACCCza2EBBB[...]ADGQAg/POl6vyWOmQ==
VSPHERE_USERNAME: Administrator@vsphere.local
VSPHERE_PASSWORD: 1234567AbC!
VSPHERE_SERVER: 10.XXX.XXX.71
VSPHERE_DATACENTER: /dc0
VSPHERE_RESOURCE_POOL: /dc0/host/cluster0/Resources/example-tkg
VSPHERE_DATASTORE: /dc0/datastore/vsanDatastore
VSPHERE_FOLDER: /dc0/vm/example-tkg
VSPHERE_TLS_THUMBPRINT: ""
VSPHERE_INSECURE: true
VSPHERE_CONTROL_PLANE_ENDPOINT: 10.XXX.XXX.75
AVI_CONTROL_PLANE_HA_PROVIDER: false
ENABLE_AUDIT_LOGGING: false
ENABLE_DEFAULT_STORAGE_CLASS: true
CLUSTER_CIDR: 100.96.0.0/11
SERVICE_CIDR: 100.64.0.0/13
ENABLE_AUTOSCALER: false

Resulting Cluster object and its subordinate objects:

apiVersion: cpi.tanzu.vmware.com/v1alpha1
kind: VSphereCPIConfig
metadata:
  name: example-cluster
  namespace: default
spec:
  vsphereCPI:
    mode: vsphereCPI
    tlsCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    vmNetwork:
      excludeExternalSubnetCidr: 10.XXX.XXX.75/32
      excludeInternalSubnetCidr: 10.XXX.XXX.75/32
---
apiVersion: csi.tanzu.vmware.com/v1alpha1
kind: VSphereCSIConfig
metadata:
  name: example-cluster
  namespace: default
spec:
  vsphereCSI:
    config:
      datacenter: /dc0
      httpProxy: ""
      httpsProxy: ""
      insecureFlag: true
      noProxy: ""
      region: null
      tlsThumbprint: ""
      useTopologyCategories: false
      zone: null
    mode: vsphereCSI
---
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: ClusterBootstrap
metadata:
  annotations:
    tkg.tanzu.vmware.com/add-missing-fields-from-tkr: TKR-NAME
  name: example-cluster
  namespace: default
spec:
  additionalPackages:
  - refName: metrics-server*
  - refName: secretgen-controller*
  - refName: pinniped*
  cpi:
    refName: vsphere-cpi*
    valuesFrom:
      providerRef:
        apiGroup: cpi.tanzu.vmware.com
        kind: VSphereCPIConfig
        name: example-cluster
  csi:
    refName: vsphere-csi*
    valuesFrom:
      providerRef:
        apiGroup: csi.tanzu.vmware.com
        kind: VSphereCSIConfig
        name: example-cluster
  kapp:
    refName: kapp-controller*
---
apiVersion: v1
kind: Secret
metadata:
  name: example-cluster
  namespace: default
stringData:
  password: 1234567AbC!
  username: Administrator@vsphere.local
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  annotations:
    osInfo: ubuntu,20.04,amd64
    tkg.tanzu.vmware.com/cluster-controlplane-endpoint: 10.XXX.XXX.75
    tkg/plan: dev
  labels:
    tkg.tanzu.vmware.com/cluster-name: example-cluster
  name: example-cluster
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 100.96.0.0/11
    services:
      cidrBlocks:
      - 100.64.0.0/13
  topology:
    class: tkg-vsphere-default-CLUSTER-CLASS-VERSION
    controlPlane:
      metadata:
        annotations:
          run.tanzu.vmware.com/resolve-os-image: image-type=ova,os-name=ubuntu
      replicas: 1
    variables:
    - name: cni
      value: antrea
    - name: controlPlaneCertificateRotation
      value:
        activate: true
        daysBefore: 90
    - name: auditLogging
      value:
        enabled: false
    - name: podSecurityStandard
      value:
        audit: restricted
        deactivated: false
        warn: restricted
    - name: apiServerEndpoint
      value: 10.XXX.XXX.75
    - name: aviAPIServerHAProvider
      value: false
    - name: vcenter
      value:
        cloneMode: fullClone
        datacenter: /dc0
        datastore: /dc0/datastore/vsanDatastore
        folder: /dc0/vm/example-tkg
        network: /dc0/network/VM Network
        resourcePool: /dc0/host/cluster0/Resources/example-tkg
        server: 10.XXX.XXX.71
        storagePolicyID: ""
        tlsThumbprint: ""
    - name: user
      value:
        sshAuthorizedKeys:
        - ssh-rsa AAAACCCza2EBBB[...]ADGQAg/POl6vyWOmQ==
    - name: controlPlane
      value:
        machine:
          diskGiB: 40
          memoryMiB: 8192
          numCPUs: 2
    - name: worker
      value:
        machine:
          diskGiB: 40
          memoryMiB: 4096
          numCPUs: 2
    - name: security
      value:
        fileIntegrityMonitoring:
          enabled: false
        imagePolicy:
          pullAlways: false
          webhook:
            enabled: false
            spec:
              allowTTL: 50
              defaultAllow: true
              denyTTL: 60
              retryBackoff: 500
        kubeletOptions:
          eventQPS: 50
          streamConnectionIdleTimeout: 4h0m0s
        systemCryptoPolicy: default
    version: KUBERNETES-VERSION
    workers:
      machineDeployments:
      - class: tkg-worker
        metadata:
          annotations:
            run.tanzu.vmware.com/resolve-os-image: image-type=ova,os-name=ubuntu
        name: md-0
        replicas: 1
        strategy:
          type: RollingUpdate

The example object spec file above includes the following placeholder text:

  • TKR-NAME in tkg.tanzu.vmware.com/add-missing-fields-from-tkr: TKR-NAME: Set by tanzu cluster create to a compatible Tanzu Kubernetes release (TKr), depending on your configuration. For example, tkg.tanzu.vmware.com/add-missing-fields-from-tkr: v1.27.5---vmware.1-tkg.1. v1.27.5---vmware.1-tkg.1 is the default TKr for this Tanzu Kubernetes Grid release.
  • KUBERNETES-VERSION in version: KUBERNETES-VERSION: Set by tanzu cluster create to a compatible Kubernetes version, depending on your configuration. For example, version: v1.27.5+vmware.1-tkg.1. v1.27.5+vmware.1-tkg.1 is the default Kubernetes version for this Tanzu Kubernetes Grid release.
  • CLUSTER-CLASS-VERSION in class: tkg-vsphere-default-CLUSTER-CLASS-VERSION: Is the version of the default class. For example, class: tkg-vsphere-default-v1.1.1.
check-circle-line exclamation-circle-line close-line
Scroll to top icon