View and Update Auto-Managed Package Configuration

This topic explains how to view, customize, and troubleshoot the configuration of auto-managed packages installed from the tanzu-core repository. It also lists the Antrea, Pinniped, Calico, vSphere CPI, and vSphere CSI configuration settings that you can customize.

Some of the procedures in this topic use imgpkg. For information about how to download and install imgpkg, see Install the Carvel Tools.

View Package Configuration

Auto-managed packages, which are located in the tanzu-core repository, contain components that support basic cluster functionality, such as the Antrea or Calico container network interface and the Pinniped authentication component. In some cases, internal Tanzu Kubernetes Grid and Kubernetes resources refer to these components as addons.

To view the configuration of an auto-managed package and the add-on component that it contains, you can:

  • Retrieve the Kubernetes secret for the installed add-on component by running the kubectl get secret CLUSTER-NAME-PACKAGE-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster. Where:

    • CLUSTER-NAME is the name of your target cluster. If you want to retrieve the secret for the add-on component that is installed in a workload cluster, CLUSTER-NAME is the name of your workload cluster.
    • PACKAGE-NAME is the name of the package.
    • CLUSTER-NAMESPACE is the namespace of your target cluster.

  • Download the package configuration files from projects.registry.vmware.com/tkg/packages/core.

For example, to view the configuration of Antrea:

  • Retrieve the Antrea secret by running the following command against the management cluster:

    kubectl get secret CLUSTER-NAME-antrea-addon -n CLUSTER-NAMESPACE

  • Download the Antrea package configuration files:

    1. Locate the version tag for antrea in the Tanzu Kubernetes release (TKr) that you used to create the cluster. You can retrieve the TKr by running the kubectl get tkr command against the management cluster:

      1. Run kubectl get clusters CLUSTER-NAME -n CLUSTER-NAMESPACE --show-labels.

      2. In the output, locate the value of tanzuKubernetesRelease. For example, tanzuKubernetesRelease=v1.28.11---vmware.1-tkg.1.

      3. Run kubectl get tkr TKR-VERSION, where TKR-VERSION is the value that you retrieved above. For example:

        kubectl get tkr v1.28.11---vmware.1-tkg.1 -o yaml

      4. In the output, locate the version tag under packages/core/antrea.

    2. Download the package configuration files. For example:

      imgpkg pull -b projects.registry.vmware.com/tkg/packages/core/antrea:v1.13.3_vmware.1-tkg.1-advanced -o antrea

    3. Navigate to antrea and review the files.

To know more about the Antrea container networking features, see VMware Container Networking with Antrea 1.6.0 Release Notes.

To know more about integrating an Antrea Container Cluster with VMware NSX, see Integration of Kubernetes Clusters with Antrea CNI.

Customize Package Configuration

Because auto-managed packages are managed by Tanzu Kubernetes Grid, you typically do not need to customize the configuration of auto-managed packages in workload clusters.

But you can customize these settings if needed. How you customize auto-managed package settings depends on the type of cluster.

For plan-based clusters, you can modify and patch the values.yaml section of the component secret to individually customize an existing cluster as described in Customize values.yaml Settings.

In some cases with plan-based clusters, you can add an overlay to the package configuration secret in the management cluster, to customize clusters before you create them, as described in Customize with an Overlay.

For class-based clusters, you can customize the cluster before you create it by creating a cluster manifest as in step 1 of a two-step process described in Create a Class-Based Cluster, and then adding a custom object definition to the manifest before you create the cluster in step 2. For an example, see Customize a Class-Based Manifest.

Auto-Managed Package values.yaml Settings

You can customize the following configuration settings in auto-managed packages. These values are set in the values.yaml section of the package component secret:

Package Setting Description
Antrea antrea.config.defaultMTU Defaults to null.
Antrea antrea.config.tlsCipherSuites Include FIPS-enabled Cipher Suites, by default. To switch to other Cipher Suites, update the values under the tlsCipherSuites field.
Calico calico.config.vethMTU Defaults to “0”, which makes Calico auto-detect its maximum transmission size (MTU) setting. Set this parameter to specify a maximum packet size, in bytes, as a string.
Calico calico.config.skipCNIBinaries Defaults to true, which restricts Calico from overwriting the settings of existing CNI plugins during cluster creation. When you upgrade a cluster, to avoid overwriting, set calico.config.skipCNIBinaries=true.
Pinniped pinniped.supervisor_svc_external_dns An FQDN associated with a Pinniped supervisor, used as a callback URL in the OIDC IDP client. Depending on Pinniped’s service type, you may also need to include the port number:
  • For LoadBalancer, on vSphere with NSX ALB, do not include the port number: https://hr.tkg.example.com.
  • For NodePort, as on vSphere without NSX ALB, include the port number: https://hr.tkg.example.com:31234.
Pinniped pinniped.upstream_oidc_client_id The client ID of your OIDC provider.
Pinniped pinniped.upstream_oidc_client_secret The client secret of your OIDC provider.
Pinniped pinniped.upstream_oidc_issuer_url The URL of your OIDC provider.
Pinniped pinniped.upstream_oidc_tls_ca_data The base64-encoded CA bundle data used to verify TLS connections to your OIDC provider.
Pinniped pinniped.upstream_oidc_additional_scopes A list of additional scopes to request in the token response.
Pinniped pinniped.upstream_oidc_claims OIDC claim mapping.
vSphere CSI vsphereCSI.provisionTimeout Defaults to 300s.
vSphere CSI vsphereCSI.attachTimeout Defaults to 300s.
vSphere CSI vsphereCSI.netPermissions Defaults to null.

Customize values.yaml Settings

To customize the values.yaml section of an add-on component secret in an already-running plan-based cluster:

  1. Retrieve the secret by running the kubectl get secret CLUSTER-NAME-PACKAGE-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster. For example:

    kubectl get secret example-workload-cluster-antrea-addon -n example-workload-cluster-namespace -o jsonpath="{.data.values\.yaml}" | base64 -d > values.yaml

  2. Update the values.yaml section. You can update any of the values listed in the table above.

  3. Apply your update by base64 encoding the edited values.yaml file and replacing it in the cluster secret. This command differs depending on the OS of your environment. For example:

    Linux:

    kubectl patch secret/example-workload-cluster-antrea-addon -n example-workload-cluster-namespace -p "{\"data\":{\"values.yaml\":\"$(base64 -w 0 < values.yaml)\"}}" --type=merge

    macOS:

    kubectl patch secret/example-workload-cluster-antrea-addon -n example-workload-cluster-namespace -p "{\"data\":{\"values.yaml\":\"$(base64 < values.yaml)\"}}" --type=merge

  4. After updating the secret, check the status of the package by running the kubectl get packageinstall command. For example:

    $ kubectl get packageinstall antrea -n tkg-system
NAMESPACE    NAME                   PACKAGE NAME                      PACKAGE VERSION                 DESCRIPTION                  AGE
tkg-system   antrea                 antrea.tanzu.vmware.com           1.11.1+vmware.4-tkg.1           Reconcile succeeded          7d14h

    If the returned status is Reconcile failed, run the following command to get details on the failure:

    kubectl get packageinstall antrea -n tkg-system -o yaml

  5. Run the kubectl get app command. For example:

    $ kubectl get app antrea -n tkg-system
NAME           DESCRIPTION             SINCE-DEPLOY    AGE
antrea         Reconcile succeeded     3m23s           7h50m

    If the returned status is Reconcile failed, run the following command to get details on the failure:

    kubectl get app antrea -n tkg-system -o yaml

The example below updates the default Maximum Transmission Unit (MTU) for Antrea.

stringData:
  values.yaml: |
    #@data/values
    #@overlay/match-child-defaults missing_ok=True
    ---
    infraProvider: vsphere
    antrea:
      config:
        defaultMTU: 8900
Note

Ensure that you configure the same MTU settings for all the nodes in a cluster. The firewall settings must allow for packets of the configured MTU size. To resolve any issues caused by different MTU settings on the nodes in a cluster, see Cluster Worker Nodes in NotReady Status Due to Mismatched MTUs.

Customize with an Overlay

In some cases, you can add an overlay to the add-on component secret to customize the auto-managed package configuration of plan-based clusters before they are created created.

The example below instructs Pinniped to use the LoadBalancer service type instead NodePort, which is the default on vSphere when NSX Advanced Load Balancer (ALB) does not serve as the control plane endpoint:

...
stringData:
 overlays.yaml: |
   #@ load("@ytt:overlay", "overlay")
   #@overlay/match by=overlay.subset({"kind": "Service", "metadata": {"name": "pinniped-supervisor", "namespace": "pinniped-supervisor"}})
   ---
   #@overlay/replace
   spec:
     type: LoadBalancer
     selector:
       app: pinniped-supervisor
     ports:
       - name: https
         protocol: TCP
         port: 443
         targetPort: 8443
 values.yaml: |
   #@data/values
   #@overlay/match-child-defaults missing_ok=True
   ---
   infrastructure_provider: vsphere
   tkg_cluster_role: management

To add an overlay:

  1. Retrieve the secret by running the kubectl get secret CLUSTER-NAME-PACKAGE-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster. For example:

    kubectl get secret example-workload-cluster-pinniped-addon -n example-workload-cluster-namespace -o jsonpath="{.data.values\.yaml}" | base64 -d > values.yaml

  2. Add your overlays.yaml section under stringData.

  3. Apply your update by base64 encoding the edited values.yaml file and replacing it in the cluster secret. This command differs depending on the OS of your environment. For example:

    Linux:

    kubectl patch secret/example-workload-cluster-pinniped-addon -n example-workload-cluster-namespace -p "{\"data\":{\"values.yaml\":\"$(base64 -w 0 < values.yaml)\"}}" --type=merge

    macOS:

    kubectl patch secret/example-workload-cluster-pinniped-addon -n example-workload-cluster-namespace -p "{\"data\":{\"values.yaml\":\"$(base64 < values.yaml)\"}}" --type=merge

  4. After updating the secret, check the status of the package by running the kubectl get packageinstall and kubectl get app commands as described in Auto-Managed Package values.yaml Settings.

Customize a Class-Based Manifest

For class-based clusters, you can customize the cluster before you create it by creating a cluster manifest as in step 1 of a two-step process described in Create a Class-Based Cluster, and then adding a custom object definition to the manifest before you create the cluster in step 2.

For example, to customize the netPermissions value for vSphere CSI, modify the manifest generated with tanzu cluster create --dry-run by adding a netPermissions block like the following in the VSphereCSIConfig object’s spec.vsphereCSI definition block:

apiVersion: csi.tanzu.vmware.com/v1alpha1
kind: VSphereCSIConfig
[...]
spec:
  vsphereCSI:
    mode: vsphereCSI
    config:
      [...]
      provisionTimeout: 33s
      attachTimeout: 77s
      resizerTimeout: 99s
      netPermissions:
        PERM-1:
          ips: "*"
          permissions: READ_WRITE
          rootsquash: false
        PERM-2:
          ips: "10.20.20.0/24"
          permissions: READ_ONLY
          rootsquash: true
        PERM-3:
          ips: "10.30.30.0/24"
          permissions: NO_ACCESS

Where:

  • PERM-* is a name you give to that section of the permission settings, which translates to [NetPermissions "PERM-1"] etc. in the vSphere configuration secret.
  • ips: value is the address ranges for the file volumes that the section sets permissions for.
  • permissions: value is the permissions set for that section.

After you modify the manifest, you can create the cluster following step 2 in Create a Class-Based Cluster.

Troubleshoot Package Configuration

To troubleshoot the auto-managed package configurations in a cluster, you review and modify the package’s PackageInstall custom resource (CR) and component Secret.

To apply temporary changes to your package configuration, you may need to pause reconciliation of PackageInstall and Secret objects as described in Pause Lifecycle Management for an Auto-Managed Package below. This procedure deactivates lifecycle management for the package and should be used with caution.

Key Terms

Tanzu Kubernetes Grid uses the following resources to manage the lifecycle of auto-managed packages.

Components installed in the management cluster:

  • kapp-controller, a local package manager: When you deploy a management cluster, the Tanzu CLI installs kapp-controller in the cluster. kapp-controller installs tanzu-addons-manager and other auto-managed packages. It also installs and manages kapp-controller in each workload cluster that you deploy from that management cluster.
  • tanzu-addons-manager: Manages the add-on components that are installed, as auto-managed packages, in the management cluster and workload clusters that you deploy from your management cluster.
  • tkr-controller: Creates TKrs and BoM ConfigMaps in the management cluster.

Component installed in workload clusters:

kapp-controller installs auto-managed packages in the workload cluster in which it runs.

Objects:

  • Secret: The Tanzu CLI creates a Secret for each add-on component, per cluster. These secrets define the configuration of the add-on components. tanzu-addons-manager reads the secrets and uses the configuration information that they contain to configure the add-on components. All secrets are created in the management cluster.
  • PackageRepository CR: The tanzu-addons-manager creates a PackageRepository CR that references all add-on component Package CRs (see below).
  • Package CR: For each add-on component in the PackageRepository, kapp-controller creates a Package CR in the target cluster.
  • PackageInstall CR: For each add-on component Package, tanzu-addons-manager creates a PackageInstall CR in the target cluster to inform kapp-controller about which auto-managed packages it needs to install.
  • App CR: For each PackageInstall, kapp-controller creates an App CR in the target cluster. Then, kapp-controller reconciles the CR and installs the package.
  • BoM ConfigMap: Provides metadata information about the add-on components, such as image location, to tanzu-addons-manager.

You can use the following commands to monitor the status of these resources:

Command Description
kubectl get packageinstall PACKAGE-NAME -n tkg-system -o yaml Check the PackageInstall CR in your target cluster. For example, kubectl get packageinstall antrea -n tkg-system -o yaml.
kubectl get app PACKAGE-NAME -n tkg-system -o yaml Check the App CR in your target cluster. For example, kubectl get app antrea -n tkg-system -o yaml.
kubectl get cluster CLUSTER-NAME -n CLUSTER-NAMESPACE -o jsonpath={.metadata.labels.tanzuKubernetesRelease} In the management cluster, check if the TKr label of your target cluster points to the correct TKr.
kubectl get tanzukubernetesrelease TKR-NAME Check if the TKr is present in the management cluster.
kubectl get configmaps -n tkr-system -l ‘tanzuKubernetesRelease=TKR-NAME’ Check if the BoM ConfigMap corresponding to your TKr is present in the management cluster.
kubectl get app CLUSTER-NAME-kapp-controller -n CLUSTER-NAMESPACE For workload clusters, check if the kapp-controller App CR is present in the management cluster.
kubectl logs deployment/tanzu-addons-controller-manager -n tkg-system Check tanzu-addons-manager logs in the management cluster.
kubectl get configmap -n tkg-system | grep ADD-ON-NAME-ctrl Check if your updates to the add-on secret have been applied. The sync period is 5 minutes.

Pause Lifecycle Management for an Auto-Managed Package

Important

The commands in this section deactivate package lifecycle management. Whenever possible, use the procedures described in Updating Package Configuration above instead.

If you need to temporary pause lifecycle management for an auto-managed package, you can use the commands below:

  • To pause secret reconciliation, run the following command against the management cluster:

    kubectl patch secret/CLUSTER-NAME-ADD-ON-NAME-addon -n CLUSTER-NAMESPACE -p '{"metadata":{"annotations":{"tkg.tanzu.vmware.com/addon-paused": ""}}}' --type=merge

    After you run this command, tanzu-addons-manager stops reconciling the secret.

  • To pause PackageInstall CR reconciliation, run the following command against your target cluster:

    kubectl patch packageinstall/PACKAGE-NAME -n tkg-system -p '{"spec":{"paused":true}}' --type=merge

    After you run this command, kapp-controller stops reconciling the PackageInstall and corresponding App CR.

If you want to temporary modify the resources of an add-on component, pause secret reconciliation first and then pause PackageInstall CR reconciliation. After you unpause lifecycle management, tanzu-addons-manager and kapp-controller resume secret and PackageInstall CR reconciliation:

  • To unpause secret reconciliation, remove tkg.tanzu.vmware.com/addon-paused from the secret annotations.

  • To unpause PackageInstall CR reconciliation, update the PackageInstall CR with {"spec":{"paused":false}} or remove the variable.

check-circle-line exclamation-circle-line close-line
Scroll to top icon