Home > Articles > Management


Chapter Description

With more people accessing your business through a digital experience, application performance is more critical than ever, and managing resources across your complex IT environment has big implications on user experience and costs. In this sample chapter from Cisco Cloud Infrastructure, you will learn about workload management solutions including Intersight Workload Optimization Manager (IWO) and Cisco Intersight Kubernetes Service.

From the Book

Cisco Cloud Infrastructure

Cisco Cloud Infrastructure

$47.99 (Save 20%)

Cisco Container Platform

Setting up, deploying, and managing multiple containers for multiple micro-sized services gets tedious—and difficult to manage across multiple public and private clouds. IT Ops has wound up doing much of this extra work, which makes it difficult for them to stay on top of the countless other tasks they’re already charged with performing. If containers are going to truly be useful at scale, we have to find a way to make them easier to manage.

The following are the requirements in managing container environments:

  • The ability to easily manage multiple clusters

  • Simple installation and maintenance

  • Networking and security consistency

  • Seamless application deployment, both on the premises and in public clouds

  • Persistent storage

That’s where Cisco Container Platform (CCP) comes in, which is a fully curated, lightweight container management platform for production-grade environments, powered by Kubernetes, and delivered with Cisco enterprise-class support. It reduces the complexity of configuring, deploying, securing, scaling, and managing containers via automation, coupled with Cisco’s best practices for security and networking. CCP is built with an open architecture using open source components, so you’re not locked in to any single vendor. It works across both on-premises and public cloud environments. And because it’s optimized with Cisco HyperFlex, this preconfigured, integrated solution sets up in minutes.

The following are the benefits of CCP:

  • Reduced risk: CCP is a full-stack solution built and tested on Cisco HyperFlex and ACI Networking, with Cisco providing automated updates and enterprise-class support for the entire stack. CCP is built to handle production workloads.

  • Greater efficiency: CCP provides your IT Ops team with a turnkey, preconfigured solution that automates repetitive tasks and removes pressure on them to update people, processes, and skill sets in-house. It provides developers with flexibility and speed to be innovative and respond to market requirements more quickly.

  • Remarkable flexibility: CCP gives you choices when it comes to deployment—from hyperconverged infrastructure to VMs and bare metal. Also, because it’s based on open source components, you’re free from vendor lock-in.

Figure 5-7 provides a holistic overview of CCP.

Figure 5-7

Figure 5-7 Holistic overview of CCP

Cisco Container Platform ushers all of the tangible benefits of container orchestration into the technology domain of the enterprise. Based on upstream Kubernetes, CCP presents a UI for self-service deployment and management of container clusters. These clusters consume private cloud resources based on established authentication profiles, which can be bound to existing RBAC models. The advantage to disparate organizational teams is the flexibility to consistently and efficiently deploy clusters into IaaS resources, a feat not easily accomplished and scaled when utilizing script-based frameworks. Teams can discriminately manage their cluster resources, including responding to conditions requiring a scale-out or scale-in event, without fear of disrupting another team’s assets. CCP boasts an innately open architecture composed of well-established open source components—a framework embraced by DevOps teams aiming their innovation toward cloud-neutral work streams.

CCP deploys easily into an existing infrastructure, whether it be of a virtual or bare-metal nature, to become the turnkey container management platform in the enterprise. CCP incorporates ubiquitous monitoring and policy-based security and provides essential services such as load balancing and logging. The platform can provide applications an extension into network management, application performance monitoring, analytics, and logging. CCP offers an API layer that is compatible with Google Cloud Platform and Google Kubernetes Engine, so transitioning applications potentially from the private cloud to the public cloud fits perfectly into orchestration schemes. The case could be made for containerized workloads residing in the private cloud on CCP to consume services brokered by Google Cloud Platform, and vice versa. For environments with a Cisco Application Centric Infrastructure (ACI), Contiv, a CCP component, will secure the containers in a logical policy-based context. Those environments with Cisco HyperFlex (HX) can leverage the inherent benefits provided by HX storage and provide persistent volumes to the containers in the form of FlexVolumes. CCP normalizes the operational experience of managing a Kubernetes environment by providing a curated production quality solution integrated with best-of-breed open source projects. Figure 5-8 illustrates the CCP feature set.

Figure 5-8

Figure 5-8 CCP feature set

The following are some CCP use cases:

  • Simple GUI-driven menu system to deploy clusters: You don’t have to know the technical details of Kubernetes to deploy a cluster. Just fill in the questions, and CCP will do the work.

  • The ability to deploy Kubernetes clusters in air-gapped sites: CCP tenant images contain all the necessary binaries and don’t need Internet access to function.

  • Choice of networking solutions: Use Cisco’s ACI plug-in, an industry standard Calico network, or if scaling is your priority, choose Contiv with VPP. All work seamlessly with CCP.

  • Automated monthly updates: Bug fixes, feature enhancements, and CVE remedies are pushed automatically every month—not only for Kubernetes, but also for the underlying operating system (OS).

  • Built-in visibility and monitoring: CCP lets you see what’s going on inside clusters to stay on top of usage patterns and address potential problems before they negatively impact the business.

  • Preconfigured persistent volume storage: Dynamic provisioning using HyperFlex storage as the default. No additional drivers need to be installed. Just set it and forget it.

  • Deploy EKS clusters using CCP control plane: CCP allows you to use a single pane of glass for deploying on-premises and Amazon clusters, plus it leverages Amazon Authentication for both.

  • Pre-integrated Istio: It’s ready to deploy and use without additional administration.

Cisco Container Platform Architecture Overview

At the bottom of the stack is Level 1, the Networking layer, which can consist of Nexus switches, Application Policy Infrastructure Controllers (APICs), and Fabric Interconnects (FIs).

Level 2 is the Compute layer, which consists of HyperFlex, UCS, or third-party servers that provide virtualized compute resources through VMware and distributed storage resources.

Level 3 is the Hypervisor layer, which is implemented using HyperFlex or VMware.

Level 4 consists of the CCP control plane and data plane (or tenant clusters). In Figure 5-9, the left side shows the CCP control plane, which runs on four control-plane VMs, and the right side shows the tenant clusters. These tenant clusters are preconfigured to support persistent volumes using the vSphere Cloud Provider and Container Storage Interface (CSI) plug-in. Figure 5-9 provides an overview of the CCP architecture.

Figure 5-9

Figure 5-9 Container Platform Architecture Overview

Components of Cisco Container Platform

Table 5-2 lists the components of CCP.

Table 5-2 Components of CCP



Operating System







HyperFlex, UCS

Container Network Interface (CNI)

ACI, Contiv, Calico



Container Storage

HyperFlex Container Storage Interface (CSI) plug-in

Load Balancing

NGINX, Envoy

Service Mesh

Istio, Envoy


Prometheus, Grafana


Elasticsearch, Fluentd, and Kibana (EFK) stack

Container Runtime

Docker CE

Sample Deployment Topology

This section describes a sample deployment topology of the CCP and illustrates the network topology requirements at a conceptual level.

In this case, it is expected that the vSphere-based cluster is set up, provisioned, and fully functional for virtualization and virtual machine (VM) functionality before any installation of CCP. You can refer to the standard VMware documentation for details on vSphere installation. Figure 5-10 provides an example of a vSphere cluster on which CCP is to be deployed.

Figure 5-10

Figure 5-10 vSphere cluster on which CCP is to be deployed

Once the vSphere cluster is ready to provision VMs, the admin then provisions one or more VMware port groups (for example, PG10, PG20, and PG30 in the figure) on which virtual machines will subsequently be provisioned as container cluster nodes. Basic L2 switching with VMware vswitch functionality can be used to implement these port groups. IP subnets should be set aside for use on these port groups, and the VLANs used to implement these port groups should be terminated on an external L3 gateway (such as the ASR1K shown in the figure). The control-plane cluster and tenant-plane Kubernetes clusters of CCP can then be provisioned on these port groups.

All provisioned Kubernetes clusters may choose to use a single shared port group, or separate port groups may be provisioned (one per Kubernetes cluster), depending on the isolation needs of the deployment. Layer 3 network isolation may be used between these different port groups as long as the following conditions are met:

  • There is L3 IP address connectivity among the port group that is used for the control-plane cluster and the tenant cluster port groups

  • The IP address of the vCenter server is accessible from the control-plane cluster

  • A DHCP server is provisioned for assigning IP addresses to the installer and upgrade VMs, and it must be accessible from the control-plane port group cluster of the cluster

The simplest functional topology would be to use a single shared port group for all clusters with a single IP subnet to be used to assign IP addresses for all container cluster VMs. This IP subnet can be used to assign one IP per cluster VM and up to four virtual IP addresses per Kubernetes cluster, but would not be used to assign individual Kubernetes pod IP addresses. Hence, a reasonable capacity planning estimate for the size of this IP subnet is as follows:

(The expected total number of container cluster VMs across all clusters) + 3 × (the total number of expected Kubernetes clusters)

Administering Clusters on vSphere

You can create, upgrade, modify, or delete vSphere on-premises Kubernetes clusters using the CCP web interface. CCP supports v2 and v3 clusters on vSphere. The v2 clusters use a single master node for their control plane, whereas the v3 clusters can use one or three master nodes for their control plane. The multimaster approach of v3 clusters is the preferred cluster type, as this approach ensures high availability for the control plane. The following steps show you how to administer clusters on vSphere:

Step 1. In the left pane, click Clusters and then click the vSphere tab.

Step 2. Click NEW CLUSTER.

Step 3. In the BASIC INFORMATION screen:

  1. From the INFRASTRUCTURE PROVIDER drop-down list, choose the provider related to your Kubernetes cluster.

    For more information, see Adding vSphere Provider Profile.

  2. In the KUBERNETES CLUSTER NAME field, enter a name for your Kubernetes tenant cluster.

  3. In the DESCRIPTION field, enter a description for your cluster.

  4. In the KUBERNETES VERSION drop-down list, choose the version of Kubernetes that you want to use for creating the cluster.

  5. If you are using ACI, specify the ACI profile.

    For more information, see Adding ACI Profile.

  6. Click NEXT.

Step 4. In the PROVIDER SETTINGS screen:

  1. From the DATA CENTER drop-down list, choose the data center that you want to use.

  2. From the CLUSTERS drop-down list, choose a cluster.

  1. From the DATASTORE drop-down list, choose a datastore.

  1. From the VM TEMPLATE drop-down list, choose a VM template.

  2. From the NETWORK drop-down list, choose a network.

For v2 clusters that use HyperFlex systems:

  • The selected network must have access to the HypexFlex Connect server to support HyperFlex Storage Provisioners.

  • For HyperFlex Local Network, select k8-priv-iscsivm-network to enable HyperFlex Storage Provisioners.

  1. From the RESOURCE POOL drop-down list, choose a resource pool.

  2. Click NEXT.

Step 5. In the NODE CONFIGURATION screen:

  1. From the GPU TYPE drop-down list, choose a GPU type.

  1. For v3 clusters, under MASTER, choose the number of master nodes as well as their VCPU and memory configurations.

  1. Under WORKER, choose the number of worker nodes as well as their VCPU and memory configurations.

  2. In the SSH USER field, enter the SSH username.

  3. In the SSH KEY field, enter the SSH public key that you want to use for creating the cluster.

  1. In the ROUTABLE CIDR field, enter the IP addresses for the pod subnet in the CIDR notation.

  2. From the SUBNET drop-down list, choose the subnet that you want to use for this cluster.

  3. In the POD CIDR field, enter the IP addresses for the pod subnet in the CIDR notation.

  4. In the DOCKER HTTP PROXY field, enter a proxy for the Docker.

  5. In the DOCKER HTTPS PROXY field, enter an HTTPS proxy for the Docker.

  6. In the DOCKER BRIDGE IP field, enter a valid CIDR to override the default Docker bridge.

  1. Under DOCKER NO PROXY, click ADD NO PROXY and then specify a comma-separated list of hosts that you want to exclude from proxying.

  2. In the VM USERNAME field, enter the VM username that you want to use as the login for the VM.

  3. Under NTP POOLS, click ADD POOL to add a pool.

  4. Under NTP SERVERS, click ADD SERVER to add an NTP server.

  5. Under ROOT CA REGISTRIES, click ADD REGISTRY to add a root CA certificate to allow tenant clusters to securely connect to additional services.

  6. Under INSECURE REGISTRIES, click ADD REGISTRY to add Docker registries created with unsigned certificates.

  7. For v2 clusters, under ISTIO, use the toggle button to enable or disable Istio.

  8. Click NEXT.

Step 6. For v2 clusters, to integrate Harbor with CCP:

  1. In the Harbor Registry screen, click the toggle button to enable Harbor.

  2. In the PASSWORD field, enter a password for the Harbor server administrator.

  3. In the REGISTRY field, enter the size of the registry in gigabits.

  4. Click NEXT.

Step 7. In the Summary screen, verify the configuration and then click FINISH.

Administering Amazon EKS Clusters Using CCP Control Plane

Before you begin, make sure you have done the following:

  • Added your Amazon provider profile.

  • Added the required AMI files to your account.

  • Created an AWS IAM role for the CCP usage to create AWS EKS clusters.

Here is the procedure for administering Amazon EKS clusters using the CCP control plane:

Step 1. In the left pane, click Clusters and then click the AWS tab.

Step 2. Click NEW CLUSTER.

Step 3. In the Basic Information screen, enter the following information:

  1. From the INFRASTUCTURE PROVIDER drop-down list, choose the provider related to the appropriate Amazon account.

  2. From the AWS REGION drop-down list, choose an appropriate AWS region.

  1. In the KUBERNETES CLUSTER NAME field, enter a name for your cluster.

  2. Click NEXT.

Step 4. In the Node Configuration screen, specify the following information:

  1. From the INSTANCE TYPE drop-down list, choose an instance type for your cluster.

  2. From the MACHINE IMAGE drop-down list, choose an appropriate CCP Amazon Machine Image (AMI) file.

  3. In the WORKER COUNT field, enter an appropriate number of worker nodes.

  4. In the SSH PUBLIC KEY drop-down field, choose an appropriate authentication key.

    This field is optional. It is needed if you want to ssh to the worker nodes for troubleshooting purposes. Ensure that you use the Ed25519 or ECDSA format for the public key.

  1. In the IAM ACCESS ROLE ARN field, enter the Amazon Resource Name (ARN) information.

  1. Click NEXT.

Step 5. In the VPC Configuration screen, specify the following information:

  1. In the SUBNET CIDR field, enter a value of the overall subnet CIDR for your cluster.

  2. In the PUBLIC SUBNET CIDR field, enter values for your cluster on separate lines.

  3. In the PRIVATE SUBNET CIDR field, enter values for your cluster on separate lines.

Step 6. In the Summary screen, review the cluster information and then click FINISH.

Cluster creation can take up to 20 minutes. You can monitor the cluster creation status on the Clusters screen.

Licensing and Updates

You need to configure Cisco Smart Software Licensing on the Cisco Smart Software Manager (Cisco SSM) to easily procure, deploy, and manage licenses for your CCP instance. The number of licenses required depends on the number of VMs necessary for your deployment scenario.

Cisco SSM enables you to manage your Cisco Smart Software Licenses from one centralized website. With Cisco SSM, you can organize and view your licenses in groups called “virtual accounts.” You can also use Cisco SSM to transfer the licenses between virtual accounts, as needed.

You can access Cisco SSM from the Cisco Software Central home page, under the Smart Licensing area. CCP is initially available for a 90-day evaluation period, after which you need to register the product.

Connected Model

In a connected deployment model, the license usage information is directly sent over the Internet or through an HTTP proxy server to Cisco SSM.

For a higher degree of security, you can opt to use a partially connected deployment model, where the license usage information is sent from CCP to a locally installed VM-based satellite server (Cisco SSM satellite). Cisco SSM satellite synchronizes with Cisco SSM on a daily basis.

Registering CCP Using a Registration Token

You need to register your CCP instance with Cisco SSM or Cisco SSM satellite before the 90-day evaluation period expires. The following is the procedure for registering CCP using a registration token, and Figure 5-11 shows the workflow for this procedure.

Figure 5-11

Figure 5-11 Registering CCP using a registration token

Step 1. Perform these steps on Cisco SSM or Cisco SSM satellite to generate a registration token:

  1. Go to Inventory > Choose Your Virtual Account > General and then click New Token.

  2. If you want to enable higher levels of encryption for the products registered using the registration token, check the Allow Export-Controlled functionality on the products registered with this token check box.

  1. Download or copy the token.

Step 2. Perform these steps in the CCP web interface to register the registration token and complete the license registration process:

  1. In the left pane, click Licensing.

  2. In the license notification, click Register.

    The Smart Software Licensing Product Registration dialog box appears.

  3. In the Product Instance Registration Token field, enter, copy and paste, or upload the registration token that you generated in Step 1.

  4. Click REGISTER to complete the registration process.

Upgrading Cisco Container Platform

Upgrading CCP and upgrading tenant clusters are independent operations. You must upgrade CCP to allow tenant clusters to upgrade. Specifically, tenant clusters cannot be upgraded to a higher version than the control plane. For example, if the control plane is at version 1.10, the tenant cluster cannot be upgraded to the 1.11 version.

Upgrading CCP is a three-step process:

You can update the size of a single IP address pool during an upgrade. However, we recommend that you plan ahead for the free IP address requirement by ensuring that the free IP addresses are available in the control-plane cluster prior to the upgrade.

If you are upgrading from a CCP version, you must do the following:

  • Ensure that at least five IP addresses are available (3.1.x or earlier).

  • Ensure that at least three IP addresses are available (3.2 or later).

  • Upgrade the CCP tenant base VM.

  • Deploy/upgrade the VM.

  • Upgrade the CCP control plane.

To get the latest step-by-step upgrade procedure, you can refer to the CCP upgrade guide.

5. Cisco Intersight Kubernetes Service | Next Section Previous Section

Cisco Press Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from Cisco Press and its family of brands. I can unsubscribe at any time.


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about Cisco Press products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information

To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.


Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites; develop new products and services; conduct educational research; and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.


If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@ciscopress.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information

Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.


This site is not directed to children under the age of 13.


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information

If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.


Users can always make an informed choice as to whether they should proceed with certain services offered by Cisco Press. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.ciscopress.com/u.aspx.

Sale of Personal Information

Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents

California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure

Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact

Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice

We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020