Home > Articles > VMM Integration

VMM Integration

Chapter Description

In this sample chapter from ACI Advanced Monitoring and Troubleshooting, you will learn about Virtual Machine Manager (VMM) and its integration into Cisco Application Centric Infrastructure (ACI).

From the Book

ACI Advanced Monitoring and Troubleshooting

ACI Advanced Monitoring and Troubleshooting

$47.99 (Save 20%)

Kubernetes Integration

Kubernetes is a portable, extensible open-source platform that automates the deployment, scaling, and management of container-based workloads and services in a network. Beginning with Cisco APIC Release 3.0(1), you can integrate Kubernetes on bare-metal servers into Cisco ACI.

To integrate Kubernetes with Cisco ACI, you need to execute a series of tasks. Some of them you perform in the network to set up the Cisco APIC; others you perform on the Kubernetes server. Once you have integrated Kubernetes, you can use the Cisco APIC to view Kubernetes in the Cisco ACI.

The following are the basic tasks involved in integrating Kubernetes into the Cisco ACI fabric:

  • Step 1. Prepare for the integration and set up the subnets and VLANs in the network.

  • Step 2. Fulfill the prerequisites.

  • Step 3. To provision the Cisco APIC to integrate with Kubernetes, download the provisioning tool, which includes a sample configuration file, and update the configuration file with information you previously gathered about your network. Then run the provisioning tool with the information about your network.

  • Step 4. Set up networking for the node to support Kubernetes installation. This includes configuring an uplink interface, subinterfaces, and static routes.

  • Step 5. Install Kubernetes and Cisco ACI containers.

  • Step 6. Use the Cisco APIC GUI to verify that Kubernetes has been integrated into Cisco ACI.

The following sections provide details on these steps.

Planning for Kubernetes Integration

Various network resources are required to provide capabilities to a Kubernetes cluster, including several subnets and routers. You need the following subnets:

  • Node subnet: This subnet is used for Kubernetes control traffic. It is where the Kubernetes API services are hosted. Make the node subnet a private subnet and make sure that it has access to the Cisco APIC management address.

  • Pod subnet: This is the subnet from which the IP addresses of Kubernetes pods are allocated. Make the pod subnet a private subnet.

  • Node service subnet: This subnet is used for internal routing of load-balanced service traffic. Make the node service subnet a private subnet.

  • External service subnets: These subnets are pools from which load-balanced services are allocated as externally accessible service IP addresses.

You need the following VLANs for local fabric use:

  • Node VLAN: This VLAN is used by the physical domain for Kubernetes nodes.

  • Service VLAN: This VLAN is used for delivery of load-balanced service traffic.

  • Infrastructure VLAN: This is the infrastructure VLAN used by the Cisco ACI fabric.

Prerequisites for Integrating Kubernetes with Cisco ACI

Ensure that the following prerequisites are in place before you try to integrate Kubernetes with the Cisco ACI fabric:

  • ▪ A working Cisco ACI installation

  • ▪ An attachable entity profile (AEP) set up with interfaces that are desired for the Kubernetes deployment

  • ▪ An L3Out connection, along with a Layer 3 external network to provide external access

  • ▪ Virtual routing and forwarding (VRF)

  • ▪ Any required route reflector configuration for the Cisco ACI fabric

  • ▪ A next-hop router that is connected to the Layer 3 external network and that is capable of appropriate external access and configured with the required routes

In addition, the Kubernetes cluster must be up through the fabric-connected interface on all the hosts. The default route should be pointing to the ACI node subnet bridge domain. This is not mandatory, but it simplifies the routing configuration on the hosts and is the recommend configuration. If you choose not to use this design, all Kubernetes-related traffic must go through the fabric.

Provisioning Cisco ACI to Work with Kubernetes

You can use the acc_provision tool to provision the fabric for the Kubernetes VMM domain and generate a .yaml file that Kubernetes uses to deploy the required Cisco ACI container components. The procedure to accomplish this is as follows:

  • Step 1. Download the provisioning tool from

    https://software.cisco.com/download/type.html?mdfid=285968390&i=rm and then follow these steps:

    • a. Click APIC OpenStack and Container Plugins.

    • b. Choose the package that you want to download.

    • c. Click Download.

  • Step 2. Generate a sample configuration file that you can edit by entering the following command:

    terminal$ acc-provision--sample

    This command generates the aci-containers-config.yaml configuration file, which looks as follows:

    # Configuration for ACI Fabric
      system_id: mykube             # Every opflex cluster must have a
                                    distinct ID
      apic_hosts:                   # List of APIC hosts to connect for
                                    APIC API
      vmm_domain:                   # Kubernetes VMM domain configuration
        encap_type: vxlan           # Encap mode: vxlan or vlan
        mcast_range:                # Every opflex VMM must use a distinct
      # The following resources must already exist on the APIC,
      # they are used, but not created by the provisioning tool.
      aep: kube-cluster             # The AEP for ports/VPCs used by this
      vrf:                          # This VRF used to create all
                                    Kubernetes EPs
        name: mykube-vrf
        tenant: common              # This can be system-id or common
        name: mykube_l3out          # Used to provision external IPs
        - mykube_extepg             # Used for external contracts
    # Networks used by Kubernetes
      node_subnet:      # Subnet to use for nodes
      pod_subnet:       # Subnet to use for Kubernetes Pods
      extern_dynamic:   # Subnet to use for dynamic external IPs
      extern_static:    # Subnet to use for static external IPs
      node_svc_subnet:  # Subnet to use for service graph ←This
                                    is not the same as the
                     Kubernetes service-cluster-ip-range: Use different
      kubeapi_vlan: 4001            # The VLAN used by the physdom for
      service_vlan: 4003            # The VLAN used by LoadBalancer
      infra_vlan: 4093              # The VLAN used by ACI infra
    # Configuration for container registry
    # Update if a custom container registry has been setup
      image_prefix: noiro                   # e.g: registry.example.com/
      # image_pull_secret: secret_name      # (if needed)
  • Step 3. Edit the sample configuration file, providing information from your network, and save the file.

  • Step 4. Provision the Cisco ACI fabric by using the following command:

    acc-provision -c aci-containers-config.yaml -o
    aci-containers.yaml -f kubernetes-<version> -a -u
    [apic username] -p [apic password]

    This command generates the file aci-containers.yaml, which you use after installing Kubernetes. It also creates the files user-[system id].key and user-[system id].crt, which contain the certificate used to access the Cisco APIC. Save these files in case you change the configuration later and want to avoid disrupting a running cluster because of a key change.

  • Step 5. (Optional) Configure advanced optional parameters to adjust to custom parameters other than the ACI default values or base provisioning assumptions. For example, if your VMM’s multicast address for the fabric is different from, you can configure it by using the following:


    If you are using VLAN encapsulation, you can specify the VLAN pool for it, as follows:

        encap_type: vlan
          start: 10
          end: 25

    If you want to use an existing user, key, certificate, add the following:

        username: <name>
        certfile: <pem-file>
        keyfile: <pem-file>

    If you are provisioning in a system nested inside virtual machines, enter the name of an existing preconfigured VMM domain in Cisco ACI into the aci_config section under the vmm_domain of the configuration file:

        type: vmware
        name: myvmware

    Preparing the Kubernetes Nodes

    When you are done provisioning Cisco ACI to work with Kubernetes, you can start preparing the networking construct for the Kubernetes nodes by following this procedure:

  • Step 1. Configure your uplink interface with or without NIC bonding, depending on how your AAEP is configured. Set the MTU on this interface to 1600.

  • Step 2. Create a subinterface on your uplink interface on your infrastructure VLAN. Configure this subinterface to obtain an IP address by using DHCP. Set the MTU on this interface to 1600.

  • Step 3. Configure a static route for the multicast subnet through the uplink interface used for VXLAN traffic.

  • Step 4. Create a subinterface (for example, kubeapi_vlan) on the uplink interface on your node VLAN in the configuration file. Configure an IP address on this interface in your node subnet. Then set this interface and the corresponding node subnet router as the default route for the node.

  • Step 5. Create the /etc/dhcp/dhclient-eth0.4093.conf file with the following content, inserting the MAC address of the Ethernet interface for each server on the first line of the file:

    send dhcp-client-identifier 01:<mac-address of infra VLAN
    request subnet-mask, domain-name, domain-name-servers,
    send host-name <server-host-name>;
    option rfc3442-classless-static-routes code 121 = array of
    unsigned integer 8;
    option ms-classless-static-routes code 249 = array of
    unsigned integer 8;
    option wpad code 252 = string;
    also request rfc3442-classless-static-routes;
    also request ms-classless-static-routes;
    also request static-routes;
    also request wpad;
    also request ntp-servers;

    The network interface on the infrastructure VLAN requests a DHCP address from the APIC infrastructure network for OpFlex communication. Make sure the server has a dhclient configuration for this interface to receive all the correct DHCP options with the lease.

  • Step 6. If you have a separate management interface for the node being configured, configure any static routes that you need to access your management network on the management interface.

  • Step 7. Ensure that OVS is not running on the node.

Here is an example of the interface configuration (in /etc/network/interfaces):

# Management network interface (not connected to ACI)
auto ens160
iface ens160 inet static
    up route add -net gw

# Interface connected to ACI
auto ens192
iface ens192 inet manual
    mtu 1600

# ACI Infra VLAN
auto ens192.3095
iface ens192.3095 inet dhcp
    mtu 1600
    up route add -net dev ens192.3095
    vlan-raw-device ens192

# Node Vlan
auto ens192.4001
iface ens192.4001 inet static
    mtu 1600
    vlan-raw-device ens192

Installing Kubernetes and Cisco ACI Containers

After you provision Cisco ACI to work with Kubernetes and prepare the Kubernetes nodes, you can install Kubernetes and ACI containers. You can use any installation method you choose, as long as it is appropriate to your environment. This procedure provides guidance and high-level instruction for installation; for details, consult Kubernetes documentation.

When installing Kubernetes, ensure that the API server is bound to the IP addresses on the node subnet and not to management or other IP addresses. Issues with node routing table configuration and API server advertisement addresses are the most common problems during installation. If you have problems, therefore, check these issues first.

Install Kubernetes so that it is configured to use a Container Network Interface (CNI) plug-in, but do not install a specific CNI plug-in configuration through your installer. Instead, deploy the CNI plug-in. To install the CNI plug-in, use the following command:

kubectl apply -f aci-containers.yaml

Verifying the Kubernetes Integration

After you have performed the steps described in the preceding sections, you can verify the integration in the Cisco APIC GUI. The integration creates a tenant, three EPGs, and a VMM domain. The procedure to do this is as follows:

  • Step 1. Log in to the Cisco APIC.

  • Step 2. Go to Tenants > tenant_name, where tenant_name is the name you specified in the configuration file that you edited and used in installing Kubernetes and the ACI containers.

  • Step 3. In the tenant navigation pane, expand the following: tenant_name > Application Profiles > application_profile_name > Application EPGs. You should see three folders inside the Application EPGs folder:

    • kube-default: The default EPG for containers that are otherwise not mapped to any specific EPG.

    • kube-nodes: The EPG for the Kubernetes nodes.

    • kube-system: The EPG for the kube-system Kubernetes namespace. This typically contains the kube-dns pods, which provide DNS services for a Kubernetes cluster.

  • Step 4. In the tenant navigation pane, expand the Networking and Bridge Domains folders. You should see two bridge domains:

    • node-bd: The bridge domain used by the node EPG

    • pod-bd: The bridge domain used by all pods

  • Step 5. If you deploy Kubernetes with a load balancer, go to Tenants > common, expand L4-L7 Services, and perform the following steps:

    • ▪ Open the L4-L7 Service Graph Templates folder; you should see a template for Kubernetes.

    • ▪ Open the L4-L7 Devices folder; you should see a device for Kubernetes.

    • ▪ Open the Deployed Graph Instances folder; you should see an instance for Kubernetes.

  • Step 6. Go to VM Networking > Inventory, and in the Inventory navigation pane, expand the Kubernetes folder. You should see a VMM domain, with the name you provided in the configuration file, and in that domain you should see folders called Nodes and Namespaces.

6. OpenShift Integration | Next Section Previous Section

Cisco Press Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from Cisco Press and its family of brands. I can unsubscribe at any time.


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about Cisco Press products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information

To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.


Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites; develop new products and services; conduct educational research; and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.


If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@ciscopress.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information

Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.


This site is not directed to children under the age of 13.


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information

If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.


Users can always make an informed choice as to whether they should proceed with certain services offered by Cisco Press. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.ciscopress.com/u.aspx.

Sale of Personal Information

Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents

California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure

Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact

Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice

We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020