ACI Advanced Monitoring and Troubleshooting

Features

• Comprehensive, insider guidance for optimizing ACI in any nextgen datacenter environment
• Easy, step-by-step guidance for building your own ACI fabric
• Walks through detailed real-world ACI use cases, including failure scenarios with proven remedial actions
• Reviews management, monitoring, automation, and orchestration for software defined datacenters
• By a team of Cisco experts who’ve been helping enterprise clients succeed with ACI since its launch

• Copyright 2021
• Dimensions: 7-3/8" x 9-1/8"
• Pages: 500
• Edition: 1st

• Book
• ISBN-10: 1-58714-528-6
• ISBN-13: 978-1-58714-528-5

Advanced real-world Cisco Application Centric Infrastructure (ACI) monitoring and troubleshooting

This expert guide and reference will help you confidently deploy, support, monitor, and troubleshoot ACI fabrics and components. It is also designed to help you prepare for your Cisco DCACIA (300-630) exam, earning Cisco Certified SpecialistACI Advanced Implementation certification and credit toward CCNP Data Center certification if you choose.

Authored by three leading Cisco ACI experts, it combines a solid conceptual foundation, in-depth technical knowledge, and practical techniques. It also contains proven features to help exam candidates prepare, including review questions in most chapters, and Key Topic icons highlighting concepts covered on the exam.

The authors thoroughly introduce ACI functions, components, policies, command-line interfaces, connectivity, fabric design, virtualization and service integration, automation, orchestration, and more. Next, they introduce best practices for monitoring and management, including the use of faults, health scores, tools, the REST API, in-band and out-of-band management techniques, and monitoring protocols. Proven configurations are provided, with steps for verification. Finally, they present advanced forwarding and troubleshooting techniques for maximizing ACI performance and value.

ACI Advanced Monitoring and Troubleshooting is an indispensable resource for every data center architect, engineer, developer, network or virtualization administrator, and operations team member working in ACI environments.

• Understand Cisco ACI core functions, components, and protocols
• Apply the ACI Policy-Based Object Model to develop overall application frameworks
• Use command-line interfaces to manage and monitor Cisco ACI systems
• Master proven options for ACI physical and logical fabric design
• Establish connectivity for compute, storage, and service devices, switches, and routers
• Gain visibility into virtualization layers through VMM, and integrate hypervisors from multiple vendors
• Seamlessly integrate Layer 4 to Layer 7 services such as load balancing and firewalling
• Automate and orchestrate for fast deployment with the REST API, scripting, and Ansible
• Minimize downtime and maximize ROI through more effective monitoring and configuration
• Thoroughly master concepts and techniques for advanced ACI and VXLAN forwarding
• Build deep practical expertise for quickly troubleshooting critical events
• Gain quick visibility into traffic flows and streamline problem isolation with the ACI Visibility & Troubleshooting Tool
• Walk through multiple real-world troubleshooting scenarios step-by-step
• Forewords written by Yusuf Bhaiji, Director of Certifications, Cisco Systems; and Ronak Desai, VP of Engineering for the Data Center Networking Business Unit, Cisco Systems.

This book is part of the Networking Technology Series from Cisco Press, which offers networking professionals valuable information for constructing efficient networks, understanding new technologies, and building successful careers.

Online Sample Chapter

VMM Integration

Sample Pages

Download the sample pages (includes Chapter 6)

Table of Contents

Foreword by Yusuf Bhaiji     xxviii
Foreword by Ronak Desai     xxix
Introduction     xxx
PART I:  INTRODUCTION TO ACI
Chapter 1  Fundamental Functions and Components of Cisco ACI     1
ACI Building Blocks     8
    Hardware Specifications     8
ACI Key Concepts     14
    Control Plane     15
    Data Plane     17
    VXLAN     17
    Tenant     18
    VRF     19
    Application Profile     20
    Endpoint Group     21
    Contracts     22
    Bridge Domain     24
    External Routed or Bridged Network     25
Summary     26
Review Key Topics     26
Review Questions     27
Chapter 2  Introduction to the ACI Policy Model     31
Key Characteristics of the Policy Model     32
    Management Information Tree (MIT)     33
    Benefits of a Policy Model     37
Logical Constructs     37
Tenant Objects     38
VRF Objects     39
Application Profile Objects     40
Endpoint Group Objects     41
Bridge Domain and Subnet Objects     43
    Bridge Domain Options     45
Contract Objects     46
    Labels, Filters, and Aliases     48
    Contract Inheritance     49
    Contract Preferred Groups     49
    vzAny     50
Outside Network Objects     51
Physical Construct     52
    Access Policies     52
    Switch Policies     53
    Interface Policies     54
    Global Policies     55
Managed Object Relationships and Policy Resolution     57
Tags     58
Default Policies     58
How a Policy Model Helps in Diagnosis     60
Summary     63
Review Key Topics     63
Review Questions     64
Chapter 3  ACI Command-Line Interfaces     67
APIC CLIs     68
    NX-OSStyle CLI     68
    Bash CLI     74
ACI Fabric Switch CLIs     78
    iBash CLI     78
    VSH CLI     81
    VSH_LC CLI     83
Summary     84
Reference     84
Chapter 4  ACI Fabric Design Options     85
Physical Design     85
    Single- Versus Multiple-Fabric Design     87
    Multi-Pod     97
    Multi-Site     116
    Remote Leaf     131
    Hardware and Software Support     134
    ACI Multi-Pod and Remote Leaf Integration     143
Logical Design     149
    Design 1: Container-as-a-Service Using the OpenShift Platform and Calico CNI     149
Design 2: Vendor-Based ERP/SAP Hana Design with ACI     165
Design 3: vBrick Digital Media Engine Design with ACI     175
Summary     180
Review Key Topics     181
Review Questions     181
Chapter 5  End Host and Network Connectivity     185
End Host Connectivity     185
    VLAN Pool     186
    Domain     186
    Attachable Access Entity Profiles (AAEPs)     186
    Switch Policies     187
    Interface Policies     188
    Virtual Port Channel (VPC)     191
    Port Channel     197
    Access Port     201
    Best Practices in Configuring Access Policies     206
    Compute and Storage Connectivity     207
    L4/L7 Service Device Connectivity     210
Network Connectivity     213
    Connecting an External Bridge Network     213
    Connecting an External Routed Network     218
Diagnosing Connectivity Problems     242
Summary     245
Review Questions     245
Chapter 6  VMM Integration     249
Virtual Machine Manager (VMM)     249
    VMM Domain Policy Model     250
    VMM Domain Components     250
    VMM Domains     250
    VMM Domain VLAN Pool Association     252
VMware Integration     257
    Prerequisites for VMM Integration with AVS or VDS     257
    Guidelines and Limitations for VMM Integration with AVS or VDS     257
    ACI VMM Integration Workflow     258
    Publishing EPGs to a VMM Domain     258
    Connecting Virtual Machines to the Endpoint Group Port Groups on vCenter     259
    Verifying VMM Integration with the AVS or VDS     259
Microsoft SCVMM Integration     260
    Mapping ACI and SCVMM Constructs     261
    Mapping Multiple SCVMMs to an APIC     262
    Verifying That the OpFlex Certificate Is Deployed for a Connection from the SCVMM to the APIC     262
    Verifying VMM Deployment from the APIC to the SCVMM     263
OpenStack Integration     263
    Extending OpFlex to the Compute Node     264
    ACI with OpenStack Physical Architecture     264
    OpFlex Software Architecture     265
    OpenStack Logical Topology     265
    Mapping OpenStack and ACI Constructs     266
Kubernetes Integration     272
    Planning for Kubernetes Integration     272
    Prerequisites for Integrating Kubernetes with Cisco ACI     273
    Provisioning Cisco ACI to Work with Kubernetes     274
    Preparing the Kubernetes Nodes     277
    Installing Kubernetes and Cisco ACI Containers     279
    Verifying the Kubernetes Integration     280
OpenShift Integration     281
    Planning for OpenShift Integration     282
    Prerequisites for Integrating OpenShift with Cisco ACI     283
    Provisioning Cisco ACI to Work with OpenShift     284
    Preparing the OpenShift Nodes     287
    Installing OpenShift and Cisco ACI Containers     290
    Updating the OpenShift Router to Use the ACI Fabric     291
    Verifying the OpenShift Integration     291
VMM Integration with ACI at Multiple Locations     292
    Multi-Site     292
    Remote Leaf     295
Summary     298
Chapter 7  L4/L7 Service Integration     299
Service Insertion     299
The Service Graph     300
    Managed Mode Versus Un-Managed Mode     301
    L4L7 Integration Use Cases     302
    How Contracts Work in ACI     303
    The Shadow EPG     306
    Configuring the Service Graph     307
    Service Graph Design and Deployment Options     312
Policy-Based Redirect (PBR)     322
    PBR Design Considerations     323
    PBR Design Scenarios     324
    Configuring the PBR Service Graph     325
    Service Node Health Check     326
    Common Issues in the PBR Service Graph     328
L4/L7 Service Integration in Multi-Pod and Multi-Site     332
    Multi-Pod     332
    Multi-Site     338
Review Questions     342
Chapter 8  Automation and Orchestration     343
The Difference Between Automation and Orchestration     343
    Benefits of Automation and Orchestration     344
REST API     349
Automating Tasks Using the Native REST API: JSON and XML     351
    API Inspector     351
    Object (Save As)     353
    Visore (Object Store Browser)     355
    MOQuery     357
    Automation Use Cases     364
Automating Tasks Using Ansible     372
    Ansible Support in ACI     375
    Installing Ansible and Ensuring a Secure Connection     378
    APIC Authentication in Ansible     382
    Automation Use Cases     384
Orchestration Through UCS Director     392
    Management Through Cisco UCS Director     392
    Automation and Orchestration with Cisco UCS Director     393
    Automation Use Cases     395
Summary     402
Review Questions     402
PART II:  MONITORING AND MANAGEMENT BEST PRACTICES
Chapter 9  Monitoring ACI Fabric     405
Importance of Monitoring     405
Faults and Health Scores     407
Faults     407
Health Scores     411
ACI Internal Monitoring Tools     415
    SNMP     415
    Syslog     420
    NetFlow     426
ACI External Monitoring Tools     430
    Network Insights     430
    Network Assurance Engine     437
    Tetration     453
Monitoring Through the REST API     473
    Monitoring an APIC     475
Monitoring Leafs and Spines     482
    Monitoring Applications     499
Summary     505
Review Questions     506
Chapter 10  Network Management and Monitoring Configuration     509
Out-of-Band Management     509
    Creating Static Management Addresses     510
    Creating the Management Contract     510
    Choosing the Node Management EPG     513
    Creating an External Management Entity EPG     513
    Verifying the OOB Management Configuration     515
In-Band Management     517
    Creating a Management Contract     517
    Creating Leaf Interface Access Policies for APIC INB Management     518
    Creating Access Policies for the Border Leaf(s) Connected to L3Out     520
    Creating INB Management External Routed Networks (L3Out)     522
    Creating External Management EPGs     524
    Creating an INB BD with a Subnet     527
    Configuring the Node Management EPG     529
    Creating Static Management Addresses     530
    Verifying the INB Management Configuration     530
AAA     533
    Configuring Cisco Secure ACS     533
    Configuring Cisco ISE     542
    Configuring AAA in ACI     547
    Recovering with the Local Fallback User     550
    Verifying the AAA Configuration     550
Syslog     551
    Verifying the Syslog Configuration and Functionality     555
SNMP     556
    Verifying the SNMP Configuration and Functionality     562
SPAN     566
    Access SPAN     567
    Fabric SPAN     571
    Tenant SPAN     572
    Ensuring Visibility and Troubleshooting SPAN     575
    Verifying the SPAN Configuration and Functionality     576
NetFlow     577
    NetFlow with Access Policies     580
    NetFlow with Tenant Policies     582
    Verifying the NetFlow Configuration and Functionality     585
Summary     587
PART III:  ADVANCED FORWARDING AND TROUBLESHOOTING TECHNIQUES
Chapter 11  ACI Topology     589
Physical Topology     589
APIC Initial Setup     593
Fabric Access Policies     595
    Switch Profiles, Switch Policies, and Interface Profiles     595
    Interface Policies and Policy Groups     596
    Pools, Domains, and AAEPs     597
VMM Domain Configuration     601
    VMM Topology     601
Hardware and Software Specifications     603
Logical Layout of EPGs, BDs, VRF Instances, and Contracts     605
    L3Out Logical Layout     606
Summary     608
Review Key Topics     608
References     609
Chapter 12  Bits and Bytes of ACI Forwarding     611
Limitations of Traditional Networks and the Evolution of Overlay Networks     611
High-Level VXLAN Overview     613
IS-IS, TEP Addressing, and the ACI Underlay     615
    IS-IS and TEP Addressing     615
    FTags and the MDT     618
Endpoint Learning in ACI     626
    Endpoint Learning in a Layer 2Only Bridge Domain     627
    Endpoint Learning in a Layer 3Enabled Bridge Domain     635
    Fabric Glean     640
    Remote Endpoint Learning     641
    Endpoint Mobility     645
    Anycast Gateway     647
    Virtual Port Channels in ACI     649
Routing in ACI     651
    Static or Dynamic Routes     651
    Learning External Routes in the ACI Fabric     656
    Transit Routing     659
Policy Enforcement     661
    Shared Services     664
    L3Out Flags     668
Quality of Service (QoS) in ACI     669
    Externally Set DSCP and CoS Markings     671
CoS Preservation in ACI     672
Multi-Pod     674
Multi-Site     680
Remote Leaf     684
Forwarding Scenarios     686
    ARP Flooding     686
    Layer 2 Known Unicast     688
    ARP Optimization     690
    Layer 2 Unknown Unicast Proxy     690
    L3 Policy Enforcement When Going to L3Out     693
    L3 Policy Enforcement for External Traffic Coming into the Fabric     695
Route Leaking/Shared Services     695
    Consumer to Provider     695
    Provider to Consumer     698
Multi-Pod Forwarding Examples     698
    ARP Flooding     700
    Layer 3 Proxy Flow     700
Multi-Site Forwarding Examples     703
    ARP Flooding     703
    Layer 3 Proxy Flow     705
Remote Leaf     707
    ARP Flooding     707
    Layer 3 Proxy Flow     710
Summary     713
Review Key Topics     713
References     714
Review Questions     714
Chapter 13  Troubleshooting Techniques     717
General Troubleshooting     717
    Faults, Events, and Audits     718
    moquery     722
    iCurl     724
    Visore     726
Infrastructure Troubleshooting     727
    APIC Cluster Troubleshooting     727
    Fabric Node Troubleshooting     734
How to Verify Physical- and Platform-Related Issues     737
    Counters     737
    CPU Packet Captures     743
    SPAN     748
Troubleshooting Endpoint Connectivity     751
    Endpoint Tracker and Log Files     752
    Enhanced Endpoint Tracker (EPT) App     756
    Rogue Endpoint Detection     758
Troubleshooting Contract-Related Issues     759
    Verifying Policy Deny Drops     764
Embedded Logic Analyzer Module (ELAM)     765
Summary     769
Review Key Topics     769
Review Questions     769
Chapter 14  The ACI Visibility & Troubleshooting Tool     771
Visibility & Troubleshooting Tool Overview     771
Faults Tab     772
Drop/Stats Tab     773
    Ingress/Egress Buffer Drop Packets     774
    Ingress Error Drop Packets Periodic     774
    Storm Control     774
    Ingress Forward Drop Packets     775
    Ingress Load Balancer Drop Packets     776
Contract Drops Tab     777
    Contracts     777
    Contract Considerations     778
Events and Audits Tab     779
Traceroute Tab     780
Atomic Counter Tab     782
Latency Tab     785
SPAN Tab     786
Network Insights Resources (NIR) Overview     787
Summary     790
Chapter 15  Troubleshooting Use Cases     791
Troubleshooting Fabric Discovery: Leaf Discovery     792
Troubleshooting APIC Controllers and Clusters: Clustering     795
Troubleshooting Management Access: Out-of-Band EPG     799
Troubleshooting Contracts: Traffic Not Traversing a Firewall as Expected     801
Troubleshooting Contracts: Contract Directionality     804
Troubleshooting End Host Connectivity: Layer 2 Traffic Flow Through ACI     807
Troubleshooting External Layer 2 Connectivity: Broken Layer 2 Traffic Flow Through ACI     812
Troubleshooting External Layer 3 Connectivity: Broken Layer 3 Traffic Flow Through ACI     814
Troubleshooting External Layer 3 Connectivity: Unexpected Layer 3 Traffic Flow Through ACI     816
Troubleshooting Leaf and Spine Connectivity: Leaf Issue     821
Troubleshooting VMM Domains: VMM Controller Offline     826
Troubleshooting VMM Domains: VM Connectivity Issue After Deploying the VMM Domain     829
Troubleshooting L4L7: Deploying an L4L7 Device     832
Troubleshooting L4L7: Control Protocols Stop Working After Service Graph Deployment     834
Troubleshooting Multi-Pod: BUM Traffic Not Reaching Remote Pods     837
Troubleshooting Multi-Pod: Remote L3Out Not Reachable     839
Troubleshooting Multi-Site: Using Consistency Checker to Verify State at Each Site     841
Troubleshooting Programmability Issues: JSON Script Generates Error     844
Troubleshooting Multicast Issues: PIM Sparse Mode Any-Source Multicast (ASM)     846
Summary     860
Appendix A  Answers to Chapter Review Questions     861
Index     873

Submit Errata