Deploying a Fast and Stable Wireless Mesh Network

This chapter provides practical tips and advice for deploying a fast and stable wireless mesh network.

The wireless mesh deployment access points (AP), along with the wireless controllers, allow you to provide a secure wireless solution for outdoor environments such as a college campus or an entire city. The mesh feature allows you to place APs in areas where you have no wired Ethernet network connection. A mesh deployment consists of a root AP (RAP) and one or more mesh APs (MAP). The RAP provides the wired link back to the controller for the MAPs. In order to avoid confusion throughout the rest of the chapter, when referring to the mesh series of APs, they will be called 'mesh APs', whereas the term 'MAP' will refer to a mesh AP that is connected to the controller wirelessly through a RAP.

Originally, only the 1500 and 1520 series APs, outdoor mesh APs, supported mesh deployments. In later code releases, however, Cisco introduced Enterprise Mesh, indoor mesh APs, which allows you to use 1130 and 1240 series APs to install a mesh wireless network indoors.

MAPs use their 802.11a radios as a wireless backhaul to join the controller and send client traffic to the wired network through the RAP. The wireless clients in a mesh network associate to the mesh AP's 802.11b/radio.

Currently, you can find four different models of outdoor mesh APs in the field:

• 1505
• 1510
• 1522
• 1524

The 1505 model is a 802.11b/g only radio, so client access and the backhaul use the single radio. The 1510 model has both an 802.11a and 802.11b/g radio, which allows the AP to have a dedicated radio for the backhaul and a dedicated radio for client access. The 1505 and 1510 series APs are difficult to troubleshoot because they do not have LEDs or standard console cable access. Therefore, without taking the units down and connecting them in a lab environment, it would be nearly impossible to determine if the APs were actually powered up and working. The 1500 series models are end of sale and no longer available for purchase. In November 2013, Cisco will no longer support the 1500 series APs.

The 152x series APs are a dramatic improvement over the 1500s. They have LEDs and standard console connections. You can even enable remote Telnet on the APs and run debug and show commands directly on the APs to help with problem analysis.

Mesh Code Releases

Before the 4.2 release of code, mesh features were included in the main code base for all controllers. Mesh development, bug fixes, and features, however, were being delayed because of mainline code release timelines. To facilitate the development of mesh and get bug fixes to customers faster, Cisco split mesh into its own code branch. The original mesh-only branch uses the 4.1 code base and retains the 4.1 naming convention with an M at the end. An example is 4.1.192.22M.

Although splitting the mesh and mainline code into two separate code bases allowed Cisco to release mesh enhancements and bug fixes faster, it introduced two limitations:

• With the early mesh releases, only the 1500 and 1520 series APs were supported.

  This meant that if you had other APs models, you had to have two controllers: one for mesh and one for the indoor APs. With mesh Release 4.1.191.24M, Cisco added support for the 1000, 1100, 1200, 1230, 1130, 1240, and 1300 series APs.

• Because the base code is 4.1, 1250, 1140, and 801 series APs are not supported.

Starting with the 5.2 release of code, Cisco merged the mesh-only and mainline code releases. The 1500 series APs are not supported on 5.2 code. To address security vulnerabilities inherent in the 4.1 code base for customers who have deployed 1500 series APs, Cisco released 4.2.176.51M code in June 2009.

Mesh Deployments

You can use mesh APs in various deployment scenarios:

• Point-to-point
• Point-to-multipoint
• Mesh

In a point-to-point mesh deployment, you would have a RAP and a single MAP bridging two wired networks using their 802.11a radios (see Figure 15-1).

With a point-to-multipoint mesh installation, you have a single RAP acting as a root bridge with two or more MAPs connecting their respective wired networks. Figure 15-2 shows a typical point-to-multipoint deployment.

If you plan to enable Ethernet bridging (discussed later in the "Ethernet Bridging" section) with a point-to-multipoint installation, it is recommended that you disable VLAN Trunking Protocol (VTP) on any switch with a connected MAP. VTP can reconfigure the trunked VLANs across the mesh network, making the RAP lose its connection to the controller and bring down the wireless network.

In a true mesh deployment, you would have a RAP, or several RAPs if the deployment is large, as well as MAPs deployed across the outdoor space. The MAPs would form parent-child relationships using the 802.11a backhaul (see Figure 15-3). The 802.11b/g radios would service your wireless clients.

As you can see in Figure 15-3, the MAPs relay their wireless connections using their 802.11a backhaul radios to the RAP. The RAP sends that traffic to the controller through the Lightweight Access Point Protocol (LWAPP)/Control and Provisioning of Wireless Access Points (CAPWAP) tunnel to the controller. The 802.11b/g radios on the APs provide client access to the wireless network. Although its not shown in Figure 15-3, clients can connect to the 802.11b/g radio on the RAP as well.

How Mesh Works

Mesh is a fairly complex feature that relies on a wireless routing protocol, Cisco Adaptive Wireless Path Protocol (AWPP), that allows the MAPs to determine the best parent to relay their traffic to the RAP.

AWPP is designed specifically for wireless mesh networking in that its path decisions are based on link quality and number of hops. AWPP is also designed to provide ease of deployment, fast convergence, and minimal resource consumption. AWPP takes advantage of the LWAPP/CAPWAP wireless LAN (WLAN), where client traffic is tunneled to the controller and hidden from the AWPP process. Also, the advanced radio management features in the LWAPP/CAPWAP WLAN solution are available to the wireless mesh network and do not have to be built into AWPP.

When a MAP comes up in a mesh network, the AP is authenticated (bridge authentication). Two possible security modes are available for the bridge authentication: Extensible Authentication Protocol (EAP) and Pre-Shared Key. There is a four-way handshake using this primary key to establish an Advanced Encryption Standard (AES) session. Next, the new AP establishes an LWAPP/CAPWAP tunnel to the controller and is authenticated against the MAC filter list of the controller.

The controller then pushes the bridge shared secret key to the AP via LWAPP/CAPWAP, after which it re-establishes the AES session with the parent AP.

As previously described, the wireless mesh bridges traffic between the MAPs and the RAPs. The traffic can be from wired devices being bridged by the wireless mesh or wireless client traffic that is encapsulated in LWAPP/CAPWAP-Data and LWAPP/CAPWAP-Control traffic from the MAPs.

The AES encryption is established as part of the MAP establishing neighbor relationships with other MAPs (bridge authentication). The bridge shared secret is used to establish unique encryption keys between mesh neighbors. All APs establish an LWAPP/CAPWAP connection to the controller through AES-encrypted backhaul tunnels between the APs.

Mesh Bootup and Join Process

When a mesh AP first boots up, it must determine whether it is a RAP. This decision-making process is as follows:

• Step 1. Upon boot, an AP checks its state; if it is a RAP, it enters the Maintain state.
• Step 2. If it is not a RAP, the AP scans all the channels for Bridge Group Names (BGN).
• Step 3. The AP actively solicits neighboring APs (Seek state).
• Step 4. The AP selects the best parent from the available list.
• Step 5. The AP authenticates to the Mesh network.
• Step 6. The AP then enters the Maintain state and is willing to respond to solicitations. Solicitation allows for faster convergence, leaving more time for data transfer.

Figure 15-4 illustrates the Mesh Machine state.

A mesh AP become a RAP if it can communicate with an LWAPP/CAPWAP controller through its Ethernet interface. All 1500 and 1520 series mesh APs ship as MAPs. If the mesh AP is a RAP, it can go straight to the Maintain state. In the Maintain state, the mesh AP has established an LWAPP/CAPWAP connection to the controller, so it does not need to seek other APs; rather, it simply responds to solicitations. If the mesh AP is not a RAP, it starts a scan process where it scans all available channels and solicits information from other mesh APs.

This behavior has two main implications:

• The RAP does not change channels; therefore, the channel used to build the mesh from a RAP is defined in the RAP configuration. By default, the RAP uses channel 161 if it is an outdoor AP.
• The mesh is built from the RAP out, because initially only the RAP can respond to solicitations.

If the AP is not a RAP, it follows the state diagram in Figure 15-4 in the following modes:

• Scan: The AP scans all the backhaul channels using mesh beaconing. This mechanism is similar to the 802.11 beaconing mechanisms used by wireless access networks, except the protocol frames conform to the AWPP frames on the backhaul. The frame used for beaconing is a broadcast NEIGHBOR_RESPONSE called NEIGHBOR_UPDATE and is sent unsolicited.

  Essentially, the network advertises NEIGHBOR_UPDATE frames so that new nodes can scan and quickly discover neighbors. The generation rule is that each RAP and MAP broadcast NEIGHBOR_UPDATE frames after being connected to the network (via a controller). Any neighbor updates with signal-to-noise ratios (SNR) lower than 10 dB are discarded. This process is called passive scanning.

  The AP looks for mesh beacons advertising the same BGN as what was configured on the AP when you primed it.

  If the AP hasn't been preconfigured or primed (joined to the controller on the wire, AP role, BGN set, and so on), or if the BGN it has been configured to is not seen in mesh beacons, the AP goes into default mode and proceeds with joining. This allows you to "catch" an AP in case of a config issue. Nevertheless, Cisco recommends that you always prime the APs with the BGN.

• Seek: After the AP decides to join a mesh network it has located, bridge authentication takes place. This is done based on PSK or EAP. The AP solicits members of the mesh network. Successful responses to these solicitations become neighbors. These neighbors must have the same bridge group name and same shared secret.
• Sync: The MAP learns the path information from each of its neighbors, and the neighbor with the greatest ease becomes the parent of the soliciting MAP. If the neighbors report multiple RAPs, the RAP with the greatest ease is chosen.
• Authenticate: The MAP authenticates to the controller through a connection established through its parent AP. This AP authentication is standard LWAPP/CAPWAP AP authentication, and the MAP is already part of the mesh and using the mesh to communicate with its controller. Because MAPs are always in bridge mode, in addition to the standard LWAPP/CAPWAP authentication (bridge authentication during the Seek state), the controller requires mandatory MAC address authorization. Should you fail to add the AP's MAC address to the controller's MAC filter list (see Figure 15-6), the AP will not be able to proceed beyond the Authenticate state.
• Maintain: The MAP responds to other MAP solicitations and regularly solicits to determine any changes in the mesh. It is only after entering the Maintain state that the MAP is visible to the controller and Wireless Control System (WCS). Note that in the Maintain state, the solicitations occur only on the channel defined by the RAP, whereas a MAP in seek mode solicits on all channels, only stopping when it has found a parent AP.

AWPP uses ease to determine the best path. Ease can be considered the opposite of cost, and the preferred path is the path with the higher ease.

Ease is calculated using the SNR and hop value of each neighbor and applying a multiplier based on various SNR thresholds. The purpose of this multiplier is to apply a spreading function to the SNRs that reflects various link qualities.

In Figure 15-5, MAP2 prefers the path through MAP1 because the adjusted ease (436906) though this path is greater than the ease value (262144) of the direct path from MAP2 to RAP.

A parent AP is chosen by using the adjusted ease, which is the ease of each neighbor divided by the number of hops to the RAP:

So in Figure 15-5, the adjusted case for MAP2 through MAP1 is 873812/2 = 436906.

Configuring Mesh

Configuring the controller to support mesh is quite simple. All you need to do is add the MAC address of the mesh AP to the MAC Filter list of the controller (see Figure 15-6). For 152x outdoor mesh APs, use the Bridge-Group Virtual Interface (BVI) MAC address of the mesh AP. For 1130 and 1240 series indoor mesh APs, you would use the Ethernet MAC address. If you do not know the MAC of the AP and it is not on the exterior of the unit, you can use the console to determine the BVI and Ethernet MAC addresses using the command sh int | i Hardware. You can also run debug pm pki enable from the controller command-line interface (CLI) to see the MAC address of the AP that is trying to join, as demonstrated in Example 15-1. The description for the AP in the MAC filter list is simply a text string you enter; it has nothing to do with actually configuring the AP.

Example 15-1. debug pm pki enable Command Output

Fri Jul 10 17:55:37 2009: spamMeshRadiusProcessResponse: AP Authorization failure
for 00:0b:85:6f:9b:90

If the APs' MAC address is not listed, the AP fails authentication and cannot join the controller. This is true for both the indoor and outdoor mesh APs. When the AP rejoins the controller, set its role to be either a RAP or MAP (see Figure 15-7) using the Mesh tab. For the indoor APs, change the AP mode to Bridge and reboot it (see Figure 15-8) before you can configure the mesh-specific settings. When the AP reboots in bridge mode, you see the Mesh tab on the AP configuration page.

The Mesh tab is where you will set the AP role as well as the BGN and backhaul data rates. This is true for both outdoor and indoor MAPs.

You can use the BGN to break large mesh deployments into sectors so that only certain APs will form parent-child relationships.

By default, the security mode for bridge authentication is EAP. This is done using manufacturer-installed certificates and therefore does not need configuring. Should you want to change the security mode to PSK or use a RADIUS server to authenticate the mesh APs, you can configure those settings from the controller GUI under Wireless > Mesh, as shown in Figure 15-9.

Ethernet Bridging

Ethernet bridging allows you to connect remote wired networks to each other using the Ethernet port of the MAPs. A common use for Ethernet bridging is installing video cameras or street poles with the mesh APs. For bridging to work, every MAP and RAP in the path must have Ethernet bridging enabled.

Prior to code Release 5.2, Ethernet bridging only allowed the extension of the Layer 2 network in which the MAPs resided. So if the APs had IP addresses in VLAN 5, for example, you could only extend VLAN 5 to the remote wired network. The 5.2 release allows you to bridge multiple VLANs. Like the earlier feature, every AP in the mesh path back to the RAP and including the RAP must support bridging the same VLANs as the MAP with the wired connection. Figure 15-10 illustrates this concept.

If you do not allow the desired VLANs on all the MAPs, then in the event of a failure within the mesh network it is possible to break the bridging feature if a MAP in the new path to the RAP does not support a particular VLAN. In Figure 15-10, if MAP1 were to go down and MAP3 changed its parent to MAP2, the Ethernet bridging on MAP3 would fail for VLAN 2 because MAP2 does not support bridging VLAN 2.

After you have enabled Ethernet bridging support on your mesh APs you need to configure the VLAN tagging. Figure 15-11 shows the Ethernet configuration of an indoor RAP, and Figure 15-12 shows the same configuration on the indoor MAP.

The RAP Ethernet port is configured as a trunk port with VLAN 20 set to Native and allowing VLAN 12. You can add more VLANs by entering the VLAN into the Trunk VLAN ID box and clicking Add. With the Ethernet port set to Trunk, the AP accepts both tagged and untagged packets. Any tagged packets for a VLAN that is not in the allowed list are dropped.

Because the MAP is only bridging VLAN 12 in this case, the Ethernet port mode is Access. The AP tags the incoming untagged packet and forwards it to the RAP. Any tagged packets are dropped.

Mesh APs use VLAN transparency to perform Ethernet bridging when extending the Layer 2 network. To allow multiple VLAN bridging/tagging, you must disable VLAN transparency (see Figure 15-13) under the Wireless>Mesh>Ethernet Bridging section on the controller. When VLAN transparency is enabled, VLAN processing does not occur. This assumes that all traffic is destined to and from the same VLAN with no 802.1 tagging.

After you have disabled VLAN transparency, reboot the mesh APs for that setting to take effect.

It is important to understand the traffic flow when using Ethernet bridging. Figure 15-14 shows the traffic flow for both wired and wireless clients within the mesh network with Ethernet bridging enabled.

As you can see, with Ethernet bridging enabled, the traffic flow for wireless clients is unchanged. The wireless client packets are sent using LWAPP/CAPWAP data, which is sent through the encrypted backhaul to the controller. The controller then bridges that traffic to the wired network. The bridged wired client traffic, however, is bridged directly into the backhaul toward the RAP. The RAP then bridges the traffic directly onto the wired network. The wired bridged traffic is not sent back to the controller.

Several guidelines exist in addition to disabling VLAN transparency that allow the correct VLANs on the APs when you use the Ethernet bridging and VLAN tagging feature in 5.2 code:

• For security reasons, the Ethernet port on a mesh AP (RAP and MAP) is disabled by default. It is enabled by configuring Ethernet bridging on the MAP port.
• Ethernet bridging must be enabled on all the APs in the mesh network to allow Ethernet VLAN tagging to operate.

• VLAN mode must be set as non-VLAN transparent (global mesh parameter).

  VLAN transparent is enabled by default. To set as non-VLAN transparent, you must uncheck the VLAN transparent option in the global mesh parameters window.

• VLAN configuration on a mesh AP is applied only if all the uplink MAPs are able to support that VLAN.
• If uplink APs are not able to support the VLAN, the configuration is stored rather than applied.

• VLAN tagging can be configured only on Ethernet interfaces.

  On 152x mesh APs, three of the four ports can be used as secondary Ethernet interfaces: port 0-PoE in, port 1-PoE out, and port 3- fiber. Port 2 - cable cannot be configured as a secondary Ethernet interface.

  In Ethernet VLAN tagging, port 0-PoE in on the RAP connects to the trunk port of the switch of the wired network. Port 1-PoE out on the MAP connects to external devices such as video cameras.

• Backhaul interfaces (802.11a radios) act as primary Ethernet interfaces.

  Backhauls function as trunks in the network and carry all VLAN traffic between the wireless and wired network. No configuration of primary Ethernet interfaces is required.

• The switch port in the wired network that is attached to the RAP (port 0-PoE in) must be configured to accept tagged packets on its trunk port. The RAP forwards all tagged packets received from the mesh network to the wired network.
• No configuration is required to support VLAN tagging on an 802.11a backhaul Ethernet interface within the mesh network. This includes the RAP uplink Ethernet port. The required configuration happens automatically using a registration mechanism. Any configuration changes to an 802.11a Ethernet link acting as a backhaul are ignored and a warning results. When the Ethernet link no longer functions as a back-haul, the modified configuration is applied.
• VLAN configuration is not allowed on a port-02-cable modem port of an 152x AP. VLANs can be configured on ports 0 (PoE-in), 1 (PoE-out), and 3 (fiber).
• If you are bridging between two MAPs, enter the distance (mesh range) between the two APs that are bridging. (This is not applicable to applications in which you are forwarding traffic connected to the MAP or to the RAP access mode.)
• Up to 16 VLANs are supported on each sector. Therefore, the cumulative number of VLANs supported by RAP's children (MAPs) cannot exceed 16.
• Ethernet ports on APs function as either access or trunk ports within an Ethernet tagging deployment.
• In Access mode, only untagged packets are accepted. All packets are tagged with a user-configured VLAN called access VLAN. For this mode to take effect, the global VLAN mode should be non-VLAN transparent. This option is used for applications in which information is collected from devices connected to the MAP, such as cameras or PCs, and then forwarded to the RAP. The RAP then applies tags and forwards traffic to a switch on the wired network.
• Trunk mode requires the user to configure a native VLAN and an allowed VLAN list (no defaults). In this mode, both tagged and untagged packets are accepted. Untagged packets are always accepted and are tagged with the user-specified native VLAN. Tagged packets are accepted if they are tagged with a VLAN in the allowed VLAN list. For this mode to take effect, the global VLAN mode should be non-VLAN transparent. This option is used for bridging applications such as forwarding traffic between two MAPs residing in separate buildings within a campus.

• The switch port connected to the RAP must be a trunk.

  The trunk port configuration on the switch and the RAP trunk port must match.

• A configured VLAN on a MAP Ethernet port cannot function as a management VLAN.

• The RAP must always connect to the native VLAN (ID 1) on a switch.

  The RAP's primary Ethernet interface is by default the native VLAN of 1.

Troubleshooting Mesh

Mesh deployments add an extra level of complexity to troubleshooting because the connection between the MAPs and the controller is wireless. The MAPs need to have good signal strength to their parent AP. You do not want the SNR up and SNR down to differ by more than 10 dTS. The Fresnel zone between the APs also needs to be clear to prevent obstructions from interfering with the connection. You want to make sure that the APs are mounted correctly according to the installation guides and have similar heights. Mesh APs can transmit as far as 25 miles, so depending on the distance between them and the type of antenna, you might need to use a laser or other professional alignment tool.

AP Join Problems

Just like any controller-based AP, mesh APs have to be joined to a controller before they can start to service clients. Before the mesh network can come up, you need to have at least one RAP with a wired connection to the controller. After the RAP is joined, the MAPs can start to join the controller through the 802.11a radio of the RAP.

To troubleshoot mesh AP join issues, you can use the standard LWAPP/CAPWAP debugs on the controller that you would use for a non-MAP such as a 1242. In addition to those debugs, you can run the following debugs on the controller:

• debug mesh security events enable
• debug mesh security message enable
• debug dot1x events enable
• debug dot1x packet enable

The output of these debugs shows you if you have AP authentication issues. Example 15-2 shows sample output for debug mesh security all. (The all keyword includes events and messages.)

Example 15-2. debug mesh security all Command Output

*May 03 13:36:00.846:  00:1C:B1:07:FA:20 MESH_ASSOC_REQUEST_PAYLOAD in Association
Request for AP 00:0B:85:65:51:60
*May 03 13:36:00.846:  00:1C:B1:07:FA:20 Mesh assoc request for known AP
00:0b:85:65:51:60
*May 03 13:36:00.846:  00:1C:B1:07:FA:20 Mesh assoc request :child :
00:0b:85:65:51:60 NextHop : 00:23:5d:f1:9d:41  LradIp 192.168.1.200  vlanid: 0
mwarPort: 5246  lradPort: 35184
*May 03 13:36:00.846:  00:1C:B1:07:FA:20 Request MAC authorization for AP Child
Addr:  00:0b:85:65:51:60 AP Identity: 00:0b:85:65:51:60 AWPP Identity:
00:1c:b1:07:fa:2f
*May 03 13:36:00.847: MAC Validation of Mesh Assoc Request for00:0b:85:65:51:60 is
-4, Mode is : 0
(Cisco Controller) >*May 03 13:36:00.847: MAC authoriztion fail. REsetting MSCB
state for 00:0b:85:65:51:60 to 9

From this output, you can tell that the MAP that is trying to join is failing MAC authorization. The root cause might be a missing or incorrect MAC address in the MAC filter list of the controller or a similar problem on a RADIUS server if you are using Authentication, Authorization, and Accounting (AAA) to authenticate the APs. The controller traplogs also contain valuable information about any AP join issues:

Mesh Node '00:0b:85:65:51:60' failed to join controller, MAC address not in MAC
filter list

If you have console access to the AP that is not joining the controller (152x and indoor mesh-APs), you can enable the following debugs:

• debug mesh adjacency
• debug mesh event
• debug mesh link

Using these debugs on the MAP closest to the AP that is not joining (the RAP if no MAPs are joining), you can see the adjacencies and events. Example 15-3 shows sample output from debug mesh adjacency and mesh event.

Example 15-3. debug mesh adjacency and debug mesh event Command Output

*May  3 16:54:36.582: ADJ:Processing  child  adjacency  from  001e.1306.f65f  channel  64
*May  3 16:54:36.582: mesh_adj_add_association: client exists 001e.1306.f65f
*May  3 16:54:36.863: mesh_adj_current_backhaul: Dot11Radio1
*May  3 16:56:22.662: ADJ:Child 001e.1306.f65f timed out
*May  3 16:56:22.662: %MESH-6-LINK_UPDOWN: Mesh station 001e.1306.f65f link Down
*May  3 16:56:22.663: MESH_EVENT:mesh_lwapp_link_down: link 001e.1306.f65f
*May  3 16:56:22.665: MESH_EVENT:received mesh_lwapp_handle_request type LINK_DOWN

The sample output in Example 15-3 from a RAP shows the child adjacency timing out and a mesh link down event.

If the mesh APs are not in the same VLAN as the management interface of the controller, be sure to check your discovery methods. Incorrect DHCP options, for example, prevent the APs from learning the correct controller IP. Table 15-1 outlines the correct Vendor Class Identifiers (VCI) for the different outdoor mesh APs should you decide to use DHCP options for controller discovery.

Table 15-1. Outdoor Mesh APs DHCP VCI Strings

MAP	Code Release	VCI String
Any 1500	4.1	Cisco AP c1500
1500 OAP	4.0	Cisco AP.OAP1500
1505	4.0	Cisco AP.LAP1505
1510	4.0	Cisco AP.LAP1510
Any 1500	3.2	Airespace.AP1200
1520	4.1M or 5.2	Cisco AP c1520

An interesting behavior for the 1500 series APs with older code is that if they cannot join a controller for an extensive period of time, they can revert to a previously installed image. Therefore, an AP that was running 4.0 code could fall back to a 3.2 image. If this happens, the VCI you had set up would no longer be valid. You would need to get a wired packet capture to determine the VCI string the AP was sending to the DHCP server. The APs no longer exhibit this image fallback behavior in the 4.1 mesh-only releases and higher.

Another mesh behavior that can result in long join times is that, by design, a mesh AP gives precedence to its wired port on bootup if it detects a signal. This is common when using Ethernet bridging. Only after several failed attempts does it try to use the 802.11a radio as the primary backhaul when the Ethernet port is live. To decrease the join time, you can disconnect the device connected to the Ethernet port of the MAP.

Make sure you are using the correct APs for RAPs and MAPs. Although most mesh APs can be a RAP or MAP to a different AP model, the 1505 APs can mesh only with each other.

If you are using BGNs, a blank or incorrect BGN prevents a MAP from staying joined to the controller. If an AP has a blank BGN, it remains joined for approximately 30 minutes before it drops off the network. You need to correct the BGN during this time to prevent it from continuously dropping off the network. Should you ever want to change the BGN or backhaul data rate (not recommended), you should always start with the farthest out MAP to prevent stranding an AP.

RF Issues

Just like the radio frequency (RF) environment can affect wireless clients, it can disrupt a mesh network. High-channel utilization, interference, and radar can wreak havoc on a mesh network.

Wireless radios that operate in the UNII-2 and UNII-2 Extended bands must adhere to Dynamic Frequency Selection (DFS) and change channels when the AP detects radar on that channel. The controllers automatically enable DFS on 15 channels (52 through 140). If an AP detects radar on the channel it is currently using, it scans for a new channel and waits 60 seconds to make sure no radar is on that channel before it starts using it. An AP avoids trying to use the previous channel for 30 minutes. Even if you hard-code the AP channel and do not use Auto-RF, the AP must change channels when it detects radar.

As demonstrated in Example 15-4, when an AP detects radar and changes channels, it announces the change to any child APs. The radio change event also generates trap logs to let you know why the AP changed channels.

Example 15-4. DFS Radar Event SNMP Trap Log

Mon Feb 9 13:07:00 2009 Channel changed for Base Radio MAC: 00:1d:46:24:43:a0 on
802.11a radio. Old Channel: 60. New Channel: 161. Why:  Radar. Energy
before/after change:
0/0. Noise before/after change: 0/0. Interference before/after change: 0/0.
Mon Feb 9 13:07:00 2009 Radar signals have been detected on channel 60 by 802.11a
radio with MAC: 00:1d:46:24:43:a0 and slot 1

A radar event requires your mesh network to change, which causes outages until the network converges after the channel changes.

Besides radar, you might see poor throughput for various other reasons. Table 15-2 outlines some common RF issues, their cause, and potential resolutions.

Table 15-2. Common RF Symptoms and Resolutions

Symptom	Possible Causes	Potential Solution
Throughput is low; link test Rx is significantly less than Tx	Tx power is low	Check Tx power level
	Antenna alignment is poor	Realign antennas
	LOS1 is obstructed or Fresnel zone is not clear	Remove obstruction or raise or move APs/antennas
Throughput is low; link test Rx is significantly higher than Tx	RF has interference	Change radio channel
SNR uplink and downlink are off by 10 dB or more	RF has interference, LOS is obstructed, or hardware is bad	Change radio channel, check LOS, check hardware

The number of hops from a MAP to a RAP is also a factor. With the half-duplex nature of wireless, you are cutting your backhaul rates essentially in half with each hop. For example, the maximum throughput for an 18 Mbps backhaul is approximately 10 Mbps for the first hop, 5 Mbps for the second hop, and 2.5 Mbps for the third hop. Although 8 hops is the backhaul hop limit for outdoor mesh APs, Cisco recommends no more than 4 hops from RAP to MAP.

show Commands

Several show commands are available on the controllers that you can use to see the state of your mesh deployment and help pinpoint a problem area.

Example 15-5 reveals the mesh show commands.

Example 15-5. show mesh Command List

(MeshController) >show mesh ?
env                   Show mesh environment.
neigh                 Show AP neigh list.
path                  Show AP path.
astools               show mesh astools list
stats                 Show AP stats.
secbh-stats           Show Mesh AP secondary backhaul stats.
per-stats             Show AP Neighbor Packet Error Rate stats.
queue-stats           Show AP local queue stats.
security-stats        Show AP security stats.
ap                    Show mesh AP information.
config                Show mesh configurations.
secondary-backhaul    Show mesh secondary-backhaul
client-access         Show mesh backhaul with client access.
public-safety         Show mesh public safety.
background-scanning   Show mesh background-scanning state.
cac                   Show mesh cac.
bhrateadapt           Show Mesh Backhaul Rate Adaptation State.

Example 15-6 shows some sample output from the show mesh neigh summary command.

Example 15-6. show mesh neigh summary Command Output

(Cisco Controller) >show mesh neigh summary 1131
AP Name/Radio Mac  Channel Snr-Up Snr-Down Link-Snr Flags    State
––––––––––––––––-  ––––––- –––––– –––––––– –––––––– ––––––-  ––––-
00:18:74:FB:1E:FF  36      13      13       13        0x860    BEACON
00:1C:F9:05:9D:DF  36      13      13       13        0x860    BEACON
00:1D:A1:CD:DD:6C  64      53      53       53        0x960    CHILD BEACON

With show mesh neigh, you can see the information from any child or parent APs as well as any other surrounding APs the AP hears. Here you can see three neighbors, one of which is a child AP.

To see more detailed information about the neighbors of an AP, use show mesh neigh detail, as demonstrated in Example 15-7.

Example 15-7. show mesh neigh detail Command Output

(Cisco Controller) >show mesh neigh detail 1131
AP MAC : 00:0B:85:65:51:60
FLAGS : 1160 CHILD DEFAULT
worstDv 255, Ant 0, channel 64, biters 0, ppiters 10
Numroutes 0, snr 0, snrUp 13, snrDown 13, linkSnr 13
adjustedEase 0, unadjustedEase 0
txParent 0, rxParent 0
poorSnr  0
lastUpdate   1241358733 (Sun May  3 13:52:13 2009)
parentChange 0
Per antenna smoothed snr values: 0 0 0 0
Vector through 00:0B:85:65:51:60
AP MAC : 00:18:74:FB:1E:FF
FLAGS : 860 BEACON
worstDv 255, Ant 0, channel 36, biters 0, ppiters 0
Numroutes 0, snr 0, snrUp 13, snrDown 13, linkSnr 13
adjustedEase 0, unadjustedEase 0
txParent 0, rxParent 0
poorSnr  0
lastUpdate   1241358199 (Sun May  3 13:43:19 2009)
parentChange 0
Per antenna smoothed snr values: 0 0 0 0
Vector through 00:18:74:FB:1E:FF
AP MAC : 00:1C:F9:05:9D:DF
FLAGS : 860 BEACON
worstDv 255, Ant 0, channel 36, biters 0, ppiters 0
Numroutes 0, snr 0, snrUp 13, snrDown 13, linkSnr 13
adjustedEase 0, unadjustedEase 0
txParent 0, rxParent 0
poorSnr  0
lastUpdate   1241358199 (Sun May  3 13:43:19 2009)
parentChange 0
Per antenna smoothed snr values: 0 0 0 0
Vector through 00:1C:F9:05:9D:DF
AP MAC : 00:1D:A1:CD:DD:6C
FLAGS : 860 BEACON
worstDv 255, Ant 0, channel 64, biters 0, ppiters 0
Numroutes 0, snr 0, snrUp 0, snrDown 0, linkSnr 0
adjustedEase 0, unadjustedEase 0
txParent 0, rxParent 0
poorSnr  0
lastUpdate   1241358245 (Sun May  3 13:44:05 2009)
parentChange 0
Per antenna smoothed snr values: 0 0 0 0
Vector through 00:1D:A1:CD:DD:6C

Using show mesh ap tree, as demonstrated in Example 15-8, you can see the logical parent child mappings, the hop count, the SNR, and the BGN.

Example 15-8. show mesh ap tree Command Output

(Cisco Controller) >show mesh ap tree
 =======================================================
||  AP Name [Hop Counter, Link SNR, Bridge Group Name] ||
 =======================================================
[Sector 1]
—————————
1131[0,0,leelab]
  |-1242[1,50,leelab]
———————————————————————————————————————————————————
Number of Mesh APs...............................  2
Number of RAPs...................................  1
Number of MAPs...................................  1

As demonstrated in Example 15-9, the show mesh path command displays the path through the mesh network to the AP you specify.

Example 15-9. show mesh path Command Output

 (Cisco Controller) >show mesh path 1242
AP Name/Radio Mac  Channel Snr-Up Snr-Down Link-Snr Flags      State
––––––––––––––––-  ––––––- –––––– –––––––– –––––––– ––––––-    ––––-
1131               64      61     54       51       0x86e      NEIGH PARENT  BEACON
1131               is a Root AP.

This output also displays the channel in use and the SNR values. The MAP 1242 in this case is directly connected to the RAP 1131.

Remote Telnet and AP Debugs

You can gain a wealth of information from running show commands on the MAPs.

For the 1500 series APs, you need to run remote debug commands. To enable remote debugs for an AP, use the following command from the controller CLI:

debug ap enable AP_name

After you enable remote debugging on the AP, you can run debug commands to pull information directly from the AP:

   debug ap command "command" AP_name

Some of the commands you can run on the 1500s are as follows:

• printRadar(): Displays radar information
• printBsnRateTable(1): Displays valid radio data rates
• apCfgRadioChannelGet(0): Displays current radio channel
• keyShow(0): Displays encryption key entries per radio
• apPrintForwardList: Displays client transmit statistics
• sibSmeShow(0): Displays association table
• sibStationShow(0,3): Displays association table details
• sibShow(0,3): Displays additional client information
• sibAgingShow(): Displays queue/frame information per MAC
• dumpMeshSecBhStats(0): Displays mesh backhaul stats
• dumpAp(): Displays MAP info
• dumpAdjs(): Displays mesh adjacencies
• spamPrintRcb: Displays AP control block information
• ifShow: Displays interface information and AP IP address
• bsnPrintCrashData(): Displays AP crash data
• SpamPrintCfgFile(): Displays current AP configuration

Example 15-10 demonstrates some sample output from debug ap command dumpAdjs().

Example 15-10. debug ap command dumpAdjs()Command Output

(MeshController) >debug ap command "dumpAdjs()" lab5map1510
(MeshController) >Sun May  3 09:07:46 2009: lab5map1510: Calling "dumpAdjs" with
args 0x0, 0x0, 0x0, 0x0
Sun May  3 09:07:46 2009: lab5map1510: ADJ 1 Identity 001b.d4a6.f3e4 MA: 001c.0e
75.240f version 0x20 Error/10K txpkts 6181
Sun May  3 09:07:46 2009: lab5map1510: Flags: CHILD BEACON
Sun May  3 09:07:46 2009: lab5map1510:  Minor ver: 32, worstDv 255 Ant 0, channe
l 0, biters 0, ppiters 10
Sun May  3 09:07:46 2009: lab5map1510:  Numroutes 0, snr 0, snrUp 8 snrDown 0 li
nkSnr 0 blistExp 3 bliters 0
Sun May  3 09:07:46 2009: lab5map1510:  adjustedEase 0 unadjustedEase 0 txParent
0 rxParent 0
Sun May  3 09:07:46 2009: lab5map1510:  Secondary backhaul channel: 0 BGN  Last
exclusion cause:0
Sun May  3 09:07:46 2009: lab5map1510:  Vector through 001c.0e75.240f:
Sun May  3 09:07:46 2009: lab5map1510:  Per antenna smoothed snr values: 0 0 0 0
Sun May  3 09:07:46 2009: lab5map1510:  Subordinate neighbors: 001b.d4a6.f3e4
Sun May  3 09:07:46 2009: lab5map1510:  Time since last update: 0 Days, 00:00:02
Sun May  3 09:07:46 2009: lab5map1510: dumpAdjs Returns: 0

This output shows you the uplink and downlink SNRs for any parent or child AP as well as any other neighbors the AP hears. This particular AP is RAP with no children.

Obviously, remembering these commands and what they do is difficult. You could run several other commands from the controller CLI to get information on the different quality of service (QoS) queues, Differentiated Services Code Point (DSCP), Cisco Discovery Protocol (CDP), and more. In the 4.2 mesh release of code, Cisco has added new commands specific to the 1510 series APs, as the output in Example 15-11 reveals.

Example 15-11. show mesh 1510-ap Commands

(Cisco Controller) >show mesh 1510-ap   ?
QoS-info - Displays various Queues on Mesh Backhaul
mesh-entries - Displays Mesh Association Table Entries
backhaul-stats - Displays Mesh Backhaul Statistics
client-stats - Displays Access Radio and Client information
mesh-aging - Displays Mesh Aging Parameters
11h-info - Displays 802.11h Management Frame Parameters
secBh-stats - Displays Secondary Backhaul Statistics
mesh-adjs - Displays Mesh Adjacencies
interface-info - Displays Ethernet and Loopback Interface Information
lwapp-config - Displays LWAPP Configuration

These new commands make viewing 1510 information easier because they are easier to remember and can combine several commands into one. For example, QoS-info includes dDetails, qShow, showQreset, and dumpQueueDrops output.

For the 1520 series APs, you can enable Telnet or run remote debugs from the controller:

debug ap enable AP_name
debug ap command "test mesh enable telnet" AP_name

Once Telnet is enabled, you can Telnet to the AP's IP address and run commands directly on the AP instead of remotely through the controller like on the 1510s. You can run the following commands on the 152x AP:

• show mesh adj all: Displays mesh adjacencies
• show mesh dfs history: Displays radar history
• show mesh dfs channel channel: Displays the current DFS channel
• show controllers dot111: Displays A radio
• show controllers dot110: Displays B/G radio
• show interface dot111: Displays A radio stats
• show interface dot110: Displays B/G radio stats
• show ip int brief: Displays IP information
• config mesh linktest src AP name dst AP name 18 100: Runs a link test with a back-haul rate of 18 and 100 pps

Ethernet Bridging Troubleshooting

Before the 5.2 code release, troubleshooting Ethernet bridging with mesh APs was almost impossible. You had to rely on packet captures and Ethernet interface statistics.

Cisco added several show and debug commands with the 5.2 release to help troubleshoot and view VLAN tagging information.

From the controller, you can verify your VLAN tagging configuration using the following commands:

• show ap config ethernet AP_name
• show mesh config

The show ap config ethernet command shows you the mode of the Ethernet port, the native VLAN, and any allowed VLANs, as demonstrated in Example 15-12.

Example 15-12. show ap config ethernet Command Output

Cisco Controller) >show ap config ethernet 1131
Vlan Tagging Information For AP 1131
Ethernet 0
Mode: TRUNK
                                                                      Native  Vlan  12

Allowed Vlans:

The show mesh config command shows the status of VLAN transparency mode, as demonstrated in Example 15-13.

Example 15-13. show mesh config Command Output

(Cisco Controller) >show mesh config
Mesh Range....................................... 12000
Backhaul with client access status............... disabled
Background Scanning State........................ enabled
Mesh Security
   Security Mode................................. EAP
   External-Auth................................. disabled
   Use MAC Filter in External AAA server......... disabled
   Force External Authentication................. disabled
Mesh Alarm Criteria
   Max Hop Count................................. 4
   Recommended Max Children for MAP.............. 10
   Recommended Max Children for RAP.............. 20
   Low Link SNR.................................. 12
   High Link SNR................................. 60
   Max Association Number........................ 10
   Association Interval.......................... 60 minutes
   Parent Change Numbers......................... 3
   Parent Change Interval........................ 60 minutes
   Mesh Multicast Mode.............................. In-Out
   Mesh Full Sector DFS............................. enabled
   Mesh Ethernet Bridging VLAN Transparent Mode..... disabled

As you can see in Example 15-13, show mesh config shows the entire mesh configuration for a controller.

Here are the mesh ap debug commands:

• debug mesh ethernet bridging: Debugs Ethernet bridging
• debug mesh ethernet config: Debugs access and trunk port configuration
• debug mesh ethernet registration: Debugs VLAN registration protocol
• debug mesh forwarding table: Debugs forwarding table containing bridge groups
• debug mesh forwarding packet bridge-group: Debugs the bridge group packet forwarding

Here are the mesh ap show commands:

• show mesh forwarding table: Shows all the bridges with their mac-table entries.
• show mesh forwarding vlan mode: Displays VLAN-Transparent mode.
• show mesh forwarding vlan statistics: Displays VLAN forwarding statistics.
• show mesh forwarding vlans: Shows all the supported VLANs.
• show mesh forwarding interfaces: Displays the bridge groups and the interfaces contained in them. Useful for troubleshooting the bridge group membership.
• show mesh ethernet vlan statistics: Shows Ethernet subinterface statistics.

Example 15-14 demonstrates output from the show mesh forwarding table command.

Example 15-14. show mesh forwarding table Command Output

(Cisco Controller) >debug ap command "show mesh forwarding table" 1242
(Cisco Controller) >*May 03 13:13:49.219: 1242:
*May 03 13:13:49.219: 1242: Mesh Forwarding Table Entries
*May 03 13:13:49.219: 1242:
*May 03 13:13:49.219: 1242: Bridge Group 1   Vlan 1 :
*May 03 13:13:49.219: 1242:   0023.5df1.9d40, nh 001c.b107.fa2f, swif Virtual-
Dot11Radio1:BACKHAUL: 43 : 8
*May 03 13:13:49.219: 1242:   0023.5df1.9d06, nh 001c.b107.fa2f, swif Virtual-
Dot11Radio1:BACKHAUL: 43 : 8
*May 03 13:13:49.219: 1242:   001c.b107.fa2f, nh 001c.b107.fa2f, swif Virtual-
Dot11Radio1:AWPP: 23 : 8
*May 03 13:13:49.219: 1242:   001c.58dc.97b0, nh 001c.b107.fa2f, swif Virtual-
Dot11Radio1:BACKHAUL: 43 : 7
*May 03 13:13:49.219: 1242:   0010.a4e5.2056, nh 001c.b107.fa2f, swif Virtual-
Dot11Radio1:BACKHAUL: 43 : 8
*May 03 13:13:49.219: 1242:   0001.0386.1991, nh 001c.b107.fa2f, swif Virtual-
Dot11Radio1:BACKHAUL: 43 : 8
*May 03 13:13:49.219: 1242:
*May 03 13:13:49.219: 1242: Bridge Group 2   Vlan 12 :
*May 03 13:13:49.219: 1242:
*May 03 13:13:49.219: 1242: Table size: 128, count 3
*May 03 13:13:49.219: 1242:   0023.5df1.9d06 Virtual-Dot11Radio1  LEARNED  (life  300)
*May 03 13:13:49.219: 1242:   000f.b049.5898 FastEthernet0 LEARNED (life 300)
*May 03 13:13:49.219: 1242:   0023.5df1.9d43 Virtual-Dot11Radio1  LEARNED  (life  300)

From this output you can see that this AP has two bridge groups: one for VLAN 1 and the other for VLAN 12. There is a wired client in VLAN 12 with the MAC address of 000f.b049.5898.

The output from show mesh forwarding vlan mode tells you whether VLAN transparency is enabled or disabled on the AP, as demonstrated in Example 15-15.

Example 15-15. show mesh forwarding vlan mode Command Output

(Cisco Controller) >debug ap command "show mesh forwarding vlan mode" 1242
(Cisco Controller) >*May 03 13:15:53.025: 1242:
*May 03 13:15:53.025: 1242:  Vlan Transparent mode DISABLED
To see the VLANs that the AP is bridging, use show mesh forwarding vlan:
(Cisco Controller) >*May 03 13:15:17.697: 1242:
*May 03 13:15:17.697: 1242: Mesh Forwarding Vlans
*May 03 13:15:17.697: 1242: Vlan: 1      Supporting Bridge Group: 1
*May 03 13:15:17.697: 1242: Vlan: 12     Supporting Bridge Group: 2

To see transmit and receive stats for the bridged VLANs, use show mesh ethernet vlan statistics, as demonstrated in Example 15-16.

Example 15-16. show mesh ethernet vlan statistics Command Output

(Cisco Controller) >debug ap command "show mesh ethernet vlan statistics" 1242
Interface               Rx Packets   Tx Packets  Tbridge Reject
FastEthernet0                  119           41                 0

This output is helpful in determining whether traffic is passing over the bridge links.

To troubleshoot bridge group membership, use show mesh forwarding interfaces, as demonstrated in Example 15-17.

Example 15-17. show mesh forwarding interfaces Output

(Cisco Controller) >debug ap command "show mesh forwarding interfaces" 1242
(Cisco Controller) >*May  03 13:19:26.714: 1242: Bridge Group 1: Ethernet
Bridging  enabled
*May 03 13:19:26.714: 1242:   Virtual-Dot11Radio1: Virtual-Dot11Radio1(state is
OPEN)
*May 03 13:19:26.714: 1242:       Node 001c.b107.fa2f
*May 03 13:19:26.714: 1242: Bridge Group 2: Ethernet Bridging enabled
*May 03 13:19:26.714: 1242:   FastEthernet0: FastEthernet0(state is OPEN)
*May 03 13:19:26.714: 1242:       Node 0083.e574.0b0d
*May 03 13:19:26.714: 1242:   Virtual-Dot11Radio1: Virtual-Dot11Radio1(state is
OPEN)
*May 03 13:19:26.714: 1242:       Node 001c.b107.fa2f

Here you can see that Ethernet bridging is enabled for bridge group 1 and 2. The FastEthernet port 0 is in bridge group 2. With the preceding output of show mesh forwarding vlan, you know that bridge group 2 is forwarding VLAN 12.

Summary

Wireless mesh deployments are becoming more and more popular as colleges, towns, police and fire departments, and cities strive to provide wireless access. When installing a mesh network, it is important to remember to prime the APs on the wired network because this small step lets you know if the APs are functioning properly before you install them 40 or 50 feet off the ground on a light pole or the side of a building. You need to know what VLANs need to be trunked across the mesh network if you are going to enable Ethernet bridging. You also need to make sure all the APs in the mesh path are configured to support those VLANs. You should conduct a proper site survey to ensure that your AP placement allows for good SNR links between the APs and that the line of sight and Fresnel zones are clear. Depending on the physical distance between your APs, you might have to use laser devices to properly align the antennas. Keeping these factors in mind will help you deploy a fast and stable wireless mesh network.