Cisco ISP Software and Router Management

The importance of the loopback interface should never be overlooked, especially for general operations and management of the router. Indeed, it is surprising how few ISPs make use of this time-saving resource. The chapter continues with a discussion on how to configure router interfaces and check their status.

Following the discussion of basic management configuration is an introduction to the CEF and NetFlow capabilities that ISPs should be using on their routers. The chapter finishes with a brief look at Nagle before discussing the importance of the DNS in an ISP's operation.

Many of the features discussed here are described in the context of the ISP software covered in Chapter 1, "Software and Router Management."

IOS Software and Loopback Interfaces

The use of the loopback interface is mentioned in many instances throughout this book. Although this is not a feature unique to IOS Software, there are many and considerable advantages in making full use of the capability that the loopback interface allows. This section brings together all the occasions where the loopback interface is mentioned throughout the book and describes how they can be useful to the ISP network engineer.

Motivation for Using the Loopback Interface

ISPs endeavor to minimize the unnecessary overhead present in their networks. This unnecessary overhead can be the number of networks carried in the IGP, the number of skilled engineering staff to operate the network, or even network security. The utilization of one feature, the loopback interface on the router, goes a long way to help with each of the three scenarios mentioned here.

Control of the size of the IGP is attended to by summarization of point-to-point addresses at PoP or regional boundaries, the use of IP unnumbered on static WAN interfaces, and a carefully designed network addressing plan. ISP network security is of paramount importance, and any techniques that make the management simpler are usually welcomed. For example, when routers access core servers, ISPs apply filters or access lists to these servers so that the risk of compromise from the outside is reduced. The loopback interface is helpful here as well.

It is very common to assign all the IP addresses used for loopback interfaces from one address block. For example, an ISP with around 200 routers in a network might assign a /24 network (253 usable addresses) for addressing the loopback interface on each router. If this is done, all dependent systems can be configured to permit this address range to access the particular function concerned, whether it is security, unnumbered WAN links, or the iBGP mesh. Some examples of the use of the loopback interface in the ISP environment follow in the rest of this section.

BGP Update Source

In the following example, the iBGP mesh is built using the loopback interface on each router. The loopback doesn't ever disappear, which results in a more stable iBGP, even if the underlying physical connectivity is less than reliable.

hostname gateway1
!
interface loopback 0

ip address 215.17.1.34 255.255.255.255
!
router bgp 200
 neighbor 215.17.1.35 remote-as 200
 neighbor 215.17.1.35  update-source loopback 0

neighbor 215.17.1.36 remote-as 200
 neighbor 215.17.1.36   update-source loopback 0

!

Router ID

If a loopback interface is configured on the router, its IP address is used as the router ID. This is important for ensuring stability and predictability in the operation of the ISP's network.

OSPF chooses the designated router (DR) on a LAN as the device that has the highest IP address. If routers are added or removed from the LAN, or if a router gains an interface with a higher address than that of the existing DR, the DR likely will change if the DR or backup designated router (BDR) fails. This generally is undesirable in an ISP network because ISPs prefer to have the DR and BDR routers established deterministically. This change in DR and BDR can be avoided by ensuring that the loopback interface is configured and in use on all routers on the LAN.

The loopback interface is used for the BGP router ID. If the loopback isn't configured, BGP uses the highest IP address on the router. Again, because of the ever-changing nature of an ISP network, this value can change, possibly resulting in operational confusion. Configuring and using a loopback interface ensures stability.

Exception Dumps by FTP

Cisco routers can be configured to dump core memory to an FTP server as part of the diagnostic and debugging process. However, this core dump should be to a system not running a public FTP server, but one heavily protected by filters (TCP Wrapper even) that allow only the routers access. If the loopback interface address is used as source address from the router and is part of one address block, the filter is very easy to configure. A 200-router network with 200 disparate IP addresses makes for a very large filter list on the FTP server. Examine the following example IOS Software configuration:

ip ftp  source-interface Loopback0


ip ftp username cisco
ip ftp password 7 045802150C2E
exception protocol ftp
exception dump 169.223.32.1

TFTP Server Access

TFTP is the most common tool for uploading and downloading configurations. The TFTP server's security is critical, which means that you should always use security tools with IP source addresses. IOS Software allows TFTP to be configured to use specific IP interfaces address. This allows a fixed ACL on the TFTP server based on a fixed address on the router (for example, the loopback interface).

ip tftp source-interface Loopback0

SNMP Server Access

If SNMP is used in the network, the loopback interface again can be brought into use for security access issues. If SNMP traffic from the router is sourced from its loopback interface, it is easy to protect the SNMP management station in the NOC. A sample IOS Software configuration follows:

access-list 98 permit 215.17.34.1
access-list 98 permit 215.17.1.1
access-list 98 deny  any
!
snmp-server community 5nmc02m RO 98
snmp-server  trap-source Loopback0


snmp-server trap-authentication
snmp-server host 215.17.34.1 5nmc02m 
snmp-server host 215.17.1.1 5nmc02m

TACACS/RADIUS Server Source Interface

Most ISPs use TACACS+ or RADIUS for user authentication. Very few define accounts on the router itself because this offers more opportunity for the system to be compromised. A well-protected TACACS+ server accessed only from the router's loopback interface address block offers more security of user and enable accounts. A sample configuration for standard and enable passwords follows:

aaa new-model
aaa authentication login default tacacs+ enable
aaa authentication enable default tacacs+ enable
aaa accounting exec start-stop tacacs+
!
ip tacacs  source-interface Loopback0

tacacs-server host 215.17.1.2
tacacs-server host 215.17.34.10
tacacs-server key CKr3t#
!

When using RADIUS, either for user administrative access to the router or for dial user authentication and accounting, the router configuration to support loopback interfaces as the source address for RADIUS packets originating from the router looks like this:

radius-server host 215.17.1.2 auth-port 1645 acct-port 1646
radius-server host 215.17.34.10 auth-port 1645 acct-port 1646
ip radius source-interface Loopback0

!

NetFlow Flow Export

Exporting traffic that flows from the router to a NetFlow Collector for traffic analysis or billing purposes is quite common. Using the loopback interface as the source address for all exported traffic flows from the router allows for more precise and less costly filtering at or near the server. A configuration example follows:

ip flow-export destination 215.17.13.1 9996
ip flow-export  source Loopback0

ip flow-export version 5 origin-as
!
interface Fddi0/0/0
 description FDDI link to IXP
 ip address 215.18.1.10 255.255.255.0
 ip route-cache flow
 ip route-cache distributed
 no keepalive
!

Interface FDDI0/0/0 has been configured to capture flow records. The router has been configured to export Version 5–style flow records to the host at IP address 215.17.13.1 on UDP port 9996, with the source address being the router's loopback interface.

NTP Source Interface

NTP is the means of keeping the clocks on all the routers on the network synchronized to within a few milliseconds. If the loopback interface is used as the source interface between NTP speakers, it makes filtering and authentication somewhat easier to maintain. Most ISPs want to permit their customers to synchronize only with their time servers, not everyone else in the world. Look at the following configuration example:

clock timezone SST 8
!
access-list 5 permit 192.36.143.150
access-list 5 permit 169.223.50.14
!
ntp authentication-key 1234 md5 104D000A0618 7
ntp authenticate
ntp trusted-key 1234
ntp  source Loopback0

ntp access-group peer 5
ntp update-calendar
ntp peer 192.36.143.150
ntp peer 169.223.50.14
!

Syslog Source Interface

Syslog servers also require careful protection on ISP backbones. Most ISPs prefer to see only their own systems' syslog messages, not anything from the outside world. Denial-of-service attacks on syslog devices are not unknown, either. Protecting the syslog server is again made easier if the known source of syslog messages comes from a well-defined set of address space—for example, that used by the loopback interfaces on the routers. See the following configuration example:

logging buffered 16384
logging trap debugging
logging  source-interface Loopback0

logging facility local7
logging 169.223.32.1
!

Telnet to the Router

This might seem to be an odd example in a document dedicated to IOS Software essentials. However, remember that a loopback interface on a router never changes its state and rarely has any need to change its IP address. Physical interfaces can be physically swapped out or renumbered, and address ranges can change, but the loopback interface will always be there. So, if the DNS is set up so that the router name maps to the loopback interface address, there is one less change to worry about during operational and configuration changes elsewhere in the ISP backbone. ISP backbones are continuously developing entities. Here's an example from the DNS forward and reverse zone files:

; net.galaxy zone file
net.galaxy.   IN   SOA   ns.net.galaxy. hostmaster.net.galaxy. (
                1998072901 ; version == date(YYYYMMDD)+serial
                10800   ; Refresh (3 hours)
                900    ; Retry (15 minutes)
                172800   ; Expire (48 hours)
                43200 )  ; Minimum (12 hours)
        IN   NS   ns0.net.galaxy.
        IN   NS   ns1.net.galaxy.
        IN   MX   10 mail0.net.galaxy.
        IN   MX   20 mail1.net.galaxy.
;
localhost    IN   A    127.0.0.1
gateway1    IN   A    215.17.1.1
gateway2    IN   A    215.17.1.2
gateway3    IN   A    215.17.1.3
;
;etc etc
; 1.17.215.in-addr.arpa zone file
;
1.17.215.in-addr.arpa. IN   SOA   ns.net.galaxy. hostmaster.net.galaxy. (
                1998072901 ; version == date(YYYYMMDD)+serial
                10800   ; Refresh (3 hours)
                900    ; Retry (15 minutes)
                172800   ; Expire (48 hours)
                43200 )  ; Minimum (12 hours)
IN   NS   ns0.net.galaxy.
        IN   NS   ns1.net.galaxy.
1        IN   PTR   gateway1.net.galaxy.
2        IN   PTR   gateway2.net.galaxy.
3        IN   PTR   gateway3.net.galaxy.
;
;etc etc

On the router, set the Telnet source to the loopback interface:

ip telnet  source-interface Loopback0

RCMD to the Router

RCMD requires the operator to have the UNIX rlogin/rsh clients to enable access to the router. Some ISPs use RCMD for grabbing interface statistics, uploading or downloading router configurations, or taking a snapshot of the routing table. The router can be configured so that RCMD connections use the loopback interface as the source address of all packets leaving the router:

ip rcmd  source-interface Loopback0

Interface Configuration

Configuring interfaces involves more than simply plugging in the cable and activating the interface with the IOS Software command no shutdown. Attention should be applied to details such as whether it is a WAN or a LAN, whether a routing protocol is running across the interface, addressing and masks to be used, and operator information.

description

Use the description interface command to document details such as the circuit bandwidth, the customer name, the database entry mnemonic, the circuit number that the circuit supplier gave you, and the cable number. This sounds like overkill, especially if there is a customer database within the ISP organization. However, it is very easy to pick up all the relevant details from the router show interface command if and when an engineer needs to be onsite, when an engineer is away from the database system, or when the database is unavailable. There can never be too little documentation, and documentation such as this ensures that reconstructing configurations and diagnosing problems are made considerably easier.

bandwidth

Don't forget the bandwidth interface command. It is used by interior routing protocols to decide optimum routing, and it is especially important to set this command properly in the case of backbone links using only a portion of the available bandwidth support by the interface. For example, a serial interface (Serial0/0) on a router supports speeds up to 4 Mbps but has a default bandwidth setting of 1.5 Mbps. If the backbone has different size links from 64 Kbps to 4 Mbps and the bandwidth command is not used, the interior routing protocol will assume that all the links have the same cost and will calculate optimum paths accordingly—and this could be less than ideal.

On customer links, it might seem that this setting is superfluous because an interior routing protocol is never run over a link to a customer. However, it provides very useful online documentation for what the circuit bandwidth is. Furthermore, the bandwidth on the circuit is used to calculate the interface load variable—some ISPs monitor their customer interfaces loading by SNMP polls so that they can get advance warning of problems or congestion, or to proactively inform customers of necessary upgrades. (Some ISPs look at the load variable; other ISPs look at the five-minute average, inbound and outbound. If you monitor the load variable, you need to set the bandwidth so that it matches the true circuit bandwidth, not the default configured on the router.)

ip unnumbered

Traditionally ISPs have used IP addresses for the point-to-point links on leased-line circuits to customers. Indeed, several years ago, before the advent of CIDR, it was not uncommon to see a /26 or even a /24 used for simple point-to-point link addresses. With the advent of CIDR, /30 networks have been used instead (/30 is a block of four addresses, two of which can be used for physical interfaces). However, this led to problems because IGPs of some of the larger ISPs were starting to carry several thousand networks, affecting convergence time and resulting in an administrative and documentation nightmare.

To avoid problems with large numbers of /30s floating around the ISP's internal routing protocol, and to avoid the problems of keeping internal documentation consistent with network deployment (especially true in larger ISPs), many are now using unnumbered point-to-point links.

An unnumbered point-to-point link is one requiring no IP addresses. The configuration is such that traffic destined for one network from another simply is pointed at the serial interface concerned. ip unnumbered is an essential feature applicable to point-to-point interfaces such as Serial, HSSI, POS, and so on. It enables the use of a fixed link (usually from ISP to customer) without consuming the usual /30 of address space, thereby keeping the number of networks routed by the IGP low. The ip unnumbered directive specifies that the point-to-point link should use an address of another interface on the router, typically a LAN or more usually a loopback interface. Any networks that must be routed to the customer are pointed at the serial interface rather than the remote address of the point-to-point link, as would be done in normal instances.

Caveats

ISPs need to consider some situations before implementing an IP unnumbered system for their customer point-to-point connections. These are considerations only—bear in mind that many ISPs have used IP unnumbered for several years, mainly so that they can control the size of the IGP running in their backbone network.

• Pinging the customer—Many ISPs use monitoring systems that use ping to check the status of the leased line (customer connectivity). Even if the customer unplugs the LAN, an alarm will not be raised on the ISPs management system. This is because the customer router still knows that the LAN IP address is configured on the system and is "useable." As long as the IP address is configured on the LAN, there will be no reachability issues with using ip unnumbered.

• Routing protocols—If a routing protocol needs to be run over this link, it is operationally much easier to use IP addresses. Don't use ip unnumbered if the customer is peering with you using BGP across the link or if the link is an internal backbone link. Simply use a network with a /30 address mask. (Routing will work over unnumbered links, but the extra management and operational complexity probably outweighs the small address space advantage gained.)

• Loopback interfaces on the customer's router—These offer no advantage to addressing the ping problem, and they unnecessarily consume address space (not to mention adding complexity to the customer router configuration).

ip unnumbered Configuration Example

Using the preceding configuration commands, a typical configuration on the ISP's router would be as follows:

interface loopback 0
 description Loopback interface on Gateway Router 2
 ip address 215.17.3.1 255.255.255.255
 no ip redirects
 no ip directed-broadcast
 no ip proxy-arp
!
interface Serial 5/0
 description 128K HDLC link to Galaxy Publications Ltd [galpub1] WT50314E R5-0
 bandwidth 128
 ip unnumbered loopback 0
 no ip redirects
 no ip directed-broadcast
 no ip proxy-arp
!
ip route 215.34.10.0 255.255.252.0 Serial 5/0

The customer router configuration would look something like this:

interface Ethernet 0
 description Galaxy Publications LAN
 ip address 215.34.10.1 255.255.252.0
 no ip redirects
 no ip directed-broadcast
 no ip proxy-arp
!
interface Serial 0
 description 128K HDLC link to Galaxy Internet Inc WT50314E C0
 bandwidth 128
 ip unnumbered ethernet 0
 no ip redirects
 no ip directed-broadcast
 no ip proxy-arp
!
ip route 0.0.0.0 0.0.0.0 Serial 0

In this example, the regional or local registry has allocated the customer the network block 215.34.10.0/22. This is routed to the customer site with the static route pointing to Serial 5/0. The customer router simply needs a default route pointing to its serial interface to ensure a connection.

With this configuration, there are no /30s from point-to-point links present in the IGP, and the ISP does not need to document the link address or keep a table/database up-to-date. It all makes for easier configuration as well as easier operation of the ISP's business.

Note the contents of the description field. This example has included the following:

All of these are online documentation, seemingly superfluous, but very necessary to ensure smooth and efficient operations. All the information pertinent to the customer's connection from the cabling to the IP values is contained in the interface configuration. If the ISP's database is down or unavailable, any debug information required by operators or engineers can be found on the router itself.

Interface Status Checking

Some useful hidden IOS Software commands enable the operator to check the status of the interfaces in IOS Software. Three useful commands are show interface switching, show interface stats, and show idb.

show interface switching

The IOS Software command show interface switching provides useful information about the switching status of the router's interfaces, either on an individual interface basis or over the whole router. The full command format is show interface [int n/n] switching, where an optional argument is the specific interface in question. Command completion cannot be used for switching—it needs to be typed in up to and including the second i. Sample output might look like the following:

This sample output shows SPD¹ activity, as well as other activity on that particular interface on the router. Note the references to autonomous/SSE switching—this is applicable only to the Cisco 7000 series with Silicon Switch Engine only (a product that is now discontinued but was a significant part of the Internet core in the mid-1990s). Fast switching refers to all packets that have not been process-switched, which would include Optimum switching, NetFlow, and CEF.

show interface stats

The IOS Software command show interface stats is the second useful command to show interface status. It shows the number of packets and characters inbound and outbound on an individual router interface or all of them. The full command format is show interface [int n/n] stats, where an optional argument is the specific interface in question. Command completion cannot be used for stats—at least st needs to be typed in at the command prompt. Sample output might look like this:

As for interface switching, the output differentiates between packets that go via the processor and those that have been processed via the route cache. This is useful to determine the level of process switching taking place on the router.

On a router that supports distributed switching (for example, 7500 with VIP interfaces), the output will look like the following:

Notice that packets that have been processed via the distributed cache are counted separately from those handled via the central route cache and the processor.

show IDB

Each interface on the router has an associated interface descriptor block allocated to it. In the early days, each physical interface mapped to one IDB, and routers generally could support up to 300 IDBs (for example, the Cisco AGS+).

However, with the increasing numbers of new connection services, and with ATM and Frame Relay providing large numbers of subinterfaces, routers have had to scale to supporting several thousand IDBs. show IDB recently has become a visible command in IOS Software (CSCds89322); it allows ISPs to find out how many IDBs are configured on the router:

To find out how many IDBs are supported on different router platforms, consult Cisco.com documentation—for example, http://www.cisco.com/warp/public/63/idb_limit.html. Although most smaller router platforms still support only 300 IDBs at maximum, some of the larger platforms can go as high as 10,000 (7200/12.2T release). These values might change as future enhancements are made to Cisco IOS Software.

Cisco Express Forwarding

CEF is now the recommended forwarding/switching path for Cisco routers in an ISP environment. CEF adds increased performance, scalability, and resilience, and enables new functionality over the older optimum switching. Details on the operation and functionality of CEF are now covered in detail by the IOS Software documentation and in several whitepapers describing CEF (see references in the "Technical Reference and Recommended Reading" section at the end of this book).

Implementation is simple with either of the following commands (depending on the platform):

The key issue for ISPs is ensuring that CEF is turned on. On most platforms, CEF is not turned on by default, so ISP engineers need to ensure that CEF is turned on. Table 2-1 provides a list of the default CEF configurations for various Cisco platforms. ISPs should check their configuration scripts to ensure that CEF/dCEF is turned on, especially for the 7200-based edge platforms such as uBR, 6400, and 5800 NAS.

Table 2-1 Default Configuration for CEF on Various Platforms

CEF will be discussed in more depth in Chapter 4, "Security." One of the best security tools available for an ISP is Unicast Reverse Path Forwarding. This requires CEF to be activated on the router because the reverse path check is dependent on the FIB table, which is part of the CEF process.

NetFlow

Enabling NetFlow on routers provides network administrators with access to packet flow information from their network. Exported NetFlow data can be used for a variety of purposes, including security monitoring, network management, capacity planning (as in Figure 2-1), customer billing, and Internet traffic flow analysis.

Figure 2-1 Netflow in Its Capacity-Planning Role

NetFlow is available on all router platforms from the 2600 series upward from the 12.0 software release onward. It was first introduced in 11.1CC on the 7200 and 7500 platforms. It can be enabled on a per-interface basis on the routers, as in the following example:

interface serial 5/0

 ip route-cache flow

!

If CEF is not configured on the router, this turns off the existing switching path on the router and enables NetFlow switching (basically modified optimum switching). If CEF is configured on the router, NetFlow simply becomes a "flow information gatherer" and feature accelerator—CEF remains operational as the underlying switching process.

NetFlow Feature Acceleration

NetFlow feature acceleration works for a limited set of features that can take advantage of flow process short cuts. NFFA reserves space in the flow cache for state information belonging to features converted to use the flow acceleration. The features can then attach per-flow state to the cache entry, using NetFlow as a quick way to access information that is flow-based. For example, NetFlow policy routing (NPR) uses flow acceleration to eliminate route-map checks on a per-packet basis. NetFlow feature acceleration is turned on with the following command:

ip flow-cache feature-accelerate

As of 12.0(11)S, the following features have been converted to work with NetFlow feature acceleration:

• Numbered access lists
• Named access lists
• IP accounting
• Crypto decrypt
• Crypto encrypt
• Policy routing
• WCCP redirection

NetFlow Statistics—Basics

To view NetFlow information on the router, simply enter the command show ip cache flow. This displays the current flow cache on the terminal screen (see Example 2-1).

Example 2-1 Sample Output from Displaying Flow Information on a NetFlow-Enabled Router

gw>sh ip cache flow
IP packet size distribution (410772243 total packets):
  1-32  64  96 128 160 192 224 256 288 320 352 384 416 448 480
  .000 .168 .384 .102 .160 .107 .019 .005 .003 .001 .001 .000 .000 .000
.003
 
  512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
  .001 .000 .035 .000 .003 .000 .000 .000 .000 .000 .000
 
IP Flow Switching Cache, 4456704 bytes
 15074 active, 50462 inactive, 125120769 added
 369493980 ager polls, 0 flow alloc failures
 last clearing of statistics 4d05h
Protocol     Total  Flows  Packets Bytes Packets Active(Sec) Idle(Sec)
--------     Flows   /Sec    /Flow  /Pkt    /Sec     /Flow     /Flow
TCP-Telnet     605    0.0       44    52     0.0       8.1       9.1
TCP-FTP       3494    0.0       22    64     0.2       9.4      12.9
TCP-FTPD      4104    0.0      757   376     8.4      34.9       5.7
TCP-WWW     845158    2.3       16   281    39.1       4.5       6.8
TCP-SMTP     87119    0.2       10   201     2.5       4.2      13.1
TCP-X           59    0.0        2    68     0.0       0.4      12.0
TCP-BGP      62074    0.1        5   255     0.9       9.6      18.5
TCP-NNTP         5    0.0        3    48     0.0       8.8      19.6
TCP-Frag         2    0.0        2    40     0.0       0.1      21.2
TCP-other 11879955   32.3        5   141   174.2       2.5       7.5
UDP-DNS   70078211  191.0        3    90   586.3       4.8      19.1
UDP-NTP      31804    0.0        1    72     0.0       0.0      19.0
UDP-TFTP       327    0.0        3   153     0.0       4.8      19.2
UDP-Frag         9    0.0        4   311     0.0      22.5      18.2
UDP-other 41601240  113.4        2   157   301.3       4.1      19.1
ICMP        498404    1.3        4   170     5.7      10.7      19.0
IGMP             2    0.0      113   551     0.0       6.8      19.8
IP-other     20236    0.0        4   299     0.2      12.7      18.7
Total:   125112808  341.1        3   126  1119.2       4.4      17.9
 
SrcIf     SrcIPaddress    DstIf     DstIPaddress    Pr SrcP DstP  Pkts
Se2/0     207.69.200.110  Fa1/0     203.37.255.121  11 2245 0035     1
Fa1/0     203.37.255.121  Se2/0     207.69.200.110  11 0035 2245     1
Fa1/0     203.37.255.97   Se2/0     169.229.128.130 11 0035 0C1C     1
Se2/0     169.229.128.130 Fa1/0     203.37.255.97   11 0C1C 0035     1
Se2/0     195.28.226.121  Fa1/0     203.37.255.97   11 0408 0035     1
Fa1/0     203.37.255.97   Se2/0     195.28.226.121  11 0035 0408     1
Fa1/0     203.37.255.97   Se2/0     163.21.134.2    11 0035 0035     2
Se2/0     202.103.229.40  Fa1/0     203.37.255.97   11 0A6B 0035   248
Se2/0     163.21.134.7    Fa1/0     203.37.255.97   11 0035 0035     4
Fa1/0     203.37.255.97   Se2/0     163.21.134.7    11 0035 0035     4
Fa1/0     203.37.255.97   Se2/0     202.103.229.40  11 0035 0A6B   248
Se2/0     163.21.134.2    Fa1/0     203.37.255.97   11 0035 0035     2
Se2/0     63.87.170.77    Fa1/0     203.37.255.97   11 B034 0035     2
Fa1/0     203.37.255.97   Se2/0     63.87.170.77    11 0035 B034     2

The first part of the output displays the packet size distribution of the traffic flowing into the interfaces that NetFlow is configured on. The next portion of the output displays the flows, packet size, activity, and so on for the flows per well-known protocol. The final section displays the source and destination interfaces/addresses/ports for the currently active traffic flows.

It is also possible to export this collected data to a system that will collect the data, allowing the ISP to carry out further analysis. Public-domain software is available (cflowd from Caida and NetFlowMet from the University of Auckland, for example), as well as fully featured and supported commercial products, such as Cisco's NetFlow Collector and Analyzer packages.

NetFlow Data Export

The greatest benefits of NetFlow are found when its data is exported to collection systems and then are analyzed and processed. Cisco has adopted a broad approach to facilitate this activity. These include donations for freeware collection/analysis software, Cisco's own commercial software, tools for others to create their own software, and partnerships with companies that make commercial-grade billing systems based on NetFlow export.

To export the data, the following configuration commands are required:

ip flow-export version 5 [origin-as|peer-as]
ip flow-export destination x.x.x.x udp-port
ip flow-export source Loopback0

The first command line sets the export version to 5 (basically this includes BGP information such as AS number) and has options to include origin-as or peer-as in the exported records. Most ISPs use the origin-as option because that will record the origin AS of the prefix originating the flow. This has become a frequently asked question on the CAIDA cflowd list, with ISPs forgetting the origin-as option and then not understanding why so many of their exported records have an origin of AS 0.

The second command line configures the IP address of the destination system, the NetFlow collector system, and the UDP port that the collector is listening on. Most ISPs use high UDP ports, such as 9999 or in the 60,000s. Note that because the flow records use UDP, it is important to design the infrastructure so that the flow collector is not too far away from the originating router. Some ISPs that use NetFlow for billing purposes build a separate management network simply to support this function.

The third command line originates all the flow traffic using the IP address of the loopback interface. This makes the cflowd configuration file easier to construct for several routers because most ISPs number their router loopbacks out of one contiguous block.

To determine the status of the flow export, it is possible to check on the router to see what has been sent. Obviously the collector system should be checked as well—cflowd, for example, has extensive instructions on how to debug any flow export problems. An example of the usage of the IOS Software command follows:

A new feature as of Cisco IOS Software release 12.0(5)S is NetFlow aggregation, in which summarization/aggregation of the flow records is carried out on the router before the data is exported to the collecting system. The aim is to reduce the amount of data going across the network from router to flow collector, thereby improving the reliability of the collecting system. Flow aggregation is enabled by the following commands:

Subcommands required include enabled, which switches on the flow aggregation, and export destination, which lists the host that will gather the aggregated records. The collector host needs to support NetFlow Type 8 records to be capable of reading the aggregated information.

Turn On Nagle

The Nagle congestion-control algorithm is something that many ISPs turn on to improve the performance of their Telnet sessions to and from the router. When using a standard TCP implementation to send keystrokes between machines, TCP tends to send one packet for each keystroke typed. On larger networks, many small packets use up bandwidth and contribute to congestion.

John Nagle's algorithm (RFC 896) helps alleviate the small-packet problem in TCP. In general, it works this way: The first character typed after connection establishment is sent in a single packet, but TCP holds any additional characters typed until the receiver acknowledges the previous packet. The second, larger packet is sent, and additional typed characters are saved until the acknowledgment comes back. The effect is to accumulate characters into larger chunks and pace them out to the network at a rate matching the round-trip time of the given connection. This method is usually good for all TCP-based traffic and helps when connectivity to the router is poor or congested or the router itself is busier than normal. However, do not use the service nagle command if you have XRemote users on X Window sessions or sourcing voice over IP traffic or other real-time traffic from the router—performance will become very poor.

The IOS Software command to enable Nagle follows:

service nagle

DNS and Routers

The DNS might be an unusual topic to put into a book covering ISP network essentials and Cisco IOS Software best practices. However, it is one of the most overlooked systems topics in the ISP industry—yet it is probably the most important part of the public visibility of the network to get right. If the DNS does not work, the public thinks that the network is broken—many newspaper headlines in the last few years have displayed such apocryphal headlines simply because of operational errors or problems with the DNS.

An ISP network engineer must pay attention to two aspects of the DNS. The first is the business of putting all the name-to-address-to-name mappings in the system so that routers can be recognized by their English-language names rather than by four boring decimal numbers separated by dots. Humans aren't good at remembering the latter. The second aspect is to actually enable support for the DNS in the routers themselves. This section covers only the router aspect—Chapter 5, "Operational Practices," describes configuration and placement of the DNS systems throughout the ISP backbone.

Mapping IP Addresses to Names

Mapping domain names to IP addresses and vice versa is one of those commonly overlooked areas in a new ISP's operations. Doing a trace from Australia across the backbones in the United States to a site in the United Kingdom gives you something like Example 2-3.

Example 2-2 Example Traceroute Across the Internet from Australia to the United Kingdom

traceroute to k.root-servers.net (193.0.14.129), 30 hops max, 38 byte packets
 1 fe5-0.gw.apnic.net (202.12.29.190) 0.707 ms 0.534 ms 0.497 ms
 2 Serial1-0-3.cha8.Brisbane.telstra.net (139.130.64.97) 5.999 ms 5.131 ms 
  6.155 ms
 3 GigabitEthernet5-1.cha-core4.Brisbane.telstra.net (203.50.51.1) 6.148 ms
  4.972 ms 4.537 ms
 4 Pos2-0.chw-core2.Sydney.telstra.net (203.50.6.225) 19.355 ms 18.595 ms
  19.797 ms
 5 Pos4-0.exi-core1.Melbourne.telstra.net (203.50.6.18) 32.120 ms 32.968 ms 
  32.544 ms
 6 Pos5-0.way-core4.Adelaide.telstra.net (203.50.6.162) 50.088 ms 46.171 ms 
  44.896 ms
 7 Pos6-0.wel-core3.Perth.telstra.net (203.50.6.194) 88.296 ms 75.545 ms 
  83.527 ms
 8 GigabitEthernet4-0.wel-gw1.Perth.telstra.net (203.50.113.18) 78.172 ms
  76.116 ms 75.851 ms
 9 Pos1-0.paix1.PaloAlto.net.reach.com (203.50.126.30) 305.915 ms 309.617 ms
   314.994 ms
10 fe0.pao0.verio.net (198.32.176.47) 308.744 ms 304.431 ms 304.230 ms
11 p4-6-0-0.r02.mclnva02.us.bb.verio.net (129.250.2.246) 380.061 ms 380.639 ms 
  380.292 ms
12 p16-0-0-0.r01.mclnva02.us.bb.verio.net (129.250.5.253) 384.100 ms 384.124 ms 384.382 ms
13 p4-7-2-0.r00.nycmny06.us.bb.verio.net (129.250.3.181) 390.487 ms 390.300 ms 
  396.328 ms
14 p4-0-2-0.r01.nycmny06.us.bb.verio.net (129.250.3.130) 390.196 ms 384.921 ms 
  385.245 ms
15 gxn.d3-1-0-1.r01.nycmny06.us.bb.verio.net (129.250.16.198) 321.844 ms
  319.204 ms 319.252 ms
16 se6-1-0-llb-x-ny2.NY1.core.rtr.xara.net (194.143.164.45) 325.706 ms 320.925 ms
  320.557 ms
17 se5-1-llb-ny1.HU1.core.rtr.xara.net (194.143.164.97) 325.264 ms 322.578 ms
  321.049 ms
18 po2-0-llb-hu1.TH30.core.rtr.xara.net (194.143.164.189) 389.618 ms 390.177 ms
  388.401 ms
19 gb11-0-0-llb-x-many.TH1.core.rtr.uk.xo.net (194.143.163.130) 398.421 ms 
  388.459 ms 390.471 ms
20 fa0-0.gxn-linx.transit1.linx.net (195.66.248.33) 388.834 ms 391.937 ms
  389.687 ms
21 k.root-servers.net (193.0.14.129) 387.544 ms 391.093 ms 387.059 ms

Notice that each router IP address has a corresponding DNS entry. These very descriptive DNS names help Internet users and operators understand what is happening with their connections and which route the outbound traffic is taking. The descriptive names are an invaluable aid to troubleshooting problems on the net.

Table 2-2 shows some examples of descriptive DNS formats used by various ISPs.

Table 2-2 DNS Formats

DNS Resolver in IOS Software

You can specify a default domain name that the Cisco IOS Software will use to complete domain name requests for functions such as Telnet, TFTP, and other instances of name completion (for example, ip ospf domain-lookup). You can specify either a single domain name or a list of domain names. Any IP host name that does not contain a domain name will have the domain name that you specify appended to it before being added to the host table.

ip domain-name name
ip domain-list name

It is also advisable to include a name server for the router to resolve the DNS request:

ip name-server server-address1 [[server-address2]...server-address6]

Remember that the current practice on the Internet is to quote at least two DNS resolvers. The reason is the same as for any other situation: redundancy. If one DNS server disappears, the other one can take over. When both are there, the router will look up the servers in a round-robin fashion for each request.

Conclusion

This chapter covered the important steps necessary for an ISP to set up its initial router configuration correctly. It discussed the importance of the loopback interface, how to configure interfaces, as well as how to set up CEF and NetFlow. Nagle also was covered, and the chapter concluded with a discussion on how an ISP uses DNS for the network backbone infrastructure.

Combining this chapter with Chapter 1, the ISP now should have a workable router configuration ready for configuring routing protocols and the security features necessary for any ISP backbone.