Troubleshooting Tips for NAT Related Issues in Cisco ASA

Here are some tips I found while troubleshooting ASA NAT issues. 

Verifying Configuration

First thing is that the Auto-NAT is not showing in the show run output of Cisco ASA under object network configuration which is used to configure it.

As an example, I did the following Static Auto NAT config;

Well, it should be there in the running config, but not where I configured..

It is there in the running config, but in a different place, Pipe nat or scroll all the config, you will find it.

View NAT Translation Information

There are few commands to see NAT translations in Cisco ASA.

show nat command will only be a summary, better command will be show nat detail, well for me show nat is a useless command as show nat detail covers it all.

Untranslate hits means in the above example, the traffic is coming from OUTSIDE. Translate hits means coming from DMZ.

Dynamic NAT & PAT Pool Related Issues

show nat pool command can be used to tackle PAT issues like PAT pool exhaustion issues.

Allocated 1 means, one port from 1024-65535 pool is used.

Another useful command is show xlate, specially for verifying PAT translations.

You can bind show xlate command with a local address or a global address for more granular output.

NAT Divert Issues.

NAT is used to figure out the Egress interface for a traffic flow before even the route lookups/PBRs.

So if the interface pair is misconfigured, the entire flow will be forwarded to a wrong interface dropping the traffic.

object network WEB

   host 192.168.10.10

   nat (dmz,outside) source static 203.115.50.10

Above configuration tells the ASA that, when 192.168.10.10 tries to move from DMZ to OUTSIDE, it should be translated to 203.115.50.10 and when something hits 203.115.50.10 on OUTSIDE interface, it should be untranslated back to 192.168.10.10 and forward to DMZ interface. 

If the DMZ interface is misconfigured here, the traffic is dropped.

show nat divert-table can be useful to find such issues.

Troubleshooting Packet Drops in Cisco ASA for Non-VPN Traffic

This packet flow is something I created based on the labs I did to understand how ASA works. This will help you to troubleshoot packet drops in ASA for non VPN traffic.

RX Ring Buffer Overrun

RX Ring is the software input queue which holds the packets for processing. If the RX Ring is filled, the traffic will be dropped before even processing. This can be seen by show interface <interface-name> command like the following.

Overrun counter will indicate how many packets were dropped due to RX Ring exhaustion. Here the "hardware (0/0)" is just because I use a ASAv for lab, but for real hardware it will show some value.

Existing Connection Check

Connection table can be viewed from show conn command.

Above snap shows an existing telnet connection originated from INSIDE interface source 192.168.10.10. If a packet which belongs to same traffic comes to the INSIDE interface, ASA will forward it to directly to NAT Translations.

If the packet is a TCP SYN or a UDP, it will be forwarded to NAT Untranslations and move on.

If there is no existing connection and if the packet is a non-TCP SYN, it will be dropped with a syslog similar to the following:

ASA-6-106015: Deny TCP (no connection) from 192.168.10.10 /32256 to 203.115.50.10 /23 flags PSH ACK on interface INSIDE

NAT Untranslation 

Packet it untranslated before ACL check; this is the reason we put ACLs to your private addresses instead of your public addresses.

The packet drop can occur due to the destination address translations and diverting to the wrong interface.

ACL Check

First packet in flow is checked against ACLs. This is why you can see fewer hits in show access-list command.

Denied packets are dropped and logged like the following:

%ASA-4-106023: Deny tcp src OUTSIDE:203.115.50.10/14379 dst INSIDE:192.168.10.10/24 by access-group "OUTLIST"

Allowed packets will be logged like the following:

%ASA-7-609001: Built local-host OUTSIDE:203.115.50.10

%ASA-7-609001: Built local-host INSIDE:192.168.10.10

%ASA-6-302013: Built inbound TCP connection 87 for OUTSIDE:203.115.50.10/18653 (203.115.50.10/18653) to INSIDE:192.168.10.10/23 (192.168.10.10/23)

NAT Translation

This is the place to run the NAT translation rules. This is done before routing so that the route lookups can be performed for translated addresses.

I don't see any reason to the traffic to be dropped here. Cisco ASA software older than version 8.2 needed the NAT to be performed mandatorily but now it is not the case. So for older ASAs, there could have been traffic drops here, but not now. 

L3 Route Lookup

Egress interface is determined 1st by NAT rules or existing connection entries. If couldn't find a macth, global routing table will be consulted.

If the L3 route lookup failed, you will get a syslog similar to following.

%ASA-6-110003: Routing failed to locate next hop for TCP from INSIDE 192.168.10.10/18653 to OUTSIDE:203.115.50.20/23

L2 Address Lookup

If layer 2 lookup fails, you will not see a syslog. show arp will not show required entry and debug arp will indicate if we are not receiving ARP reply. 

TX Ring Buffer Underrun

Same as the Buffer Overrun, if the output queue if full, the buffer underrun counter will increase and packets will be dropped.

Notes on Wireshark Filtering Commands

Packet analysis is a crucial skill for application level troubleshooting. Primary tool is Wireshark and following are some commands which will often come in handy to filter the interested traffic.

Filtering Conditions

These are the filtering criteria to be matched and displayed. You just need to type the line in filter tool bar. 

Logical Operators

You can just type the name of the operator in English or you can use the logical symbols like you do in programming to make a logical filter which will display the logical truth result of 2 or more conditions.

Using above commands, conditions & operators you can write complex statements like the following to filter packets in a capture.

Ex:- !(ip.flags == 0x40) && ip.addr==10.227.54.142

Additional Notes:-

If you want to filter packets by a protocol name, you just need to type the protocol name

Ex:- 'dns' will display all the DNS packets

Ex:- 'http' will display all the http packets

If you want to find TCP packets with ACK flag set, you should use 'tcp.flags.ack == 1' 

If you want the TCP packets with acknoledgement number equal to 1 'tcp.ack == 1'

Use 'frames.len > 1514' will display the jumbo frames.

Checking Open Ports (Telnetting, TCPING & Scanners)

You can use various methods to check open ports of a server. Old school Telnet is the command prompt tool you can use easily for a 1 targeted host.

Telnet

Let's say I need to know whether https (TCP port 443) is available at facebook.com

The command would be;

telnet facebook.com 443

If it is open, it will give an output telling connecting... and then the screen will be totally black..

To quit from  the connection, you should hit Ctrl + ] keys and type quit.

Now let's try the same thing from Cisco CLI;

To quit from the session, you should hit Ctrl + Shift + 6 and release and enter X

It will go back to the Cisco CLI again but will not really disconnect the session, If you hit Enter it will continue the session again. So you need to type disconnect and hit enter..

TCPING

TCPING is a small Windows command prompt tool I use which does basically the same thing in a different way. You can download it from their official site

https://www.elifulkerson.com/projects/tcping.php

I have unzipped the folder and pasted the contents into the System32 folder.

It is simple; tcping facebook.com 443 command will show the replies if open.

Scanners

If you need to scan an entire network/subnet for open ports, you should use a network scanner. There are plenty of them and they are very easy to use. One I use in Windows is Advanced IP Scanner..

Allow Pings (ICMP) & Traceroutes through Cisco ASA Firewalls

ASA Firewalls does not allow ICMP traffic to pass through it's interfaces by default. For real scenarios it is better that way in terms of security concerns. But for lab purposes and to verify implementations you will need it to be allowed from Firewall.

Why ICMP is blocked by ASA?

Short answer is because it is not in the list of state full inspection protocols. 

You can see the default inspection protocols list on the capture.

So you may need to configure access-control rules for both source and destination interfaces but it will bypass the firewall functionality. Firewall should remember the legitimate ICMP traffic and allow only the return traffic to pass through.

So you will need to add ICMP to the default inspection policy in global policy which is under the service policies.

Doing it in CLI is simple;

policy-map global_policy

 class inspection_default

  inspect icmp

  inspect icmp error

Make ASA Visible in Traceroutes

By default ASA is not visible in traceroutes as it does not decrement the TTL. To make it visible in a traceroute, we will need to add the following configuration to the default class in global policy.

policy-map global_policy

 class class-default

  set connection decrement-ttl

Troubleshooting Multicast RPF Failure in Control Plane

I have looked into a RPF (Reverse Path Forwarding) issue in data plane here. In this post let's look into an issue where the control plane may have an impact of RPF. This is basically the RP (Rendezvous Point) is learnt from a different interface on unicast routing table while the multicast feed is coming from another interface. Of course this example is in PIM Sparse Mode and I am configuring the RP manually.

OSPF is configured for all links in routers but PIM is not enabled on E0/2 of R1 and R3.
RP is configured to be the R1.

Let's join PC to the group 239.1.1.1

PC(config)#int e0/0
PC(config-if)#ip igmp join-group 239.1.1.1

Looks like the feed does not get replies..
Let's look at the multicast routing table of R3;

Looks like RPF issue is there to the RP because RPF neighbor seems to be 0.0.0.0 and incoming interface as Null.

So the issue is clear now. Unicast routing table shows 1.1.1.1 can be reached via 192.168.13.1 (E0/2) but the multicast packets can traverse through E0/0 only.

To fix this with a multicast static route, let's find out what can be the interface,

It should be via 192.168.23.2 (E0/0) so the command should be;

R3(config)#ip mroute 1.1.1.1 255.255.255.255 192.168.23.2

The RPF issue is resolved.

As soon as I hit this command, feed got the replies..

Now what can be the routing table looks like?

Now there is a new entry for (192.168.1.10, 239.1.1.1) group which is the Source Path Tree (SPT). But looks like the traffic is not using it because the incoming interface is Null and RPF neighbor is 0.0.0.0 So the traffic is using the old entry which is (*, 239.1.1.1) which I guess is the Root Path Tree (RPT).

Why can it be like this?

Because surely there must be a RPF issue in Data Plane with the source 192.168.1.10

This can also be solved by entering another static mroute for 192.168.1.10

Troubleshooting Multicast RPF Failure in Data Plane

Loop prevention mechanism of multicast is to check the source IP with it's unicast routing table and do a RPF check (Reverse Path Forwarding). If the source is learned from the interface which the feed is received, the traffic is allowed to go through the router.

Let's take an example.

OSPF is configured for all links in routers but PIM is not enabled on E0/2 of R1 and R3.
What will happen is 192.168.1.0/24 will be learnt by R3 via OSPF from R1 but the multicast traffic will come from R2..

Let's join PC to the group 239.1.1.1

PC(config)#int e0/0
PC(config-if)#ip igmp join-group 239.1.1.1

Well the pings are failing..
Let's look at the multicast routing table,

It shows the incoming interface as Null and RPF neighbor as 0.0.0.0

Let's look at the unicast routing table for the 192.168.1.10 source.

Well there was a good debug command debug ip mpacket which should be used along with no ip mroute-cache interface level command to see rpf issues in multicast routing. Unfortunately Cisco has removed it from IOS version 15.0

But you can still check for RPF for the source using the following command.
R3#show ip rpf 192.168.1.10

Let's fix this issue by using a static mroute. First have to find the next-hop using following command.
R3#show ip pim neighbor

Now the command to route is;
R3(config)#ip mroute 192.168.1.10 255.255.255.255 192.168.23.2

As soon as the route entered, IPTV_Svr has replies,

Now let's see the routing tabe of R3;

Show Command Outputs of Working MPLS VPN Configuration with PE-CE BGP (With a Route Reflector in the SP-CORE)

Following post will help in troubleshooting MPLS VPN configurations when a route reflector is in the SP-CORE network. These are the widely using show command outputs which you can compare and decide which has configured incorrectly.

To learn theory related to this configuration, please go through the following posts.

IPv4, IPv4 VRF & VPNv4 Address Families of BGP
Configuring MPLS L3 VPN with PE-CE BGP

For a same kind of post where there is no route reflector in the SP-CORE, please go through the following post.

Show Command Outputs of Working MPLS VPN Configuration with PE-CE BGP

Following is the diagram used in the post.

P-CORE will be the route-reflector. E0/0 interfaces of PE1 and PE2 are configured with sub interfaces and they are assigned to VRFs per customer. CE means customer edges and Green color areas are belong to customer 1 and the purple areas are belong to customer 2. SP CORE runs OSPF and MPLS and PS means Provider-Edge Switches. Loopbacks are configured like shown in the diagram and following are the configuration for each devices. Since this post is about to understand the show command outputs, configuration will not be explained, placing here only for the reference..

Configuration of the Custer Edge Routers..

Configuration of the Provider Edge Routers..

Configuration of the Provider Edge Switches..

Configuration of the Provider Core Router (The Route Reflector)..

Now let's go through the show commands..

show ip route
show ip route vrf <VRF>

show bgp summay
show ip bgp summay
show bgp ipv4 unicast summary
show ip bgp ipv4 unicast summary

All the above commands will give the same result and only the Customer Edge routers will have something to show for these commands..

show bgp vpnv4 unicast all summary

show bgp vpnv4 unicast all
show ip bgp vpnv4 all summary
show ip bgp vpnv4 all

show bgp vpnv4 unicast vrf <VRF> summary
show bgp vpnv4 unicast vrf <VRF>
show ip bgp vpnv4 vrf <VRF> summary
show ip bgp vpnv4 vrf <VRF>

These commands will be show outputs on Provider Edge routers where VRFs and VPNs are configured.. Provider Core will respond for vpnv4 all commands as it is the route reflector for VPNv4 address family.

Note:-
Note that the 1st 4 commands will result the both IPv4 VRF and VPNv4 family results while the last 4 commands will only show the IPv4 VRF family results. So there is no command to view only the VPNv4 family results..

show mpls forwarding-table

This is for Provider Edge and Provider Core routers.

Traceroutes and Forwarding

Now let's see what a trace route will show. Let's initiate one from a Customer Edge Router to its other side Customer Edge router.

However a traceroute from a Provider Edge to a Customer Edge router of the other side will not work because there is no return route advertised to that customer router..

But if you look at the mpls labeling for that same destination from the same Provider Edge router, you will see there are are 2 labels because it goes through the MPLS VPN.

16 will be the outer label and 20 will be the inner label in this case. What PE1 does is that it will add 20 and 16 on top of that and forward it to the 3.3.3.3 which should go through the 2.2.2.2 as per the topology (look at the mpls forwarding tables above). When 2.2.2.2 receives it, it will remove the label 16 and it will forward it to 3.3.3.3 and when 3.3.3.3 receives it, it will remove the label 20 and route as per the routing table.

Provider Core  Routers will not maintain any Customer Edge route as it is in the middle of the VPN..