Troubleshooting Tips for NAT Related Issues in Cisco ASA

Here are some tips I found while troubleshooting ASA NAT issues. 

Verifying Configuration

First thing is that the Auto-NAT is not showing in the show run output of Cisco ASA under object network configuration which is used to configure it.

As an example, I did the following Static Auto NAT config;

Well, it should be there in the running config, but not where I configured..

It is there in the running config, but in a different place, Pipe nat or scroll all the config, you will find it.

View NAT Translation Information

There are few commands to see NAT translations in Cisco ASA.

show nat command will only be a summary, better command will be show nat detail, well for me show nat is a useless command as show nat detail covers it all.

Untranslate hits means in the above example, the traffic is coming from OUTSIDE. Translate hits means coming from DMZ.

Dynamic NAT & PAT Pool Related Issues

show nat pool command can be used to tackle PAT issues like PAT pool exhaustion issues.

Allocated 1 means, one port from 1024-65535 pool is used.

Another useful command is show xlate, specially for verifying PAT translations.

You can bind show xlate command with a local address or a global address for more granular output.

NAT Divert Issues.

NAT is used to figure out the Egress interface for a traffic flow before even the route lookups/PBRs.

So if the interface pair is misconfigured, the entire flow will be forwarded to a wrong interface dropping the traffic.

object network WEB

   host 192.168.10.10

   nat (dmz,outside) source static 203.115.50.10

Above configuration tells the ASA that, when 192.168.10.10 tries to move from DMZ to OUTSIDE, it should be translated to 203.115.50.10 and when something hits 203.115.50.10 on OUTSIDE interface, it should be untranslated back to 192.168.10.10 and forward to DMZ interface. 

If the DMZ interface is misconfigured here, the traffic is dropped.

show nat divert-table can be useful to find such issues.

Troubleshooting Packet Drops in Cisco ASA for Non-VPN Traffic

This packet flow is something I created based on the labs I did to understand how ASA works. This will help you to troubleshoot packet drops in ASA for non VPN traffic.

RX Ring Buffer Overrun

RX Ring is the software input queue which holds the packets for processing. If the RX Ring is filled, the traffic will be dropped before even processing. This can be seen by show interface <interface-name> command like the following.

Overrun counter will indicate how many packets were dropped due to RX Ring exhaustion. Here the "hardware (0/0)" is just because I use a ASAv for lab, but for real hardware it will show some value.

Existing Connection Check

Connection table can be viewed from show conn command.

Above snap shows an existing telnet connection originated from INSIDE interface source 192.168.10.10. If a packet which belongs to same traffic comes to the INSIDE interface, ASA will forward it to directly to NAT Translations.

If the packet is a TCP SYN or a UDP, it will be forwarded to NAT Untranslations and move on.

If there is no existing connection and if the packet is a non-TCP SYN, it will be dropped with a syslog similar to the following:

ASA-6-106015: Deny TCP (no connection) from 192.168.10.10 /32256 to 203.115.50.10 /23 flags PSH ACK on interface INSIDE

NAT Untranslation 

Packet it untranslated before ACL check; this is the reason we put ACLs to your private addresses instead of your public addresses.

The packet drop can occur due to the destination address translations and diverting to the wrong interface.

ACL Check

First packet in flow is checked against ACLs. This is why you can see fewer hits in show access-list command.

Denied packets are dropped and logged like the following:

%ASA-4-106023: Deny tcp src OUTSIDE:203.115.50.10/14379 dst INSIDE:192.168.10.10/24 by access-group "OUTLIST"

Allowed packets will be logged like the following:

%ASA-7-609001: Built local-host OUTSIDE:203.115.50.10

%ASA-7-609001: Built local-host INSIDE:192.168.10.10

%ASA-6-302013: Built inbound TCP connection 87 for OUTSIDE:203.115.50.10/18653 (203.115.50.10/18653) to INSIDE:192.168.10.10/23 (192.168.10.10/23)

NAT Translation

This is the place to run the NAT translation rules. This is done before routing so that the route lookups can be performed for translated addresses.

I don't see any reason to the traffic to be dropped here. Cisco ASA software older than version 8.2 needed the NAT to be performed mandatorily but now it is not the case. So for older ASAs, there could have been traffic drops here, but not now. 

L3 Route Lookup

Egress interface is determined 1st by NAT rules or existing connection entries. If couldn't find a macth, global routing table will be consulted.

If the L3 route lookup failed, you will get a syslog similar to following.

%ASA-6-110003: Routing failed to locate next hop for TCP from INSIDE 192.168.10.10/18653 to OUTSIDE:203.115.50.20/23

L2 Address Lookup

If layer 2 lookup fails, you will not see a syslog. show arp will not show required entry and debug arp will indicate if we are not receiving ARP reply. 

TX Ring Buffer Underrun

Same as the Buffer Overrun, if the output queue if full, the buffer underrun counter will increase and packets will be dropped.

Notes on Wireshark Filtering Commands

Packet analysis is a crucial skill for application level troubleshooting. Primary tool is Wireshark and following are some commands which will often come in handy to filter the interested traffic.

Filtering Conditions

These are the filtering criteria to be matched and displayed. You just need to type the line in filter tool bar. 

Logical Operators

You can just type the name of the operator in English or you can use the logical symbols like you do in programming to make a logical filter which will display the logical truth result of 2 or more conditions.

Using above commands, conditions & operators you can write complex statements like the following to filter packets in a capture.

Ex:- !(ip.flags == 0x40) && ip.addr==10.227.54.142

Additional Notes:-

If you want to filter packets by a protocol name, you just need to type the protocol name

Ex:- 'dns' will display all the DNS packets

Ex:- 'http' will display all the http packets

If you want to find TCP packets with ACK flag set, you should use 'tcp.flags.ack == 1' 

If you want the TCP packets with acknoledgement number equal to 1 'tcp.ack == 1'

Use 'frames.len > 1514' will display the jumbo frames.

Checking Open Ports (Telnetting, TCPING & Scanners)

You can use various methods to check open ports of a server. Old school Telnet is the command prompt tool you can use easily for a 1 targeted host.

Telnet

Let's say I need to know whether https (TCP port 443) is available at facebook.com

The command would be;

telnet facebook.com 443

If it is open, it will give an output telling connecting... and then the screen will be totally black..

To quit from  the connection, you should hit Ctrl + ] keys and type quit.

Now let's try the same thing from Cisco CLI;

To quit from the session, you should hit Ctrl + Shift + 6 and release and enter X

It will go back to the Cisco CLI again but will not really disconnect the session, If you hit Enter it will continue the session again. So you need to type disconnect and hit enter..

TCPING

TCPING is a small Windows command prompt tool I use which does basically the same thing in a different way. You can download it from their official site

https://www.elifulkerson.com/projects/tcping.php

I have unzipped the folder and pasted the contents into the System32 folder.

It is simple; tcping facebook.com 443 command will show the replies if open.

Scanners

If you need to scan an entire network/subnet for open ports, you should use a network scanner. There are plenty of them and they are very easy to use. One I use in Windows is Advanced IP Scanner..

Allow Pings (ICMP) & Traceroutes through Cisco ASA Firewalls

ASA Firewalls does not allow ICMP traffic to pass through it's interfaces by default. For real scenarios it is better that way in terms of security concerns. But for lab purposes and to verify implementations you will need it to be allowed from Firewall.

Why ICMP is blocked by ASA?

Short answer is because it is not in the list of state full inspection protocols. 

You can see the default inspection protocols list on the capture.

So you may need to configure access-control rules for both source and destination interfaces but it will bypass the firewall functionality. Firewall should remember the legitimate ICMP traffic and allow only the return traffic to pass through.

So you will need to add ICMP to the default inspection policy in global policy which is under the service policies.

Doing it in CLI is simple;

policy-map global_policy

 class inspection_default

  inspect icmp

  inspect icmp error

Make ASA Visible in Traceroutes

By default ASA is not visible in traceroutes as it does not decrement the TTL. To make it visible in a traceroute, we will need to add the following configuration to the default class in global policy.

policy-map global_policy

 class class-default

  set connection decrement-ttl