This chapter is from the book
This chapter is from the book
This chapter is from the book
Troubleshooting Your Firewall
The firewall services discussed in this chapter provide a point-and-click graphical user interface in Server Admin for simplified configuration. Of course, this simplification makes troubleshooting a little more complicated when things do not run as expected, especially with the firewall service.
The firewall service requires planning. You should make one change at a time and ensure that each change has the intended effect. Implementing too many changes at once makes it difficult to locate the source of a problem. Documenting changes is essential. When several administrators have permission to make changes to the firewall rules, it is imperative that each administrator be kept current with the actions of the others.
When rules are not working and time is not an issue, you can employ several tools and methods—such as ping, telnet, Network Utility, and tcpdump—to troubleshoot the firewall service.
Let’s examine a potential firewall configuration issue and the troubleshooting steps necessary to identify and remedy that issue. To simplify the exercise, we will utilize a server that is on an internal network and demonstrate restrictions and permissions on that internal network.
- Open Server Admin.
- Select the name of your server in the leftmost pane.
- In the Toolbar, click the Settings button.
- Select the Services tab.
- Select the checkbox to enable configuration of the firewall.
- In the lower-right corner of the Server Admin window, click Save. The Firewall service will now have an entry in the Services section of the left pane of the Server Admin window as seen in the following figure:
- In the left pane, click the Firewall service to view its settings, but do not start the service.
- If necessary, click the Address Group tab.
- Notice that the default address groups are 192.168-net and 10-net, which may or may not reflect your network settings. You will address these items first.
- Double-click 192.168-net and change the name to your internal network’s IP range. For example, the server used in this exercise has an internal IP address of 172.16.100.130 and a net mask of 255.255.255.0, so 172.16.100-net is appropriate.
- Enter your internal network range in the “Addresses in group” field.
- Click the 10-net address group to select it, then delete it by clicking the Delete (–) button at the bottom of the configuration window.
)
)
Test the Firewall Using Command-Line Tools
You can test and verify the performance of your firewall using commands in the Terminal application.
- On your client machine, open the Terminal application.
- ping your server’s IP address using your server’s internal IP address instead of the address shown below.
ping 172.16.100.130
- Press Control-C to exit the ping. The report should look like this:
PING 172.16.100.130 (172.16.100.130): 56 data bytes 64 bytes from 172.16.100.130: icmp_seq=0 ttl=64 time=0.344 ms 64 bytes from 172.16.100.130: icmp_seq=1 ttl=64 time=0.586 ms 64 bytes from 172.16.100.130: icmp_seq=2 ttl=64 time=0.545 ms 64 bytes from 172.16.100.130: icmp_seq=3 ttl=64 time=0.537 ms 64 bytes from 172.16.100.130: icmp_seq=4 ttl=64 time=0.171 ms 64 bytes from 172.16.100.130: icmp_seq=5 ttl=64 time=0.721 ms.
This report shows that a ping packet was sent from your client machine (which utilizes the ICMP protocol) to the server, and the server responded. The output of the command displays the number of bytes received, the address received from the sequence (or the order of the ping request), the time to live, and the response time in milliseconds. Enabling stealth mode in the advanced section of the firewall configuration will block ping and other network status requests. Stealth mode does not block these requests from the internal network unless an explicit rule is set in the advanced tab of the firewall configuration.
- Click the Logging tab and then select “log all denied packets.” Click Save.
To test whether a specific port is blocked or allowed, you will test the AFP service using telnet to check if the connection can be established.
- If the AFP service does not appear in the left pane of your server admin window, repeat steps 1 through 5 of the previous task to add the AFP service. Click the AFP service, and then click the Start button.
- In the Terminal window on your client machine, use the following telnet command, inserting your server’s internal network
IP address to replace 172.16.100.130 (note that 548 is added to the command because AFP utilizes port 548):
telnet 172.16.100.130 548
The command output should resemble the listing below, which shows that telnet was able to establish a connection to the AFP service on port 548:
Trying 172.16.100.130... Connected to 172.16.100.130. Escape character is '^]'..
- Press Control-J to close the connection.
- At the bottom of the Server Admin window, click Start Firewall to enable the default firewall rules.
- In the Terminal window on your client machine, type the following telnet command, inserting your server’s internal network
IP address instead of 172.16.100.130.
telnet 172.16.100.130 548
The command output should look something like the listing below, illustrating that telnet was able to establish a connection to the AFP service on port 548:
Trying 172.16.100.130... telnet: connect to address 172.16.100.130: Operation timed out telnet: Unable to connect to remote host
- In the Server Admin toolbar, click the Log button and examine the log.
- To filter the results and view only the DENY messages, at the right side of the Toolbar, type DENY in the filter area. Your output should be similar to this:
Nov 22 13:41:43 server ipfw[2086]: 65534 Deny TCP 172.16.100.1:50971 172.16.100.130:548 in via en0 Nov 22 13:41:46: --- last message repeated 3 times ---.
Looking at the output from the log the information displayed in order from left to right, you’ll see the date, the time, the host name of the server, the firewall component that responded (in this case, ipfw), the rule number, the action that was taken on specified ports and Ethernet card. Notice that this action occurred an additional three times and that the server IP address and requested port number is displayed last, not first.
Another method to test whether the firewall allows a port through or denies it is to use the Network Utility application located in /Applications/Utilities.
- Open Network Utility on your client machine and click the Port Scan tab. Enter your server IP address in the “Enter an internet
or IP address to scan for open ports” field and enter 548 in both “Only test ports between” fields. Verify that the “Only test ports between” checkbox is selected, and then click
the Scan button.
As seen below, the port scan is initiated and completed with no indication of an error. This is the normal behavior—Network Utility is reporting that it found no open port at the addresses specified.
- Using Server Admin on your server, click the Stop Firewall button and repeat the network scan described in step 12.
This time the output includes a line indicating that the network traffic connected to the server on the specified port.
Open TCP Port: 548 afpovertcp
)
Using a Packet Sniffer
If rules look like they should be working, but they are not, and time is not an issue, a packet sniffer will allow you to view the problem from the inside. Mac OS X ships with the tcpdump packet sniffer. This popular open source sniffer can print information in real time for a quick determination, or it can write its dump to a file for later, offline analysis.
- To dump all packets passing through the en0 interface to stdout, use the following commands:
# tcpdump -i en0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on en0, link-type EN10MB (Ethernet), capture size 96 bytes 06:42:37.514597 IP mainserver.pretendco.com.ssh > 72.14.228.89.11667: P 3011092458:3011092650(192) ack 3151495640 win 33312 <nop,nop,timestamp 1488505430 575729407> 06:42:37.523288 IP 72.14.228.89.11667 > mainserver.pretendco.com.ssh: . ack 0 win 65535 <nop,nop,timestamp 575729408 1488505364> 06:42:37.589744 IP 72.14.228.89.11667 > mainserver.pretendco.com.ssh: . ack 192 win 65535 <nop,nop,timestamp 575729409 1488505430> 06:42:37.637216 IP comproxy2.example.net.53730 > 192.168.1.1.9090: UDP, length 74 06:42:37.760644 arp who-has 192-168-232-92.in- addr.arpa.example.com tell 192-168-232-1.in-addr.arpa.example.com 06:42:38.137423 IP comproxy2.example.net.53730 > 192.168.1.1.9090: UDP, length 74 06:42:38.320315 arp who-has 69-55-228-118.in- addr.arpa.johncompanies.com tell 69-55-228-1.in- addr.arpa.johncompanies.com ^C 316 packets captured 349 packets received by filter 0 packets dropped by kernel
- Press Control-C to stop the capture.
Each line displays the time that the packet was received; the protocol; the source of the packet; its destination; and, optionally, any set flags.
- To limit the tcpdump capture to a specific IP address, specify the host in the tcpdump command:
# tcpdump -i en0 host 10.1.0.1
- Perhaps even more useful when troubleshooting a firewall problem is to limit the capture to a specific port—port 80, in this
exercise:
# tcpdump -i en0 port 80
You can also negate a filter using the not keyword. For example, if you are troubleshooting remotely via the Secure Shell protocol (SSH), the SSH traffic that makes up your session is typically noise. Furthermore, you can combine filters with the and conditional.
- To listen for all traffic from the host at 192.168.55.8, but filter out traffic on port 22, use this command:
# tcpdump -i en0 host 10.1.0.1 not port 22
Sometimes, analyzing tcpdump on the fly is not enough. For deeper analysis, you can write dumps to a file and analyze them offline using a more powerful program, such as the open source application Wireshark. For deeper analysis, you can also increase the size of the packet capture. By default, tcpdump captures only the first 96 bytes of each packet. For a deeper look, you should capture the entire packet.
- To capture all traffic of unlimited packet size and write it to a file, you can use the -s (size) and -w (write) switches:
# tcpdump -i en0 -s0 -w server_trace.pcap
The -s0 (“ess zero”) designates unlimited packet size, and the -w switch in this example writes all capture data to the file server_trace.pcap.
A common error when configuring the firewall is accidentally creating conflicting rules. Review your rules and the order in which they are executed to ensure that your configuration is correct. Utilize the tools above to test the traffic through the firewall and adjust your rules accordingly.
In most cases, all you’re looking for on a server is any activity getting through the firewall or to the local interface. The more detailed traces are typically more helpful from the client side. In a worst-case scenario, you can document and back up the current firewall rule set, and then flush the current rules entirely. Then you can add half of the rules back in at a time, starting the firewall service each time that you introduce a new set of rules. Incrementally adding back rules will make it easier to determine which rule is causing the undesired behavior.
If you become completely lost with the firewall configuration, you may want to back it up and start from scratch. For instructions on resetting the firewall to default values, see “Resetting the Firewall to the Default Setting” in the Apple Network Services Administration document at: http://images.apple.com/server/macosx/docs/Network_Services_Admin_v10.6.pdf.