FreeBSD and Internet Attacks:Protecting yourself from denial of service, by Avleen Vig |
The Internet is no longer the cute and fluffy cloud it once was.
This article applies equally to FreeBSD 4.x and 5.x
Protecting your servers, workstations and networks can only go so far. Attacks which consume your availible Internet-facing bandwidth, or overpower your router's CPU can still take you offline. This paper is aimed at mitigating the effects of such attacks, and guiding you in what you should do if you are attacked.
Denial of Service attacks as their name implies, set out to remove a service from functional use by its clients. Web servers will stop serving web pages, e-mail servers will stop accepting or delivering e-mail, and routers will go dark taking you off the Internet all together.
Denial of a particular service will come in one of two forms:
Over the last few years, attackers have refined their methods. As developers make software more reliable and more resiliant to DoS, the attack vectors have changed to target hard to secure parts of a service. In this paper we will discuss the first type of attack, and what we can do to protect our services from it.
Protecting your services from attack follows many of the same idiologies as tuning your
services for maximum performance. The greater the load you can handle, the more resilliant you
are. Things change slightly when the attack changes the profile of your service.
For example, if you have a webserver tuned to transfer mostly large file transfer and your
attack forces through a lot of small, shortlived transactions, you could find you run out of
network memory buffers very quickly. I would recommend starting by reading the papers on
tuning FreeBSD for different applications:
http://silverwraith.com/papers/freebsd-tuning.php
The paper describes good ways to start tuning your servers. Also, the tuning(7) man page is an excellent resource on performance improvements.
The first step to protecting yourself from an attack is to understand the nature of different types of attacks. As we said earlier, resource consumption attacks target your system in places which can cause bottlenecks. The most popular targets are network bandwidth, system memory, network stack memory, disk I/O, operating system limitations such as a limit on the number of open file handles, and the CPU. These bottlenecks can either be on your systems or in your network hardware.
Attacks against your network bandwidth are amongst the hardest to defend, and how you deal with them depends heavily on your network topology and how helpful your ISP is. To start with ask yourself the following questions:
We are fortunate today, that most attacks are simple in their nature. They choose one or two styles of attack and at most a small number of IP addresses. This makes sense - bandwidth is as hard for attackers to acquire as it is for us to defend. If your Internet peering bandwidth is not saturated, the accepted approach is to block traffic to the attacked host(s) at your gateway. It is a good idea to run tcpdump on the attacked servers if you can, to see what kind of attack is taking place. Look for floods of very similar packets - all TCP SYN, UDP or ICMP. Look for packets all going to a particular port. If you find the number of source IP addresses is reasonably small, blocking out the packets based on source address may be possible. However if the source addresses are highly volatile in addressing, this can be an indicator that they are spoofed (forged). When this is the case, you may need to look for other similarities in the attack such as packet size, window size, fragmentation, etc. If you have the ability to block based on these less common criteria you may want to investigate here further. In today's multi-gigabit networks, it is not unusual for an Internet connection to have more bandwidth than the local LAN, and so it may be possible for you to block the attack at your Internet gateway.
More often than not though, this does not apply, and having your Internet bandwidth consumed can be a tiring and frustrating ordeal. This might be the right time to call your ISP, if you have one who is willing to work with you on these problems. Before you make the call, try to analyze the attack. This is most certainly help your ISP in selectively filtering the attack off your network. If filtering is possible, one of two common options will usually be available, selectively filtering out the attacking systems, or dropping all packets to the attacked servers. The latter is often preferred as it is easier to manage and would be more effective in the event that the attack profile changes against those hosts.
If you run Border Gateway Protocol (BGP) at your Internet gateway to announce your IP space to the Internet you may have a third option, one which is becoming popular with a lot of ISP's. UUNet, C&W, XO, and many others are allowing users to export routes as small as /32, with a special community string which causes all incoming data for the route to be dropped at the ISP's border. This is a highly effective method of dropping an attack on the floor with the least damage to yourself and your ISP. Of course this only works well if the number of hosts being attacked is small, or if your ISP offers such functionality. Contact your ISP to find out. The obvious downside of this is that the IP addresses you export in this fashion will lose ALL connectivity to the Internet.
In general is it a good idea to keep your network clean; only allow traffic on to it which is needed for your services to operate. Allow TCP to ports 80 and 443 on your servers and allow UDP to your game servers. Allow SSH connections only from trusted hosts. All of these limit the options of the attackers when they come to visit.
If your bandwidth is not saturated, it is most likely that the attack is against your systems and the services they host, rather than your entire network. Again, the remedy depends on the nature of the attack. Systems contain all of the possible bottlenecks which can be targeted, and you may find that more than one bottleneck is exposed at any one time. Attacks on systems and their services generally fall into the following categories:
System-targetting attacks can be some of the most frustrating as people have a hard time defending against them. There is however some special magic which FreeBSD lets us use to help out.
By default, each time your network card receives a packets, it generates an interrupt to the CPU along its IRQ. The CPU will catch this and dedicate an amount of time to fetch this packet from the interface. Under normal operations this can happen several thousand times per second which is well within the capabilities of even low end CPU's. It is quite likely with older CPU's that you will start to see performance impacts at around 25,000 to 50,000 packets per second. With packet sizes of 1500 bytes, this works out to around 40Mbytes/sec to 75Mbytes/sec, which is quite a lot for most older CPU's to serve anyway. Most 1Ghz systems will begin to feel pressure around 75,000 packets per second. The problem is exasperated by two factors:
As we discussed previously, each time an IRQ is generated some CPU time is taken. If enough IRQ's can be generated, the CPU will have no time to do anything other than serve the interrupts. Inbound packets do not get processed, applications get no CPU time, and your system is effectively dead in the water. This is known as "Live-lock". Your system is still live, in so much that it has not crashed, but it is locked from performing any useful functions. Once packets stop coming in to the interface, the CPU starts to process all of the backlogged packets it has already accepted. This can take anything from a few minutes to several hours.
There are several things you can do to prevent or mitigate the effects of a high rate of packets, before you need to go out and buy any hardware upgrades. All of these are performed using FreeBSD's sysctl(8) command. Here are the settings you will need, you can place them in /etc/sysctl.conf:
You may find these settings to either be too aggressive, or not aggressive enough. You should tune them until you receive satisfactory results.
Finally, if you are blessed enough to own one of the following network cards you can enable a kernel feature call DEVICE_POLLING:
DEVICE_POLLING changed the way that interrupts are handled. Actually with DEVICE_POLLING,
they are not handled at all! DEVICE_POLLING causes interrupts to be effectively ignored.
Instead, at certain times, the CPU will poll the network card, and pick up an packets that are
waiting for processing. This can significantly reduce the amount of CPU time used in
processing inbound traffic, but only the above cards are supported as the drivers have to be
written to support DEVICE_POLLING. The FXP cards generally work best with the feature as their
drivers are very well developed, as is their hardware. The hardware design and quality of RL
cards is a lot lower - without sufficient CPU (usually around 1Ghz), they have a hard time
achieving the full 100MB/s at all. If you are looking for a new network card, you will get
what you pay for!
You can learn more about DEVICE_POLLING at the author's home page:
http://info.iet.unipi.it/~luigi/polling/.
You can also find good installation and tuning instructions there, as well as some statitics
from comparative tests with DEVICE_POLLING enabled and disabled.
Attacks can come from inside and outside your network and obviously one is easier to isolate than the other. Tracking the sources of attacks requires some familiarity with packet sniffing tools such as TCPDump, ngrep, or ethereal. Unless you have spent several months, carefully profiling your network traffic and set up monitoring specifically to alert you of anomalies, the chances of discovering you are under Denial of Service conditions before someone else does are slim. More often than not it is complaints such as "The Internet is slow", or "I can't get my e-mail" that lead us to find the truth. It is important to realize two things:
What does this mean to you? It means that when you start to look for why "your Internet is slow" or why people cannot get their e-mail, remember that the source of the problem could be from any machine on your network or the Internet, and that the denial may not be deliberate.
A good place to start is the point of bottleneck. This could be the CPU on your HTTP proxy, or maybe your Internet gateway. If your bottleneck is a system process such as a proxy server, examine the logs for this. Is there a single system or small number of systems making an unusually large number of requests, or using more resources than they should? If your bottleneck is your Internet gateway (which we assume is running FreeBSD), you can use a command like this to view what IP packets are passing through your gateway:
router# tcpdump -n -i <interface> -c 100
This command will display a summary of the first 100 packets (-c 100) it sees,
on the <interface> (-i <interface>), and will not resolve the IP
addresses to host names (-n) which can take extra time and may itself fail if you
are having connectivity issues. An example output line may look like this:
04:59:53.915324 192.168.0.3.2327 > 192.168.0.10.1214: S 3199611726:3199611726(0) win
16384 <mss 1460,nop,nop,sackOK> (DF)
Let us look at the first few parts of this output which can be useful to us.
What you may see during an actual attack is hard to predict, as denial of service attacks come in so many shapes and sizes. A typical attack involves flooding a listening port on your server, with SYN packets. The idea is to make your system so busy processing the new connections that it cannot do anything else. Here you may be a large number of SYN packets. Usually these should be well balanced with packets of other types.
WraithNet is a personally run network for free web hosting for anyone publishing documents
or data relating to the open-source community.
If you would like a sub-domain for your mail and web pages, send an email to:
webmaster@silverwraith.com
This is a list of references I found when writing this paper. Most of them contain much more detail on their particular subject than I have provided here. Hopefully they will be as useful to you as they were to me.
If you find a problem with this website, please email:
webmaster@silverwraith.com
| Counter: 38987 page |
| views since Jan 2004 |