[Soekris] high CPU usage for network interrupts on 4801

[Jan Hoevers]

I'm using a Soekris net4801-60 as a router, with a lan1641 in the PCI 
slot, giving me 7 ethernet ports. I'm running pfSense 1.0.1, a routing 
package based on FreeBSD 6.1 and PF. For quite some time everyting was 
running file.

Recently I've joined the NTP pool (http://www.pool.ntp.org/). My ntpd 
doesn't run on the Soekris, but on an old PC sitting behind it, the NTP 
traffic passes the Soekris on its way. Note that my question is about 
the Soekris, not about my NTP server.

The NTP pool doesn't generate much traffic, but it comes in bursts of 
approximately an hour. Yesterday I was closely monitoring such a burst, 
with a bandwith of 230 kbit/sec, quite stable for about 30 minutes. 
Nothing extreme for a 4801, I would say.
It surprised me that the Soekris was running > 50% CPU, the "top" 
display revealed that it was almost entirely interrupt processing.

At the same time I was watching "top" on the NTP server (an 8 year old 
300 MHz PC running FreeBSD 6.2 with a low cost Realtec network card in a 
PCI slot, not exactly impressive equipment). This CPU was using < 10% 
CPU, of which 3% for interrupts, while of course handling the same 
amount of traffic (well, make that half the traffic, it passes the 
Soekris twice of course).

This would mean that I will run into trouble with a NTP spike of 500 
kbit/sec. That surprises me because I remember posts on this list 
stating about 40 Mbit/sec throughput for a 4801.

I've tried downloading a large file to the NTP machine at a much higher 
speed (6800 kbit/sec), which takes about 75% CPU for interrupt 
processing. I guess the small size of NTP packets is inefficient. 
However, this 75% is stil much higher than I would expect.
I've tried moving the NTP machine from an ethernet port on the 1641 to 
one on the 4801 board in an attempt to avoid the PCI bus, but it makes 
little or no difference. (Maybe I'm not avoiding the PCI bus this way, 
I'm not sure.)

I'm not a hardware or unix guru and I've run out of clues. What's going 
wrong? Is there anything I can do? Or do I see a problem that doesn't exist?
Would it be better to leave pfSense and install FreeBSD and PF from scratch?
Of course I could leave the NTP pool, but I would prefer something 
better as a solution.

Any thoughts or suggestions would be highly appreciated.

Jan Hoevers.


P.S. After I wrote this I saw a 410 kbit/sec NTP burst using 80% CPU for 
interrupts, with spikes up to 100%. Seems as if it's linear with the 
number of incoming NTP requests.