[Soekris] Bug at VPN1201 card or on OpenBSD3.1

[Brian Camp]

I ran into this bug some time ago.  I'm still not sure if the bug is in
openbsd, the soekris boards or the hifn.  Heres a bug report that I
compiled last month.  Since then I tried loading the latest/greatest
bios and there was no change.


-Brian


A simple network map

Workstation-A                          Workstation-B
    |                                       |
   LAN-A                                   LAN-B
    |                                       |
  Router-A                                Router-B
    |					    |
    ---------------Wireless Link-------------
			 |
		   WorkstationC

A more complicated one

Workstation-A                          Workstation-B
   fxp0                                    fxp0
    |                                       |
   LAN-A                                   LAN-B
    |                                       |
   sis0(bridge0)                           sis0(bridge0)
  Router-A                                Router-B
   sis1                                    sis1
IPSEC(gif0(bridge0) - - - - - - - - - - -gif0(bridge0))IPSEC
    ---------------Wireless Link-------------
			 |
                        wi0
		    WorkstationC


Routers:
* -current with GENERIC (nmbclust increased and hifn disabled via UKC
  during testing)
* The two routers are bridging their respective LANs with gif tunnels
  to each other with ipsec flows(3des/sha1) to and from each other.
  (I followed the "IPSEC BRIDGE" example in brconfig(8).)
* soekris net4501s (http://www.soekris.com/net4501.htm)
* minipci hifn7951, soekris vpn1211
  (http://www.soekris.com/vpn1201.htm)
* sis0 is the LAN interface and sis1 is the wireless link
  (each is connected to an ethernet converter device)

Workstations:
* -current

I've had this setup running for around six months and it works great...
for the most part.  From the beginning, I've had the link go down every
two weeks or so at non peek times for no apparent reason.  Up until
yesterday (7/23/02) both routers were running 3.1-stable which ran into
the same issue.  Yesterday, while playing with nmap, I found a way to
reproduce the problem on demand...:) Ok, heres what happens:

* Workstation-A runs `nmap Workstation-B` no more than 3 times and the
  link appears to go down.
* `tcpdump -ni enc0` on Router-A shows only LAN-A -> LAN-B traffic
* `tcpdump -ni gif0` on Router-A shows LAN-A -> LAN-B traffic
* `tcpdump -ni enc0` on Router-B shows only LAN-B -> LAN-A traffic
* `tcpdump -ni gif0` on Router-B shows LAN-B -> LAN-A traffic
* `tcpdump -ni wi0` on WorkstationC shows traffic from RouterB, but
  none from RouterA.
* Router-A is able to ping Router-B's Wireless IP but not its LAN IP
  (and the opposite is true)
* I bring down all of Router-A's interfaces, delete all of its bridge
  entries and bring all of the interfaces back up again... and nothing
  changes.
* I run `ipsecadm flush` on Router-A and Workstation-C starts to see
  Router-A sending the tunnel packets - Reloading the flows causes
  it to stop sending again.
* Reboot Router-A and everything is back to normal.

The following seem to have no affect:
* OS version - 3.0, 3.1 and now current all run into the issue
* Disabling pf on the routers
* Bringing interfaces down and back up again
* Increasing nmbclust (no change from 2048 to 16384)

The following seem have an affect:
* Disabling hifn - I cannot reproduce the problem with hifn disabled
* Traffic type - I cannot reproduce the problem 'on demand' with heavy
  downloads (just under 3mbit) or pingflooding but `nmap host`
  reproduces it within three tries (usually only takes one)

Router dmesg:
(ignore the wi0, its not used)
OpenBSD 3.1-current (GENERIC) #42: Mon Jul 22 03:52:53 MDT 2002
    deraadt at i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: AMD Am486DX4 W/B or Am5x86 W/B 150 ("AuthenticAMD" 486-class)
cpu0: FPU
real mem  = 66695168 (65132K)
avail mem = 56205312 (54888K)
using 839 buffers containing 3436544 bytes (3356K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 20/20/24, BIOS32 rev. 0 @
0xf00c0
pcibios0 at bios0: rev. 2.0 @ 0xf0000/0x10000
pcibios0: pcibios_get_intr_routing - function not supported
pcibios0: PCI IRQ Routing information unavailable.
pcibios0: PCI bus #0 is the last bus
bios0: ROM list: 0xc8000/0xa000
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "AMD ElanSC520 Host-PCI" rev 0x00
hifn0 at pci0 dev 16 function 0 "Hifn 7951" rev 0x01: 128KB sram, irq 10
wi0 at pci0 dev 17 function 0 "Intersil PRISM2.5 Mini-PCI WLAN" rev
0x01: irq 11
wi0: PRISM 2.5 ISL3874A(PCI), Firmware 1.0.5 (primary), 1.3.4 (station),
address 00:05:5d:ee:58:bf
sis0 at pci0 dev 18 function 0 "NS DP83815 10/100" rev 0x00: irq 5
address 00:00:24:c0:2f:bc
nsphyter0 at sis0 phy 0: DP83815 10/100 integrated, rev. 1
sis1 at pci0 dev 19 function 0 "NS DP83815 10/100" rev 0x00: irq 9
address 00:00:24:c0:2f:bd
nsphyter1 at sis1 phy 0: DP83815 10/100 integrated, rev. 1
sis2 at pci0 dev 20 function 0 "NS DP83815 10/100" rev 0x00: irq 12
address 00:00:24:c0:2f:be
nsphyter2 at sis2 phy 0: DP83815 10/100 integrated, rev. 1
isa0 at mainbus0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
wdc0 at isa0 port 0x1f0/8 irq 14
wd0 at wdc0 channel 0 drive 0: <PQI ATA Rev6.0>
wd0: 1-sector PIO, LBA, 123MB, 984 cyl, 16 head, 16 sec, 251904 sectors
wd0(wdc0:0:0): using BIOS timings
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom0: console
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
biomask 4000 netmask 5e20 ttymask 5e22
pctr: no performance counters in CPU
dkcsum: wd0 matched BIOS disk 80
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302
Automatic boot in progress: starting file system checks.
/dev/rwd0a: file system is clean; not checking