Practical Troubleshooting

I love pfSense. So far it's superior to every Linux-based routing appliance. No product is perfect, but the 2.0 release is very promising. I have been troubleshooting tunnels which inexplicably do not work. I have been recieving the following error during phase1 connection:

racoon: ERROR: couldn't find configuration

This usually means a significant mismatch exists in phase1 negotiations. Despite my meticulous efforts the tunnels would not start. I watched the IPsec logs hopelessly, trying many different things. What finally worked was connecting to the console, killing racoon and starting it manually as follows:

racoon -d -v -F -f /var/etc/racoon.conf

By monitoring the output, I discovered during debugging that the packets were coming from the wrong source IP address. One of my sites has multiple WAN links, and racoon was using the wrong source address for IPsec negotiation. The phase1 arrival was clearly logged and rejected - because it didn't match any existing configuration.

Once complete, I was quickly able to determine what to do. However, if you don't have access to a host behind the pfSense firewall then you may have problems creating IPsec tunnels. I used this to force a packet:

My firewall's LAN address, which is part of the IPsec local subnet scope, is 192.168.0.1. The remote network is 10.1.1.0/24. I need to create a single packet from 192.168.0.1 to something in 10.1.1.x.

ping -S 192.168.0.1 -c 1 10.1.1.3

What could be better.

Feedback for improvement would begin with one admonition: Don't trust the log output of Racoon. I should have used TCP-dump on both ends, watching for packets setting up sessions.

[2.0-RC1][root@gateway.site-a.com]/root(1): tcpdump -ni re0 port 500
16:39:07.697695 IP 192.168.81.126.500 > 10.1.101.217.500: isakmp: phase 2/others I inf[E]
16:40:26.980944 IP 10.1.101.217.500 > 192.168.81.126.500: isakmp: phase 1 I agg
16:40:36.982740 IP 10.1.101.217.500 > 192.168.81.126.500: isakmp: phase 1 I agg
16:40:46.983927 IP 10.1.101.217.500 > 192.168.81.126.500: isakmp: phase 1 I agg
16:40:56.985122 IP 10.1.101.217.500 > 192.168.81.126.500: isakmp: phase 1 I agg
16:41:06.986307 IP 10.1.101.217.500 > 192.168.81.126.500: isakmp: phase 1 I agg

Had I done this at both ends, I would have clearly seen that the wrong interface was emitting packets. Both ends. That was my mistake. I trusted logs at Site-A, and I never verified my problem at Site-B. Hours of painstaking troubleshooting for no good reason.

Work Around

My current imperfect workaround is to add the following line to each of my remote sites crontab:

cat crontab|grep newsys






root    /sbin/ping -S 192.168.0.1 -c 10.1.1.1

Obviously I turfed this above, I just thought I would share it with everyone. This has the net effect of a 60 second tunnel keep alive. May not be appropriate for some environments. Good luck.

In which Joshua discusses his new-found love of Extreme Networks Switches.

A New Client

A client turned up our first full-time Mail.app Mac user with Snow Leopard today. I was called in because of attachment sending problems. It seems that files around 7MB would attach and send, but anything larger was failing. The entrenched support reported watching logs, etc. IIS was returning a 401 then a 500 for the sessions that failed, and there was no clear reason.

Troubleshooting

After a little inspection, I thought it might be the request size / http post size. After a quick verification, i determined that the registry limit was not interfering. I next examined the web.config, located at C:\Program Files\Microsoft\Exchange Server\ClientAccess\exchweb\EWS on the server in question. Sure enough, at the bottom:

    </customErrors>
    < httpRuntime maxRequestLength="819200" />
  </system.web>
</configuration>

I increased the limit, and restarted IIS for good measure. The attachments which the server balked at now sent easily and quickly. Yay!.

Errata

During these fateful events, while working and trying to send a 150MB iso to another mail box, Mail.app went nuts and began eating RAM. In less then one minute, it had consumed 1.2GB and swap began on my poor Mac. Quick fingers and a killall Mail in a ubiquitous terminal window solved the problem.

For certainty, I wiped my Mail directory from my Library (I don't use Mail.app for this reason). Problem Solved.

Question

Why is this setting so obscure in Exchange?

If you have to work in networks with any Windows products, coupled with MSDE or SQL Express you will eventually run into memory consumption problems. Apparently no GUI interface deals with it. I have seen numerous complaints on the Internet for sqlservr.exe consuming loads of memory. Some psychos recommend "uninstalling and re SBS Diva has a great article which I will here condense:

osql is the command-line tool for monkeying with MS-SQL200*.

c:\> osql -E -S MYSERVER\instancename
1>

So, first add the "Process ID" column to Task Manager. Note the PID of the offending SQL process. Next, open command prompt, and run tasklist /svc. Locate the PID matching the process, and find the name you want:

sqlservr.exe                  1972 MSSQL$SBSMONITORING
sqlservr.exe                  2020 MSSQL$SHAREPOINT

The part after the '$' is the instancename. (Hopefully you already know your machinename.)

Once you have the instance, run osql as shown above.

c:\> osql -E -S MYSERVER\instancename
1> sp_configure 'show advanced options',1
2> reconfigure
3> go
1> sp_configure 'max server memory',128
2> reconfigure with override
3> go

max server memory: this option is in megabytes. This will change the 'MSSQL$INSTANCENAME' to operate at 128 Megabytes. (When set this way, most of my instances reported between 160M and 180M when in use.)

Notes

It's simple to list all of the configured parameters for the server, simply load osql, as shown above, and run:

1> sp_configure 'show advanced options',1
2> reconfigure
3> go
1> sp_configure
2> go

That will dump all the configured options. It of course enabled advanced options.

I hate Apache. I really do. I refuse to vindicate that hatred. There are great aspects about it, but the things I want to do are hampered by things like the sewer-refuse-styled configuration syntax.

I like Nginx. It is fast, simple, and is amazing. It does proxy, reverse proxy, rewrite, ssl, and everything else. Cliff Wells cooked up a wiki. As linked before, it cleans up when facing off with Apache, Pound, Lighttpd, etc. It has all the core features that sane people need.

I just wrote a recipie for Trac + Nginx over at Edgewall. Trac + Nginx + PostgreSQL Kicks ass.

Note: I still use Apache for moddavsvn, and a mod_python only application. (But that will be fixed soon enough). SVN is another question entirely.

One problem that I have frequently is remembering how to list NFS exports on a remote server. It's really simple:

showmount

osXlt:~ joshua$ showmount -a gambit
All mount points on gambit:
osXlt:~ joshua$ showmount -a forge
All mount points on forge:
:/data
,bubbles.mynetwork.com:/data
,bubbles.mynetwork.com:/usr/portage
,mdd.mynetwork.com:/data
,shelob.mynetwork.com:/data
,shelob.mynetwork.com:/home
,shelob.mynetwork.com:/usr/portage
,thrall.mynetwork.com:/data
*,thrall.mynetwork.com:/home

It's simple. I just keep forgetting. Score one for perpetuity.

Task: Upgrade Symantec Antivirus for Microsoft Exchange.
Diffculty: Symantec Continues to Suck.

Despite a clear desire to escape from these things it can be difficult. Todays installment brought a new error: "Please Insert Disk 1". Despite all my best attempts, I could not divine what disk '1' was/is.

After much searching, I finally found a link reporting that this bug was fixed - ??? - and it was related to "Installing MSE gateway from removable media". Since I couldn't find a download link, or other way to update, I copied the entire install directory to my c:\ drive, and viola! It works. Yay!

This is a simple, but cool, recipe for querying Exchange from Postfix. This is used with a Windows 2003 Small Business Server, running (s)Exchange 2003.

We don't often use this, because of the obvious problem of being unable to receive mail when Exchange crashes or must be rebooted. It is nice though, and a straight forward solution.

exchangemap.cf

binddn          = cn=Spamfilter User,cn=Users,dc=MySBS,dc=org
bindpw          = kill$pam
scope            = sub
searchbase      = dc=MySBS,dc=org
serverhost      = ldaps://Server.MySBS.org:636
starttls        = no
version          = 3
resultattribute = mail
queryfilter     = (&(objectClass=user)(|(mail=%s)(proxyAddresses=SMTP:%s)))

test

firewall postfix #  postmap -q kelly@MySBS.org ldap:/etc/postfix/exchangemap.cf
kelly@MySBS.org
firewall postfix #  postmap -q fakeuser@MySBS.org ldap:/etc/postfix/exchangemap.cf
firewall postfix #  postmap -q joshua@imrnet.com ldap:/etc/postfix/exchange_map.cf

main.cf

.....
relayrecipientmaps  =  ldap:/etc/postfix/exchange_map.cf
.....

So, Kelly exists, but fakeuser, and Joshua are (mysteriously) absent.

Everyone wants to know what to do with Postfix. I have tried to find a way to publish something. Everyone who does seems to always be out of date.

Not so for this guy or that guy.