(Available in HTML at <http://canonical.org/~kragen/crywrap.html>.) So, in March, we upgraded our machine from Sarge to Etch. We had been using `sslwrap` in Sarge, but `sslwrap` doesn't exist in Etch. According to Jonathan McDowell, the guy who used to maintain the Debian `sslwrap` package: sslwrap (2.0.6-18) unstable; urgency=low * Users might like to consider switching away from sslwrap to crywrap or investigating whether more recent versions of the services they're sslwrapping are themselves now ssl enabled. It is envisaged that at some point in the future I will request removal of sslwrap from the archive, though I hope to investigate the possibility of a smooth upgrade path to crywrap before that happens. sslwrap is effectively dead upstream and I think it's probably better to consider the existing alternatives that can perform the same function than continue to work on sslwrap long term. -- Jonathan McDowell <noodles at earth.li> Sat, 13 Aug 2005 13:01:06 +0100 (from <http://ubuntu2.cica.es/ubuntu/ubuntu/pool/universe/s/sslwrap/sslwrap_2.0.6-18.diff.gz>) See also <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=374521>, where the maintainer requested its removal from Debian. Why I Tried, Then Gave up on `CryWrap` ------------------------------------ `apt-cache search sslwrap` found only `crywrap`, and `apt-cache show crywrap` said: > `CryWrap` is intended to be a drop-in replacement for `sslwrap`. This is more or less a blatant lie. `CryWrap`'s command-line options have nothing in common with `sslwrap`'s, and `sslwrap` is written to run from `inetd` --- for example, it reports its errors through `syslogd`, not by printing them to standard error. `CryWrap` is not, even though it has an `--inetd` option. Here are the problems I encountered trying to use `CryWrap`: 1. `CryWrap` doesn't support `sslwrap` command-line options. 2. `CryWrap` reports its errors to `stderr`. 3. When I did get `CryWrap` to work, it reliably took about 110 seconds to negotiate an SSL connection, which is longer than Thunderbird is willing to wait. 4. `CryWrap`'s `sslwrap` wrapper doesn't support `inetd`. 5. `CryWrap`'s documented `-v` flag doesn't work as documented. I wasted an hour trying to get `CryWrap` to work, and eventually gave up and installed `stunnel4` instead. Here are some details, in case you find yourself in a similar situation: ### `CryWrap` doesn't support `sslwrap` command-line options. ### The line in our /etc/inetd.conf for `sslwrap` looked like this, all on one line: pop3s stream tcp nowait root /usr/sbin/tcpd /usr/sbin/sslwrap -cert /etc/sslwrap/server.pem -addr 127.0.0.1 -port 110 I `apt-get install`ed `crywrap`, changed `sslwrap` to `crywrap`, and hoped for the best. Initially, that failed because `tcpd` was specially configured to allow connections to `sslwrap` from weird places, in `/etc/hosts.deny`: ALL EXCEPT sslwrap: PARANOID EXCEPT <censored> Our Argentine ISP, TeleCentro, is so incompetent that our reverse DNS maps our IP address to a name that doesn't exist. So `tcp_wrappers`'s `PARANOID` rule won't allow us to connect to tcp-wrapped services. So I changed that to say ALL EXCEPT crywrap: PARANOID EXCEPT <censored> Then I ran into the problem that we had removed the `/etc/sslwrap` directory and the `server.pem` file inside it that contained the server's private key. After a little bit of digging, I found out where to get the private key file, stuck it in `/etc/crywrap/server.pem`, and put the following completely wrong line in `/etc/inetd.conf`: pop3s stream tcp nowait root /usr/sbin/tcpd /usr/sbin/crywrap -cert /etc/crywrap/server.pem -addr 127.0.0.1 -port 110 You see, at this point, I still believed the package description that claimed that `CryWrap` was "a drop-in replacement". I got a log line (apparently from `tcpd`) that said the connection had been made: Mar 29 19:51:03 panacea crywrap[27458]: connect from 190.55.55.32 (190.55.55.32) But the `crywrap` process had died, and there were no error messages in any of the `/var/log` files explaining why. This is because of problem #2, "`CryWrap` reports its errors to `stderr`," which I explain below. Upon consulting the man page and debugging from the error message for a while, I ended up with the following line in `/etc/inetd.conf` instead: pop3s stream tcp nowait root /usr/sbin/tcpd /home/kragen/crywrap -d 127.0.0.1/110 -i `/etc/crywrap/server.pem` is the default location for `CryWrap` to look for a server certificate, so I omitted it from the command line. ### `CryWrap` reports its errors to `stderr`. ### In order to find out what was wrong, I temporarily ran `/home/kragen/crywrap` instead of `/usr/sbin/crywrap` from `inetd`. `/home/kragen/crywrap` is this script: #!/bin/sh /usr/bin/strace -s4096 -o /tmp/crywrap.strace /usr/sbin/crywrap "$@" And it turned out that `CryWrap` was writing its error messages to `stderr`, file descriptor 2, instead of to a log file. `stderr`n in a process run from `inetd` is actually connected to the socket talking to the client, so writing error messages to it is almost certain to violate the protocol expected by the client. Here's one sample error message (a result of problem #4, "`CryWrap`'s `sslwrap` wrapper doesn't support `inetd`", below) from strace's output (wrapped for readability): write(2, "crywrap", 7) = 7 write(2, ":", 1) = 1 write(2, " ", 1) = 1 write(2, "Could not resolve address: `/\'", 30) = 30 write(2, "\n", 1) = 1 write(2, "Try `crywrap --help\' or `crywrap --usage\' for more information.\n", 64) = 64 exit_group(64) = ? This was at the very end of the file. Now, this would not be such a heinous sin in a program that was intended to speak, say, SMTP. If a fatal error message gets sent to an SMTP client, it's likely to end up somewhere that a human being can see it and diagnose the problem. But SSL is a different matter. SSL connections are normally full of random toxic binary data, so almost no SSL-speaking programs will dump out that data on a human when there's a connection failure. So the only way I was able to find these error messages was by running the program under `strace(1)`. ### `CryWrap` took about 110 seconds to negotiate an SSL connection. ### Once I got `CryWrap` to run, my wife Beatrice was still reporting failures getting her mail in Thunderbird. `strace` showed that `CryWrap` was running and receiving data (`less /tmp/crywrap.strace` and then typing `>F` was very helpful to watch this in real time), but it was receiving it very slowly, a few bytes every few seconds. At Paul Visscher's suggestion, I tested the connection myself with the OpenSSL package's `openssl` command: openssl s_client -connect panacea.canonical.org:pop3s This did eventually connect and allow me to speak POP (simulated copy-and-paste here may contain errors): ... Timeout : 300 (sec) Verify return code: 21 (unable to verify the first certificate) --- +OK USER imaptest +OK PASS <censored> +OK QUIT DONE However, it took about a minute and 51 seconds. This is apparently more than Thunderbird's timeout. I don't know enough about SSL to know why this might be. `CryWrap` reported it with these `syslog` messages (wrapped and trimmed for readability): crywrap[27830]: Accepted connection from 190.55.55.32 on 0 to 127.0.0.1/110 crywrap[27830]: Handshake failed: A TLS packet with unexpected length was received. I never did figure out why this happened, and so I gave up on `CryWrap` and switched to stunnel4 (see below). ### `CryWrap`'s `sslwrap` wrapper doesn't support `inetd` ### There is a shell script in /usr/share/crywrap/`sslwrap` that intends to make crywrap act like `sslwrap`, but it doesn't consider the case of trying to run from inetd (-i or --inetd) flag. Because it's a badly-written shell script, it doesn't notice that its "listen port" parameter is missing; it merely tries to invoke `CryWrap` with "-l /" (`CryWrap` uses a slash to separate IP address from port, instead of the traditional colon; in this case, both the IP address and the port are missing, leaving only the lonesome "/", like a girl who's been stood up on a date. `CryWrap` reports this by sending the helpful message: crywrap: Could not resolve address: `/' to the would-be SSL client. I extracted it from an `strace` output file in `/tmp`, except that I had to use `strace -ff` to follow the children of the `/usr/share/crywrap/sslwrap` script. (I guess I could have just redirected `stderr` to a file instead of using `strace`.) ### `CryWrap`'s documented `-v` flag doesn't work as documented. ### `CryWrap`'s man page documents a `-v` flag. `-v 0` is documented to turn off client certificate validation, although having it turned off is documented to be the default. We thought that perhaps the default was actually something other than what it was documented to be, because on the successful `openssl s_client` connections (see above under #3), we were getting this message: crywrap[28190]: Error getting certificate from client: The peer did not send any certificate. And it seemed plausible that this might explain the slowness (#3). So I tried adding `-v 0` to the command line, because the man page says: > `--verify` (`-v`) [LEVEL] > > > Set the level of client certificate verification. Level > > one simply logs the result, level two and above abort if > > the certificate could not be verified. > > Default is 0. If you actually try running crywrap with `-v 0`, you get this error message: kragen at panacea:~$ /usr/sbin/crywrap -l /3802 -d /110 -v 0 crywrap: Too many arguments Try `crywrap --help' or `crywrap --usage' for more information. Except that I didn't originally get the error message at the command line; I had to dig it out of `strace` output in `/tmp` after editing `/etc/inetd.conf` and restarting `inetd`. It turns out that `-v0` is the supported syntax, despite what the man page says, and in violation of the usual Unix conventions. No space is permitted. Success with `stunnel4` ----------------------- I did this: $ sudo apt-get install stunnel4 Then, after skimming the `stunnel` man page, I stuck this in `/etc/inetd.conf` (all on one line) in place of the `crywrap` line: pop3s stream tcp nowait root /usr/sbin/tcpd /usr/bin/stunnel -p /etc/crywrap/server.pem -r 110 That worked. Then I moved `/etc/crywrap/server.pem` to `/etc/stunnel/server.pem` and all was good. The total elapsed time since giving up on `CryWrap` was just under eleven minutes. Things I Learned ---------------- Or was reminded of. 0. It's easy to underestimate how much of a pain in the ass your software will be for other people. Presumably `CryWrap`'s author wouldn't have had any of the above problems (except for #3, and he could have probably diagnosed that one). 1. If I write software and claim it's a "drop-in replacement" for something else, someone is going to be sad. Or pissed off. Because I'll probably forget something. (Although hopefully I'll do better than this!) 2. It's good to be careful about where error messages go. 3. I should try to make sure that my software handles errors (e.g. missing listen port) in a graceful fashion, i.e. by bombing out with an error ("listen port required") instead of proceeding to invoke something else with some broken default (in this case, the empty string) and relying on it to emit a useful error message (``crywrap: Could not resolve address: `/'``). Generally it's pretty easy to make this mistake in shell scripts, but in this case the listen port was explicitly set to the empty string before command-line parsing, as a default, so the problem would have been the same regardless of language. 4. It takes as long to write stuff like this up as it does to experience it. 5. Violating established conventions is likely to cause some frustration; be sure you're doing it for a good reason. By convention `-v 0` is equivalent to `-v0` when `-v` takes an argument; the violation of this convention made the software harder to use. 6. `stunnel` rocks and can do what `sslwrap` did. `CryWrap` sucks and can't. 7. OpenSSL has the `openssl s_client` command, which is like an SSL version of `netcat`, and also `openssl s_server`. These should be very handy for troubleshooting SSL stuff in general. 8. I'm not a great sysadmin, and I tend to be too persistent when I should give up and try something else a little sooner. Credits ------- Thanks to Gergely Nagy for writing `CryWrap`, Jonathan McDowell for maintaining the `sslwrap` Debian package for so long, Rick Kaseguma for writing `sslwrap` in the first place, Beatrice Murch for having the patience to help me test the mail server after the upgrade, Paul Visscher for helping me out with most of the above stuff and also doing a bunch of the work of the Etch upgrade on our machine, and Brett Smith and Jason Cook for doing most of the rest of that work.
