prerequisite concepts: prelude, basic configuration

This post is as much about customizing the root shell as it is about SSH environment variables, but I’m adding this to my OpenSSH collection because it’s applicable to any user.

I occasionally work on servers where, for a variety of reasons, I share an account with one or more other users; this is almost always suboptimal, but it does happen nonetheless. Over time I’ve grown somewhat partial to zShell, so one method I’ve used is to log in to a default shell, usually bash, then run zsh.

Even on server’s where I am the sole administrator, I usually don’t change the default shell– not so much because of days gone by when doing such a thing would break boot scripts & such, but because I try to practice the good habit of logging in as a normal user and using sudo for escalated privileges.

Eventually I struck upon the idea to have some code in the shell init script (e.g. $HOME/.bash_profile) switch to the shell of my choosing automatically as I log in. What I came up with looks something like this:

if [ -n "$CDFX_SHELL" ]; then
    tty -s && exec $CDFX_SHELL
fi

Briefly this code says, “if $CDFX_SHELL isn’t empty and the tty program says we’re connected to a terminal (on STDIN), then replace this shell by running the command in $CDFX_SHELL without creating a new process.”

Readers familiar with shell initialization may recognize potentially unnecessary checks in this example but this avoids having to delve into the differences between shell sessions which are interactive, login, both, or neither, as well as how this relates to scp and rsync. Also noteworthy are the checks which should be in the code before it’s used on a production server, such as verifying that $CDFX_SHELL specifies a valid shell. This isn’t intended to be cut-n-paste code.

Two steps are necessary for this to work: obviously $CDFX_SHELL must be set in the local environment for SSH to have anything to pass to the server, but less obviously the server must be configured to allow this variable to be set. This can be configured in the sshd config file (e.g. /etc/ssh/sshd_config) by adding this line:

AcceptEnv CDFX_SHELL

I prefer this method over those which require enabling PermitUserEnvironment because it’s less prone to unintended side-effects, as noted in the man page. In addition to (or in lieu of) using exec to switch shells, this method could also be used to set custom environments in the same shell, for different users or the same user, anytime it’s useful to have login behavior change based upon variables set in the SSH client.

prerequisite concepts: prelude, basic configuration, port forwarding

Network address translation (NAT) is a very common method of providing secure access to hosts on a private network. Given the limited amount of IPv4 addresses, computer networks with relatively few, very few, and even a single public IP address are common. A typical small business customer of my consulting practice has one or more Linux servers on an office network protected by a firewall. The following is a close look at Example Industries, the theoretical owners of example.com; this customer receives support for two Linux servers, a mail server and a PBX, but only one public IP address between them. Through NAT, public services (namely mail and VoIP) on both servers are accessible via example.com. This works well for inbound mail and phone calls, which only need to access one or the other host, but SSH access is the lifeblood of remote system administration, and there’s the rub– when I enter ssh example.com I land at the mail server. SSH access to the PBX would seemingly threaten to litter my command line with unsightly extra characters, if not subsequent commands outright.

My carpals are tunneled enough, I don’t want to type more than ssh mail and ssh pbx to access these servers, and while I’m at it I want to have scripted log-ins as well– securely, not those namby-pamby no-password keys. In fact, I don’t even want to have private keys on either server.

Fear not! With the power of OpenSSH, I can fix this.

Recipes for SSH proxies are like homespun cure-alls: few do much good and some are actually harmful. As indicated in my introductory rant, I have a few criteria for this sort of thing:

• No interactive passwords.
• No password-less keys.
• No private keys on servers.
• Avoid command line options.
• Demur scripts.

Previous installments in this series have intentionally focused on declarations which will help define my Example Industries SSH client configuration; here’s the file so far:
# /home/garrison/.ssh/config
 
# Global Options
Host *
ForwardAgent yes
 
Host mail
HostName example.com
 
Host pbx
HostName example.com
LocalForward 8080 localhost:80
LocalForward 3306 localhost:3306

At this point ssh mail and ssh pbx are functionally equivalent in that both land me on the mail server, and I want forwarded access to pbx’s web configuration and database but I end up with webmail and mail user database (which I can access without port forwarding). While I can certainly connect to pbx once I’m in mail, I must specify it’s internal address (ssh 192.168.1.20) and the services I require will not be forwarded to my workstation. What I need is a way to tell SSH to bypass mail and connect me directly to pbx.

Rubbish. In life, particularly in technology, we often confound our challenges by mischaracterizing the requirements of a solution as I just have. It is no coincidence that my most successful customers are also the ones who make use of my ability to solve problems, rather than simply implementing solutions. Often the most challenging part of a solution is correctly stating the problem. What I really need is just for ssh pbx to connect me to pbx and forward my ports; I don’t really care whether mail is bypassed or not, so long as it stays out of my way.

As it turns out, the solution I favor does not bypass the gateway host (mail) at all, but uses it as a proxy for my connection to pbx. With the addition of a ProxyCommand directive to my SSH config, I can achieve all my objectives.

The ProxyCommand directive is a subtle beast, and my early attempts to use it were unsuccessful. At the time I was doing something similar on the command line: ssh -t example.com ssh 192.168.1.20 Because I initially hoped to “do the same thing in the config file” I mistakenly assumed that ProxyCommand would allow my to connect to mail and immediately fire off a connection to pbx; LocalCommand behaves this way but doesn’t allow me to accomplish what I can with ProxyCommand.
After a few unsuccessful syntax variations, I began to suspect that I might have the wrong idea about this directive.

When I eventually sorted out the correct syntax, I knew I had the wrong notion because I had no clue why one version worked and the others did not. Richard Silverman, one of the authors of the snail book was kind enough to set me straight. He explained:

ProxyCommand specifies a program which the SSH client will use to contact the remote SSH server. Instead of opening a TCP connection, it runs this program and uses its stdin/stdout as the communications channel.

I then understood that with ProxyCommand in play, SSH expects the command it executes to provide the TCP connection between mail and pbx; netcat, a phenomenally useful tool, was designed for just this sort of task:

ProxyCommand ssh example.com nc -v %h %p

Adding this directive to the Host pbx section of my config gets the whole proxy business out of my way and I can connect with just ssh pbx; tho forwarded HTTP and MySQL connections are just the beginning. I can use scp, sftp, FuseSSH, sshfs or anything built on SSH just as if pbx had a public IP. One more example:

rsync -Hav pbx:/usr/stuff backup:/archive

I often use such a command to transfer data from a machine with no public IP address to a backup server which also has no public IP and lives on another private network in a different town, state, or country. This is all done with ProxyCommand directives, over secure SSH connections, and most importantly, with no special command line syntax. What could be easier?

prerequisite concepts: prelude, basic configuration

Port forwarding is a versatile feature which informs several popular concepts, including X Forwarding and tunneling which are briefly explained below; more advanced port magic will be addressed elsewhere.

X Forwarding
At the end of the previous installment of this series is an example SSH client configuration file, usually located at ~/.ssh/conf; a more complete description of the global declarations shown was deferred until this section, where they are more relevant.
# global declarations
This line is a comment and while it is ignored by SSH, it is very helpful to use comments and white space to maximize readability, and maintainability.
Host *As previously stated, Host declarations define the context of all following lines until the next Host declaration. Because the ‘*’ will match any host, these declarations will be applied to all hosts– unless overridden by a later declaration or a command line option.
ForwardAgent yesThe key agent allows a user to store one or more authenticated keys in memory, enabling dual key authentication; this is sometimes (incorrectly) referred to as passwordless authentication, but a password check is still required to load the key into the agent. Agent forwarding allows one not only to rely on keys instead of passwords to connect to a remote server, but to use the same agent to connect to a third host and so on as long as the keys are recognized and ForwardAgent is enabled on each host.
ForwardX11 yesThis declaration turns on port forwarding for X Windows authentication; briefly put, this allows an X session (i.e. the active GUI environment) on a Linux workstation to interact with X Windows on a remote host using an encrypted tunnel. The practical benefit is that programs on a remote host needing or allowing a graphic interface may use one provided by the client. The canonical example is that I run xclock on a remote server and the clock appears on my screen.

Tunneling
X forwarding may be fun, but port forwarding has many more practical uses. Suppose you need to connect to a web application on, perhaps FreePBX to configure a Linux PBX, but you cannot connect directly because port 80 is blocked– there are a few reasons why this might occur but port forwarding can work equally well in all likely scenarios.

If you log in to the remote host with ssh -L 8080:localhost:80 joe.telco@pbx.example.com SSH will create an encrypted tunnel between port 8080 on your workstation and port 80 on the server, ergo you can now get to the web app by pointing your browser to http://localhost:8080. Because the connection is tunneled through SSH, it works even if port 80 is restricted by IP address or an interceding router, firewall, or cable modem; because of SSH’s strong encryption, this technique also provides a secure connection to servers which do not offer SSL.

Many users mistakenly assume that the localhost in the preceding command is the one referenced in the subsequent URL; in fact, the given example is connecting (client) localhost:8080 to (server) localhost:80 and could be entered as:
ssh -L localhost:8080:pbx.example.com:80 joe.telco@pb»
The notable concept is that the latter host:port pair is evaluated on the remote host, meaning it’s a bit like saying:

ssh -L localhost:8080:localhost:80 joe.telco@pbx.exam»
ssh -L 192.168.1.7:8080:localhost:80 joe.telco@pbx.ex»
ssh -L *:8080:localhost:80 joe.telco@pbx.example.com
The explicit use of localhost in the first of the preceding examples restricts listening port 8080 for local use only; in contrast, the second example binds the port to a specific network interface, and other users on the network may use the forwarded port at the specified IP address; the last example avails the port on all interfaces. If the bind address is not specified, the port is bound to the loopback address unless GatewayPorts is enabled, wherein the wildcard address issued.

The config file is the place for complexity, and the following example specifies that connections to pbx.example.com should be made as user joe.telco, and forwarded HTTP and MySQL connections should be available on all interfaces using the specified ports.
Host pbx
HostName pbx.example.com
GatewayPorts yes
User joe.telco
LocalForward 8080 localhost:80
LocalForward 3306 localhost:3306
As previously illustrated, this simplifies the command line syntax such that in lieu of:
ssh -g -u joe.telco -L 8080:localhost:80 \
-L 3306:localhost:3306 pbx.example.comone need only enter:
ssh pbxMuch more information on the port forwarding capabilities of SSH are available in the man pages as well as previously cited sources; however, the examples here lay the foundation for the next installment of this OpenSSH series: Proxy Connections.

Having discovered the advantages of á la carte VoIP pricing, I pondered how to extrapolate my experience for general discussion while avoiding the pitfalls of interpolation and abridgement. The Reference Book of Rates, Price Indices, and Household Expenditures for Telephone Service published by the FCC’s Wireline Competition Bureau provides a rough estimate of wireline telephone expenses averaging $45 per month in 2007, based on market research by TNS Telecoms. This isn’t too far from my own experience with residential VoIP plans which have tended to average about $35 monthly, including additional fees and charges, which can be significant: on BroadVoice’s “Unlimited World” plan, for example, “Taxes & Surcharges” account for about 35% of the monthly total. Based on these data, I use an estimated $35-$45 for generic comparison of monthly residential phone bills, or an average average of $40. As I designed our current, á la carte plan, I surmised that after discounting business use, the residential remainder was unlikely to ever exceed $30 in a single month. As the plan took shape, however, I realized that intelligent planning could lower that even further; somewhere in the neighborhood of a $20 monthly average would certainly exemplify what custom VoIP plans can offer, and half the average isn’t a bad talking point.

Though less obvious, another great feature of á la carte or “on demand” plans is scalability, if I suddenly find myself needing to host frequent call-in conference calls between a customer, their overseas manufacturing division, regional sales reps, and myself, the only change I’ll see on my invoices will be in usage. I am not aware of any “unlimited” residential plans which offer unlimited channels (simultaneous callers). With currently just three phone numbers, my setup is small enough, and with just enough complexity to provide a good example.

I use one number for my consulting, which has separate extensions, voice mail, etc.; I have a fax number for the luddite crowd, and a home number associated with a family voice mail, options for the caller to forward the call to my wife’s or my mobile phone, and a ring group which includes a line in my office. I’ll use an even usage split for comparison; for although Codefix Consulting has its own phone number, those who know me well tend to call my home number rather than risk my having a life outside of work.

My primary VoIP provider is VoiceMeUp.com and I have two DIDs (phone numbers) ($4.95 ea) and a prepaid, on-demand plan which bills 30/6 at $0.0185/min. My backup provider is CallWithUs.org who bill $0.0125 in whole minutes; while I hadn’t originally intended to acquire a DID through CallWithUs.org, I found one for $6/mo which includes 3000 free inbound minutes and couldn’t pass it up. My base VoIP price is therefore 4.95 * 2 + 6 = $15.90 plus usage, or $7.95 on an even split. Theoretically this leaves me with just over 650 minutes before exceeding my $20 target, but this doesn’t account for incremental billing, free VoIP to VoIP calls, and other variables which impinge cost.

It’s now more than a month since I dumped Broadvoice, ergo September’s charges and complete usage data are available for a real world comparison against a $40 average, a $35 example, and a $20 target. As it turned out we made no outbound calls on the (secondary) CallWithUs.com trunk and didn’t exceed the 3000 inbound minutes, so all billable usage was on the VoiceMeUp.com trunk which makes accounting easier. September’s total was 9.95 + 6 + 23.38 = $39.28 or $19.64 per split which helps validate my “less than $20 phone bill” theory. Our total usage was 36h 7m 33s (2167.55 min) or nearly 1,100 “home” minutes and more than 2,000 unused inbound minutes– how much do you talk?

prerequisite concepts: prelude

If you’re not already using a config file (~/.ssh/config) you should peruse the documentation to see what it offers; an ongoing benefit I enjoy is that it allows me to accomplish more while typing less. Suppose, for example, you need to access two mail servers which are both behind a firewall and sharing a single public IP address. One server uses NAT (port forwarding) to provide user access via IMAP-SSL, POP3-SSL, and perhaps even webmail, all on default ports; similarly SSH can be accessed on port 22. The other server happens to be a mail relay, which handles all of the spam and virus scanning for inbound and outbound mail; while the SMTP, SMTPS, and submission services all enjoy a NAT configuration using default ports, SSH access is on port 23 because port 22 already forwards to the IMAP server and the sysadmin hasn’t read this series of articles.

As an added bonus, your accounts have usernames which differ from each other (let’s use “fred” and “barney”) as well as from your workstation. To log in to these machines using the command line, you would type:

ssh fred@example.com
ssh -p 23 barney@example.com

This isn’t a great deal of typing but already one can see that differentiating more complex connections may be confusing when distinguished only by the port used. We can clarify things a bit with a config file like this:

Host imap
HostName example.com
User fred
Host smtp
HostName example.com
Port 23
User barney

Now our SSH commands look nicer:

ssh imap
ssh smtp

The config file can always be overridden with command line options, so ssh admin@smtp will log in as admin rather than barney, but will still use port 23 and any other options set in ~/.ssh/config. Once you start using LocalForward and ProxyCommand command line options quickly become tedious and unwieldy, even if you can remember all options for every host you attend.

One final note, in addition to acting as a convenient alias, the host keywords may also be used to make declarations for groups of servers, or all servers, by using wildcards and pattern-lists. The ssh_config man page (or your preferred documentation) has a detailed PATTERNS section, but a below is a brief example to whet your appetite:

# global declarations
  Host *
  ForwardAgent yes
  ForwardX11 yes

# just for example.com servers
  Host *.example.com
  ServerAliveInterval 60
  StrictHostKeyChecking no

This is a prelude to a series of articles focused on how the sophisticated power of OpenSSH may be harnessed to great advantage with less effort than one might think. Readers already familiar with OpenSSH and passwordless authentication may wish to skip ahead:

OpenSSH: Basic Configuration
OpenSSH: Port Forwarding
OpenSSH: Proxy Connections
OpenSSH: Environmental Override
Planned: Reverse Connections
Planned: Connection Multiplexing

I can add little to the vast collection of SSH HowTo’s already posted elsewhere^[1], a quick Google search will usually yield plentiful information specific even to operating systems, distributions, and software versions. The best inclusive reference is Barrett & Silverman’s snail book, and I would especially like to thank Richard E. Silverman for helping me grasp the subtleties of ProxyCommand. For similar reasons, I will not delve deeply into the merits of the various SSH authentication mechanisms; however, some general remarks may help set the stage for this series.

It is likely that if you are using SSH keys on a Linux desktop, your key agent is well integrated with your desktop environment by default, and does an excellent job making SSH2 authentication as unobtrusive as possible; even oppressed Windows users can easily install PuTTY and Pageant to the same end, although I can’t say whether all the techniques illustrated will work identically with Windows as I don’t touch the stuff myself.

1. SSH, Encrypted Keys, and Cron is a great example. ^

AppArmor, introduced to Ubuntu with Gutsy, is yet another security tool unleashed upon the infosphere. In part, AppArmor is intended as an alternative to SELinux, which can easily be seen as daunting to configure; unfortunately, many such projects are daunting for those admins forced to walk the plank of unfamiliarity above a sea of expectations. Despite a troubled history, the project seems to be here to stay so it is likely only a matter of time before audit messages crop up in one’s kernel log. For those who find AppArmor unnecessary, unpalatable, or just untimely, herein lies a quick-and-dirty guide for telling AppArmor where to stick its audit complaints.

Ubuntu’s community documentation has some basic commands for starting, stopping, disabling, and enabling AppArmor, but if we want to know as little as possible about AppArmor then it’s not unlikely that we’re just trying to dispense with a plethora of audit complaints in our kernel log. The first time this happened to me, it was on a box which had some MySQL data in a nonstandard location, resulting in a flood of log entries similar to:

Sep 23 11:22:17 bluebeard kernel: [4960023.353512]
audit(1222183337.704:68500): type=1502
operation="inode_permission" requested_mask="r::"
denied_mask="r::" name="/u1/mysql/"
pid=1573 profile="/usr/sbin/mysqld"
namespace="default"

To allow mysqld to do its thing in /u1/mysql without sending AppArmor into a conniption fit, just add permissions to its profile, located by default at /etc/apparmor.d/usr.sbin.mysqld on Ubuntu Hardy systems.

# custom permissions
/u1/mysql/ r,
/u1/mysql/** rwk,

The first line is a comment which makes clear that the lines which follow are not default permissions. The second line grants read access to /u1/mysql/, which AppArmor expects to be a directory due to the trailing slash. The final line uses the /** syntax to specify a group of files and subdirectories, to which are granted access to read, write, and perform locking operations. Note that these rules were simply adapted from the default rules for /var/lib/mysql; such “cut-and-paste” adaptation helps avoid typos and lessens the need for a detailed understanding of the configuration syntax. Note further that this is an observation rather than a recommendation.

Once all profiles have been edited to perfection, reload AppArmor and we’re off to the races.

sudo /etc/init.d/apparmor reload

When this information proves inadequate, complete documentation and additional resources can be found at OpenSuse.

Verizon is a great company, doing great things, but that doesn’t mean they’re not evil. I’ve found that this is an effective maxim which allows me to extol the virtues of Verizon without sounding like I’m drinking the kool-aid. Today I’m hoping it works inversely as well.

If you subscribe to Verizon FIOS broadband and television, then you have an Optical Network Terminal mounted to an outside wall of your home, from which sprout a coaxial cable and an ethernet cable which connect the ONT to a router inside your home, such as the ActionTec MI424-WR.

Verizon technicians will insist that the router they supply must be connected directly to the ONT for your service to function correctly.

They are lying.

The ActionTek router they use isn’t bad, but it pales in comparison to the wireless gigabit router I’ve customized with DD-WRT firmware; however, what really burns my toast is when some call center drone (“tech support” is a double misnomer) tells me that their service requires me to reconfigure my network to be less robust, slower, and less secure.

There is absolutely, unequivocally no fathomable reason to use two routers, unless, of course, one has a good reason. In truth, there are legitimate reasons to use multiple routers, firewalls, and access points but technological ignorance is not among them. The arrogant superiorism of the misinformed miscreant who tried to sell me this snake oil only makes me wish he would be boiled in his own vomit and bile.

I feel better now.

Welcoming a new Verizon router into your network can be an easy, painless process. I make the following recommendations under the assumption that you require an already existing router to preside over the Verizon router and either have both routers configured on different subnets or don’t need to. You should also heed that as I am not privy to the Mysterious Ways of Verizon, this text may contain factual errors, your mileage may vary, and should you break something you should neither find me culpable nor burn my effigy. What would you want with my effigy anyway?

Because I initially set up the Verizon router in accordance with the lies they told me, subnet conflicts were an issue resolved long ago and I had also disabled it’s wireless transmitter. To make my life easier, I also enabled remote (WAN) administration and assigned a static IP address so I can access the router from my local network, which is a public network from the Verizon router’s point of view.

If you leave the Verizon router connected only via the coaxial cable, you’ll soon discover that it needs internet access to retrieve channel and schedule information for your television. To fix this you need to plug an ethernet cable into a LAN port on your router and the WAN port on the Verizon router; you may also need to forward port 4567 to the Verizon router but as I write this I have yet to try disabling it or complete my research on how Verizon uses this port.

Shortly after I last upgraded my mail server, one user reported that his mail client was failing to connect with the message:

"Unable to connect to your IMAP server. You may have exceeded the maximum number of connections to this server..."

He was the only one known to be having this issue, so after a cursory check of the server with no obvious problems, I suggested that this might be an error on his end, such as connecting to the secure IMAP port without using SSL/TLS. Occam’s Razor suggests that a server error is more likely than a client error which just happens to coincide with a server upgrade, so I eventually decided to dig up some infrequently used commands and perform a thorough analysis.

Testing proved that there was indeed a server problem with certain SSL connections, while others worked every time. Deft Googling revealed that the imapd-ssl config file shipping with Gutsy Gibbon had TLS_PROTOCOL=SSL3, whereas it should be TLS_PROTOCOL=SSL23. The user who first reported the error is tech-savvy, so I sent him the commands I used to diagnose the problem and promised to eventually write this post. Below are the commands which are now my first step in diagnosing mail connection problems.

# Test secure SMTP
openssl s_client -connect example.com:25 -starttls smtp -showcerts
#
# Test secure IMAP
openssl s_client -ssl2 -connect example.com:993 -showcerts
openssl s_client -ssl3 -connect example.com:993 -showcerts
openssl s_client -tls1 -connect example.com:993 -showcerts

To test secure POP, simply substitute 995 for 993 in the above commands, likewise if you run secure SMTP on a port other than 25, you will need to alter the first command.

I have resisted the urge to display caller id on my MythTV as somewhat obvious. I’m always looking for ways to demonstrate the freedom which comes from using open source software, but I prefer the zesty freshness of an original idea rather than anything that’s been done, redone, and done again. My wife, however, thought that Myth caller id sounded like a great idea and asked me to set it up. What follows is how I did this with the least possible effort.

I already have a couple Custom-Apps which handle direct inward dialing on my inbound routes, so sending caller id to my MythTV box was as simple as adding one line to each Custom-App in extensions_custom.conf:

exten => s,4,System(/bin/echo "Caller: ${CALLERID(name)} ${CALLERID(number)}" | /usr/bin/nc -w 3 myth.example.com 1234)

On the MythTV side I decided to use xosd rather than mythosd because the latter will only work when you are watching live TV or recordings. I created a simple wrapper script called osd.sh:

#!/bin/sh
DISPLAY=:0.0
FONT="-xos4-terminus-*-*-*-*-32-*-*-*-*-*-*-*"
osd_cat --font=$FONT --align=centre --shadow=2 --color=SeaGreen –delay=15


At this point I have the PBX sending caller id to MythTV and osd_cat to display the data on screen, but I need a simple way to catch the data on port 1234 and feed it to my osd.sh script. I did this with one command:

micro_inetd 1234 ./osd.sh &

Micro_inetd is a nice replacement for inetd, xinetd, etc. if, as here, only one service needs to be managed. I set this to always start with xfce4-autostart-editor which naturally only works with the Xfce 4 desktop environment. Gnome, KDE, and other desktops have their own session managers.

Of course, now that we have caller id on the TV, I’ll need to create a caller id lookup to compensate for the wretched state of telco support; fortunately I can put that off until next year

Update: change osd.sh to format phone numbers

#!/bin/sh
DISPLAY=:0.0
FONT="-xos4-terminus-*-*-*-*-32-*-*-*-*-*-*-*"
cid=`cat | perl -pe 's/(1?\d{3})(\d{3})(\d{4})/$1.$2.$3/'`
echo "$cid" | osd_cat --font=$FONT --align=centre --shadow=2 \
--color=SeaGreen --delay=15