Keith Smiley - Who?

IPSEC/L2TP VPN on a Raspberry Pi running Arch Linux

After you buy a Raspberry Pi, or two, you need to figure out what to use them for. While you'll get a ton of interesting ideas from Googling "uses for a Raspberry Pi," I didn't particularly find them any more than a thought exercise. Making a VPN stood out as an actually useful configuration.

Originally when I got my (accidentally chosen) Model A, I spent a little while going through this guide using Raspbian. That seemed to work fine until I recently purchased a Model B to replace it and couldn't reproduce the configuration. I decided to write the steps that I was finally able to use to get a functional VPN running on Arch Linux.

I started out by following this guide hoping that it would get me a functioning VPN without too much work. Most of this setup will be based on that article with some tweaks for what I had to do to make the settings stick. Unfortunately while it worked after the setup the configuration did not persist after restart. For this configuration, like I said earlier, I wanted to use the ARM version of Arch Linux rather than Raspbian for the install. You can download the Raspberry Pi compatible Arch image from their downloads page. I'm not sure I would recommend Arch for people who haven't installed it before or at least gotten through their Beginners' Guide. The ARM Image, like the normal image, doesn't come with a GUI, perfect for this use of the Pi.

I'm not going to bother with making sure this works before restarting, since that doesn't seem like much of an issue with actual usage (although you can just run the scripts we create and it should work fine). I wouldn't recommend doing much configuration before doing this intial setup. I did this the first time and after an hour of configuration my VPN did not work correctly, I ended up nuking the work I had done and starting over.

Start by installing the necessary components:

pacman -Sy openswan xl2tpd ppp lsof python2

You need to do some configuration of the firewall and redirects:

echo "net.ipv4.ip_forward = 1" |  tee -a /etc/sysctl.conf
echo "net.ipv4.conf.all.accept_redirects = 0" |  tee -a /etc/sysctl.conf
echo "net.ipv4.conf.all.send_redirects = 0" |  tee -a /etc/sysctl.conf
echo "net.ipv4.conf.default.rp_filter = 0" |  tee -a /etc/sysctl.conf
echo "net.ipv4.conf.default.accept_source_route = 0" |  tee -a /etc/sysctl.conf
echo "net.ipv4.conf.default.send_redirects = 0" |  tee -a /etc/sysctl.conf
echo "net.ipv4.icmp_ignore_bogus_error_responses = 1" |  tee -a /etc/sysctl.conf

To make these settings persist we need to create a script that gets launched by systemd each time we restart the system. As recommended in the original article, and being a Homebrew user I created the script in /usr/local/bin/vpn-boot.sh:

#!/usr/bin/bash

iptables --table nat --append POSTROUTING --jump MASQUERADE

for vpn in /proc/sys/net/ipv4/conf/*; do
    echo 0 > $vpn/accept_redirects;
    echo 0 > $vpn/send_redirects;
done

sysctl -p

There are a few things that differ here to the original article. First the hashbang path was changed since the default $PATH on the ARM version of Arch didn't include /bin. I would run which -a bash on your install to make sure this works for you. This obviously doesn't have to be changed, but I think it's better in the long run. I also added sysctl -p since these settings didn't seemed to be applied otherwise. Then you must make this script executable with something like:

chmod +x /usr/local/bin/vpn-boot.sh

Since Arch uses systemd to this script has to be launched by creating a service to be ran through systemd. You can create this file in /etc/systemd/system/vpnboot.service

[Unit]
Description=VPN Settings at boot
After=netctl@eth0.service
Before=openswan.service xl2tpd.service

[Service]
ExecStart=/usr/local/bin/vpn-boot.sh

[Install]
WantedBy=multi-user.target

I added a few things here as well. I wanted to make sure that the boot command would launch after the network settings had been established and before the other VPN software was launched. I'm not sure how many of these changes would be required for systemd to do what I wanted it to but the order really seemed to matter for here. After you create this service enable it within systemd with:

systemctl enable vpnboot.service

I also made some changes to /etc/ipsec.conf (note the comments in the default file for some more info on these settings):

config setup
  dumpdir=/var/run/pluto/
  nat_traversal=yes
  virtual_private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12,%v4:25.0.0.0/8,%v6:fd00::/8,%v6:fe80::/10
  oe=off
  protostack=netkey
  plutoopts="--interface=eth0"

conn L2TP-PSK-noNAT
  authby=secret
  pfs=no
  auto=add
  keyingtries=3
  ikelifetime=8h
  keylife=1h
  type=transport
  # Your server's IP (I used my internal IP, assuming you're using NAT)
  left=172.16.1.1
  leftprotoport=17/1701
  right=%any
  rightprotoport=17/%any
  rightsubnetwithin=0.0.0.0/0
  dpddelay=10
  dpdtimeout=20
  dpdaction=clear

Then for the /etc/ipsec.secrets (use the same server IP address):

%SameIP%  %any: PSK "super random key"

The make systemd start openswan on boot as well:

systemctl enable openswan

I also edited the openswan service file in /etc/systemd/system/multi-user.target.wants/openswan.service

[Unit]
Description=Openswan daemon
After=netctl@eth0.service vpnboot.service
Before=xl2tpd.service

[Service]
Type=forking
ExecStart=/usr/lib/systemd/scripts/ipsec --start
ExecStop=/usr/lib/systemd/scripts/ipsec --stop
ExecReload=/usr/lib/systemd/scripts/ipsec --restart
Restart=always

[Install]
WantedBy=multi-user.target

As you can see I removed the original network dependency and added a new dependency of netctl's default network interface (we haven't enabled this yet).

Next for /etc/xl2tpd/xl2tpd.conf:

[global]
ipsec saref = yes
saref refinfo = 30

[lns default]
ip range = 172.16.1.70-172.16.1.89
local ip = 172.16.1.1
require authentication = yes
ppp debug = yes
pppoptfile = /etc/ppp/options.xl2tpd
length bit = yes
unix authentication = yes

Where local ip is the server's ip and the ip range is the range of IP addresses you want to use for VPN clients. You need to enable this service too with:

systemctl enable xl2tpd

I also edited the systemd file for xl2tpd at /etc/systemd/system/multi-user.target.wants/xl2tpd.service:

[Unit]
Description=Level 2 Tunnel Protocol Daemon (L2TP)
After=syslog.target netctl@eth0.service openswan.service
Requires=openswan.service

[Service]
Type=simple
PIDFile=/run/xl2tpd/xl2tpd.pid
ExecStart=/usr/bin/xl2tpd -D
Restart=on-abort

[Install]
WantedBy=multi-user.target

The other guide also recommends creating the xl2tpd control folder with:

mkdir /var/run/xl2tpd/

Now we need to create/edit /etc/ppp/options.xl2tpd:

ipcp-accept-local
ipcp-accept-remote
ms-dns 8.8.8.8
ms-dns 8.8.4.4
auth
mtu 1200
mru 1000
crtscts
hide-password
modem
name l2tpd
proxyarp
lcp-echo-interval 30
lcp-echo-failure 4
login

/etc/pam.d/ppp:

auth    required        pam_nologin.so
auth    required        pam_unix.so
account required        pam_unix.so
session required        pam_unix.so

And /etc/ppp/pap-secrets:

*       l2tpd           ""              *

If you'd like you can also restrict the users accounts that can access the vpn. This way you can separate your login user from your VPN users who can have much stronger passwords. You'd do that in your /etc/ppp/pap-secrets:

vpnuser   l2tpd         ""              *

To enable the startup of the default netctl eth0 interface you need to run:

netctl enable eth0

You'll probably want to disable any other netctl systemd functions that are enabled by default. Check /etc/systemd/system/mutli-user.target.wants to for other netctl profiles.

So at this point you should be able to enable VPN clients using the super secret keys you enabled before and the username and passwords you've created previously. You can create new users for specifically VPN usage with something like this:

useradd -s /sbin/nologin vpnuser

This disallows users from being able to be used for login which is probably more secure for your VPN (although not required). For testing you can use the root/root defualt user and a less secure key, although you should definitely change these before allowing access to the outside world.

Troubleshooting

Undoubtedly you'll have to deal with something that doesn't work exactly how my setup works. The most useful things to seeing what was happening were these:

netstat -tulpan
systemctl status openswan
systemctl status xl2tpd
journalctl -f

You can glance at some of the other guides to see what should be going on. You probably shouldn't see any red in the openswan status and you should see ports open under pluto with netstat. You can check out the ipsec manpage or the openswan wiki page for a little more information on some of the settings. Also I used this page for some more info on how systemd settings work. Please let me know if there's anything here that could be done easier/better for this configuration.

iTerm theme based on the time of day

One of the great things about Vim's textual configuration is it's ability to contain logic based on outside factors. For the purpose of this post I'm referring to the ability to set your colorscheme based on the time of day with something like this.

Having this functionality in Vim with the Solarized theme at night really made me want this in iTerm as well. Unfortunately iTerm's conifguration doesn't allow anything similar to this. The closest you get is profiles which you can assign keyboard shortcuts to for quickly opening windows with different colorschemes. Luckily, thanks to this pull request two years ago from Piet Jaspers, support was added for scripting iTerm's entire colorscheme with AppleScript. Using these AppleScript bindings I was able to create a script that changes the entire colorscheme of iTerm based on the time of day between Solarized light and dark. As you can see the bulk of this script is just setting different color attributes based on the theme you want. While you could do this conversion by hand to 65535 flavored RGB, I made a tiny Objective-C app to automate the process which is on Github. You can download the signed binary here.

Using this newly created AppleScript I then made a zsh function so that I could call colorize from anywhere to update the color scheme of the current terminal. I also chose to do this at the end of my .zshrc here. This way everytime I open a new session my theme is automatically set.

If you have any input on how I could optimize this let me know.

Global htaccess

When starting a new web project one of the first things I do is download the most up to date HTML5 Boilerplate. It provides a great starting point for the HTML you need in a project. It also comes with an extremely complete .htaccess file. While this is very nice for a single site they recommend you do something different for multiple sites at the very top.

(!) Using .htaccess files slows down Apache, therefore, if you have access to the main server config file (usually called httpd.conf), you should add this logic there: http://httpd.apache.org/docs/current/howto/htaccess.html.

This got me to their awesome collection of server configs which has their, and in many ways the communities, recommended settings depending on your webserver. The apache configs have the same .htaccess file so I decided to dig into how to do this.

They direct you to the apache article about using .htaccess files which has a similar comment about their use.

You should avoid using .htaccess files completely if you have access to httpd main server config file Using .htaccess files slows down your Apache http server. Any directive that you can include in a .htaccess file is better set in a Directory block, as it will have the same effect with better performance.

So I decided to set this up on my Linode VPS which is running Ubuntu 10.04. As stated in the original file comment they recommend using the httpd.conf file for your custom configuration like this. But apparently that file could be overwritten on updates of Apache which would be pretty annoying. Luckily the default Apache config file (apache2.conf on 10.04) includes the contents of the conf.d folder which is in the same location. By creating a foo.conf file in that directory Apache should immediately load its contents. As mentioned in the comment from the Apache site the custom configuration needs to be wrapped in a Directory block. The block expects you to provide a path to the files you want to be affected by the contained configuration. Since I wanted this to work for all the sites being served by Apache I simply used /srv/www/*/ which includes my entire sites directory.

Besides the speed increased gained by using a global .htaccess file this allows you to have much shorter custom files for site specific configuration. For example only required configuration for one of my sites was the ErrorDocuments. Now my .htaccess file went from 300+ lines to

ErrorDocument 403 /403.php
ErrorDocument 404 /404.php
ErrorDocument 500 /500.php

OS X Crash Report Symbolication

As you may know I write a small OS X called Sail. Over the past few months that it has been available I've received a few crash reports about an issue I wasn't able to reproduce. Today I decided I wanted to dive into them and see if I could at least figure out the root of the issue and fix it with my next release.

This lead me down the rabbit hole of symbolication, something I personally hadn't dealt with myself before (since Crashlytics does it for you). I was hoping I would be able to find something around the internet about this, unfortunately what I mostly came up with was a lot of iOS related answers that didn't seem to work the same way and two links to Apple documentation that have been removed. Other than the process for symbolicating reports for OS X apps seems to be different than iOS apps which there is plenty of documentation for (I'm not bitter). Daniel Jalkut has a post about these but his exact method didn't seem to work for me.

Here is what did work for me. For my first abridged crash report I had this

Process:         Sail [35072]
Path:            /Applications/Sail.app/Contents/MacOS/Sail
Load Address:    0x106823000
Identifier:      com.keithsmiley.SailOSX
Version:         4 (1.2.0)
Code Type:       x86_64 (Native)
Parent Process:  launchd [207]

Date/Time:       2013-07-19 16:09:24.097 +0200
OS Version:      Mac OS X 10.8.4 (12E55)
Report Version:  8

Thread 0:
13  Accounts                        0x00007fff839fd1b1 -[ACAccountStore accountTypeWithAccountTypeIdentifier:] + 230
14  Sail                            0x00000001068308f7
15  Sail                            0x0000000106830798
16  Sail                            0x0000000106825249
17  CoreFoundation                  0x00007fff82465eda _CFXNotificationPost + 2554
18  Foundation                      0x00007fff8611b7b6 -[NSNotificationCenter postNotificationName:object:userInfo:] + 64
31  AppKit                          0x00007fff812cc1a3 -[NSApplication run] + 517
32  AppKit                          0x00007fff81270bd6 NSApplicationMain + 869
33  libdyld.dylib                   0x00007fff869d07e1 start + 0

Binary Images:
  0x106823000 - 0x106896fff  com.keithsmiley.SailOSX (1.2.0 - 4) <D1F313B6-21F6-341B-8627-5480C5D1DB20> /Applications/Sail.app/Contents/MacOS/Sail

Just glancing at this crash report it's not too difficult to understand a bit about what was going on. A notification was sent, some methods were called in my application and then accountTypeWithAccountTypeIdentifier was called. Based on the small number of times I call that method I was quickly able to assume where the issue was but I still wanted to see exactly what methods of mine were being called first.

This brings me to atos the command line tool Apple provides to symbolicate these reports. This is where my experience differs with most of what I found online. My usage looked like this:

atos -arch x86_64 -o Sail.app.dSYM/Contents/Resources/DWARF/Sail -l 0x106823000

This uses my dSYM file that was generated with the archive build I submitted to the app store along with the knowledge that it was running on an x86_64 architecture and the most important part, for me, the load address.

To find my dSYM file that was generated when I did my archive build I simply noted the version and build number from the crash report, went to the Archives tab in the Xcode organizer, found the build with the same number, right clicked and clicked "Show in Finder." This takes you directly to the .xcarchive file on disk which you can right click and click "Show Package Contents." From there I copied my dSYM to the desktop so I didn't overwrite anything unintentionally.

The load address is the starting memory address of your application. The tool uses this address as an offset to find the correct methods in your symbols. In the above crash report Load Address is a provided field. This was the only report I saw that had that, typically I needed to look under the Binary Images section for the address range of my application. In this example it was 0x106823000 - 0x106896fff.

The atos command then provides an interactive prompt where you can paste addresses into the stdin and it will tell you the corresponding methods. Mine looked like this:

0x00000001068308f7
-[KSAccountsPreferences dealloc] (in Sail) (KSAccountsPreferences.m:77)
0x0000000106830798
-[KSAccountsPreferences viewDidLoad] (in Sail) (KSAccountsPreferences.m:73)
0x0000000106825249
-[KSAppDelegate openAboutWindow:] (in Sail) (KSAppDelegate.m:59)

Here I can see that the notification that was being posted was probably an NSApplicationDidFinishLaunchingNotification starting off some methods in my app delegate. I then load the accounts preferences, which would make sense to call the ACAccountStore method, but then dealloc is called. Seeing this was an immediate red flag since KSAccountsPreferences should be retained since it provides information about available accounts to the rest of the application.

I had another crash report from a different issue that was a little bit harder to parse without symbolicating the methods.

Process:         Sail [47027]
Identifier:      com.keithsmiley.SailOSX
Version:         1.2.0 (4)
Code Type:       X86-64 (Native)
Parent Process:  launchd [45696]
User ID:         502

Application Specific Information:
*** Terminating app due to uncaught exception 'NSRangeException', reason: '*** -[__NSArrayM objectAtIndex:]: index 0 beyond bounds for empty array'
abort() called
terminate called throwing an exception

Application Specific Backtrace 1:
2   CoreFoundation                      0x000000010c5008ec -[__NSArrayM objectAtIndex:] + 252
3   Sail                                0x000000010b838eed Sail + 61165
4   Sail                                0x000000010b838f36 Sail + 61238
5   Sail                                0x000000010b838b19 Sail + 60185
6   Sail                                0x000000010b83785d Sail + 55389
7   libdispatch.dylib                   0x000000010fa07f01 _dispatch_call_block_and_release + 15

Binary Images:
  0x10b82a000 - 0x10b89dff7 +com.keithsmiley.SailOSX (1.2.0 - 4) <47EC2733-B543-31EA-A6AA-9D998FB65803>

Obviously this was caused by an invalid access to an array but that's a little harder to track down. So I again used atos with the dSYM and new memory location.

atos -arch x86_64 -o Sail.app.dSYM/Contents/Resources/DWARF/Sail -l 0x10b82a000

I got this output for my memory addresses

0x000000010b838eed
-[KSAccountsPreferences selectedADNUser] (in Sail) (KSAccountsPreferences.m:238)
0x000000010b838f36
-[KSAccountsPreferences selectTwitterUsername] (in Sail) (KSAccountsPreferences.m:243)
0x000000010b838b19
-[KSAccountsPreferences populateTwitterAccounts] (in Sail) (KSAccountsPreferences.m:211)
0x000000010b83785d
__36-[KSAccountsPreferences viewDidLoad]_block_invoke (in Sail) (KSAccountsPreferences.m:70)

This ended up giving me the exact line where out of bounds issue was happening depending on a certain number of accounts. I noticed that this issue had been fixed since my last release so I did a diff on the tag I created for that specific release with

git difftool HEAD..1.2.0\(3\)

Then in Kaleidoscope I was able to figure out what changed had fixed the issue. Just because it's so pretty here is what it looked like.

Diff

Symbolicating crash reports is definitely vital to tracking down bugs your users are experiencing when you can't reproduce them yourself. Once you figure out how it's obviously worth it.

The 'Best' Text Editor

I'm tired of people asking about the 'best' IDE for xyz purpose. The answer to this question is there is no best. The answer is always 'it depends.' Not only does it depend on what you're doing but more importantly it depends on you. It depends on your work flow. It depends tons of other indiscernible factors.

It seems like people think they work in exactly the same way as enough other people. That asking this question will yield a useful result. The truth is that there are far fewer text editors than people who need text editors so it's impossible not to overlap with someone. We misconstrue this overlap in thinking that now this person knows exactly what we want. In reality they just happen to share some arbitrary subset of the way we work and therefore ended up with the same text editor.

So how can you decide which editor is best for you? Try them. This sounds obvious to you? Good, this article is not for you and you can safely leave now. These days text editors are either free, cheap or have trials. So download them all try them out and see if they make sense to you. Weed out the ones you really hate or the ones that crash and spend a little more time with the remaining editors. Some, like Vim, you may have to spend a little more time with to grasp but this still doesn't seem like a high order.

But please stop asking questions on StackOverflow and similar sites where you expect people to throw their vote into the hat for the 'best' editor and make a decision for yourself.