Bypassing a VPN Using Cgroups

Posted on Thu 30 June 2022 in Hackery

Article summary: useRalativeImagePath

Changes

2022-08-27

Fixed small bug in /usr/local/bin/if-up-novpn script

2022-08-10

Added ~/bin/bash-novpn script example.
Made /usr/local/bin/if-up-novpn script more robust

Setting up a VPN bypass for applications

You use a VPN and all your traffic gets routed through it. But what if you want some traffic to bypass it?

In the past I've just used SSH to tunnel traffic to another machine acting as a SOCKS proxy. Then it's just a matter of setting the application's proxy to point to it. Example:

$ ssh -D 8080 me@kids-linux-machine

Now there is a SOCK5 proxy at socks://localhost:8080 as long as that SSH connection is up. Set your application's proxy settings to this and it works great. For some applications. And only if they support using a proxy.

But what if the application is complex? Maybe it uses a lot of non HTTP-based protocols, uses UDP, runs other applications all which you need to bypass the VPN?

In this tutorial I'll be using the Control Groups(cgroups) kernel feature to control which routes an application's network data uses.

Assumptions

This tutorial assumes:

you are using a recent version of Debian. I'm using Debian Bullseye.
You are running all the commands as root.
You are using a VPN like openVPN. Others should work also but they have not been tested.

Routing table

Create a routing table by creating a file named /etc/iproute2/rt_tables.d/novpn-route.conf containing:

  11 novpn

Like this:

  echo  11 novpn | sudo tee /etc/iproute2/rt_tables.d/novpn-route.conf

Add routes and policy to the routing table

You will need to find a place to add two commands for creating the bypass route. Where you do this depends on your particular setup.

ip rule add fwmark 11 table novpn
ip route add default via YOUR_GATEWAY_IP table novpn

The first one routes all packets marked with "11" to the "novpn" routing table. The second line does the actual routing.

In order to do this automatically I created the following script and put it in /etc/network/if-up.d:

/usr/local/bin/if-up-novpn:

#!/bin/bash

gw=`ip rou list default | perl -ne "/^default via (\S+) dev $IFACE / && print \\$1"`

test -n "$IFACE" || exit 0
test -d /sys/class/net/$IFACE || exit 0
test  "`cat /sys/class/net/$IFACE/type`" -eq 1  ||  exit 0
# ignoring all but ethernet and wireless interfaces
test -d /sys/class/net/$IFACE/wireless -o -L /sys/class/net/$IFACE/phy80211 || exit 0

ip rule flush table novpn
ip route flush table novpn

if [ -z "`ip rule show table novpn`" ] ; then
  ip rule add fwmark 11 table novpn
fi
if [ -z "`ip rou show default table novpn`" ] ; then
  IP=`ip -4 route get 1.1.1.1 dev $IFACE | grep '^[1-9]' | cut -f7 -d' ' `
  NET=`ip addr show dev $IFACE|grep -P '^\s+inet '|cut -f6 -d' '`
  ip route replace default via $gw src $IP table novpn
  ip route add dev $IFACE `ip rou show dev $IFACE $NET` table novpn
fi

Notice that a route also needs to be added for the local LAN. My novpn routing table looks like this:

default via 192.168.1.1 dev wlan0 src 192.168.1.100 
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.100 metric 600

Mark packets

Then we need to mark all packets coming from the cgroup classid of 0x00110011 with mark "11". This will cause the packets to use the "novpn" routing table we created above.

iptables -t mangle -A OUTPUT -m cgroup --cgroup 0x00110011 -j MARK --set-mark 11

How you do this really depends on the firewall you are using. I use Firehol. I just added the line to my firehol.conf file above the interface stanza.

Depending on how restrictive your firewall rules are, you may also need to add a rule to allow these marked packets through. With Firehol I add this to the appropriate interface stanza: client4 all accept rawmark 11

cls_group module

Load the cls_group module as we'll need to to be present for the next step:

modprobe cls_group

You may need to load the cls_cgroup kernel module on boot by creating a file named /etc/modules-load.d/cls_cgroup-novpn.conf with a sigle line in it: cls_cgroup

What is this module? Net_cls tags packets so that the Linux traffic controller can identify packets coming from specific cgroup processes. Packets tagged with net_cls can then be acted upon by iptables/net_filter.

Putting applications into cgroups

Now we need a way of putting running applications into cgroups. There is a simple daemon that will do this automatically based on rules listed in a file. It's called cgrulesengd. It is part of libcgroup. In debian cgrulesengd is found in the cgroup-tools package. Lets install that along with daemontools which we'll use to start cgrulesengd. (FIXME: create a working systemd unit file)

apt-get install cgroup-tools daemontools-run

Set up the cgroup and net_cls controller. Create /etc/cgconfig.conf:

mount {
  net_cls = /sys/fs/cgroup/net_cls;
}
group novpn {
  net_cls {
    net_cls.classid = "0x00110011";
  }
}

Create a rule for testing in /etc/cgrules.conf:

*:/tmp/traceroute   net_cls     novpn/

This example will put any process whose command path is /tmp/traceroute into the novpn/ group of the net_cls controller. We'll test this later on.

Also remove any other rules that were in the cgrules.conf file. I strongly suggest you read the cgrules.conf man page to get an idea of how to set up your rules.

Create the file daemontools needs to set up and start cgrulesengd. Create /etc/service/cgrulesengd/run:

#!/bin/bash
cgconfigparser -l /etc/cgconfig.conf
mkdir -p /sys/fs/cgroup/net_cls/novpn
echo 0x00110011 > /sys/fs/cgroup/net_cls/novpn/net_cls.classid
exec cgrulesengd -n

Set it executable and it should start up immediately:

chmod +x /etc/service/cgrulesengd/run

cgconfigparser loads the /etc/cgconfig.conf file and mounts the net_cls filesystem under /sys/fs/cgroups.

You can add -v, -vv, or -vvv switches for increased logging verbosity. I found that the logging of when it actually changes a cgroup only happens when maximum verbosity(-vvv) is turned on. Not ideal. I turn off all verbosity when I am sure cgrulesengd is doing what it should. Otherwise it really logs a lot of junk.

A note about cgrulesengd

From the man page:

The  list  of  rules  is  read  during the daemon startup is are cached in the daemon's memory.  The daemon
reloads the list of rules when it receives SIGUSR2 signal.  The daemon reloads the list of  templates  when
it receives SIGUSR1 signal.

This can be done like this:

killall -SIGUSR1 cgrulesengd
killall -SIGUSR2 cgrulesengd

Testing

Turn on your VPN if you have not already.

Open a terminal and do a traceroute to some IP:

traceroute -n 4.2.2.2

Now try this:

cp `which traceroute` /tmp
/tmp/traceroute -n 4.2.2.2

You should see that the traceroute produced by the traceroute binary located in /tmp bypassed your VPN.

In syslog ( if you have run cgrulesengd with -vvv) something like this:

Apr  4 18:15:41 ebk CGRE[505289]: EXEC Event: PID = 505321, tGID = 505321
Apr  4 18:15:41 ebk CGRE[505289]: Scanned proc values are 0 0 0 0
Apr  4 18:15:41 ebk CGRE[505289]: Scanned proc values are 0 0 0 0
Apr  4 18:15:41 ebk CGRE[505289]: Found matching rule * for PID: 505321, UID: 0, GID: 0
Apr  4 18:15:41 ebk CGRE[505289]: Executing rule * for PID 505321... 
Apr  4 18:15:41 ebk CGRE[505289]: Will move pid 505321 to cgroup 'novpn/'
Apr  4 18:15:41 ebk CGRE[505289]: Adding controller net_cls
Apr  4 18:15:41 ebk CGRE[505289]: OK!
Apr  4 18:15:41 ebk CGRE[505289]: Cgroup change for PID: 505321, UID: 0, GID: 0, PROCNAME: /tmp/traceroute OK

Debugging

Stop cgrulesengd:

svc -d /etc/service/cgrulesengd

Run it in debug mode in the foreground:

cgrulesengd -d

You should see your processes that match the rules in /etc/cgrules.conf handled by cgrulesengd.

Check for processes in the "novpn" cgroup:

ps -e -o pid,user,cmd,cgroup | grep novpn

Check the routing rules and tables:

# ip rou show table novpn
default via 192.168.100.1 dev wlp9s0 src 192.168.100.5 
# ip rule show
0:  from all lookup local
32765:  from all fwmark 0xb lookup novpn
32766:  from all lookup main
32767:  from all lookup default

To help with troubleshooting(and I found this is really helpful in general) I created a script to get me a bash prompt:

~/bin/bash-novpn:

#!/bin/bash
exec bash

Then I added a rule for it: *:/home/myusername/bin/bash-novpn net_cls novpn/

Potential problems

If your VPN software forces the system to use different DNS servers and these DNS servers refuse to service queries from outside the VPN address space you will need to use some method of changing the DNS servers in use on a per-process basis. I found the proxc script to work well for this.

Example - ~/bin/firefox-novpn:

#!/bin/bash
CMD="firefox -P NoVPNProfile"

if [ -z "`ip rou show default|grep tun0`" ] ; then
    $CMD
else
    proxc  -d 4.2.2.2 -c $CMD
fi

If there is no route going to the VPN interface(tun0) it will just run normally. Otherwise Firefox will be run via proxc using 4.2.2.2 as the DNS server. Use your ISP DNS server or any other public access DNS server.

Additional sources:

https://serverfault.com/questions/669430/how-to-bypass-openvpn-per-application
https://translate.yandex.ru/translate?url=https%3A%2F%2Fyurinb.livejournal.com%2F55588.html&lang=ru-en

This work is licensed under CC BY-NC 4.0 - https://creativecommons.org/publicdomain/zero/1.0/