If you’re also running into trouble with openvswitch-switch and really have no clue why it is not running as it should, you might also have a mismatch of a default no-subscription installation (it comes out of the box this way) and the default repo configured to work with the subscription/enterprise version only.

Unless you have a subscription key you will never get any updates for the PVE system itself – so before changing the network configuration to work with openvswitch-switch, change your repo in /etc/apt/sources.list.d/pve-enterprise.list as follows:

#deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise
deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

Update the package lists (apt update) and upgrade (apt upgrade) – you will get updates for openvswitch-switch!

Since pfSense is not actually rerouting router traffic itself (such as DNS, VPN, …) but only incoming traffic when a gateway goes down and another one is configured in the same gateway group, I have written the following script that you can use in a cron job. It will change the IPv4 default route for basically all traffic not specifically treated via FW rules – including the internal services.

• MOBILE1 needs to be set to your second gateway, in my case a mobile LTE device
• MOBILE2 and MOBILE3 need to be set to rarely used IPs – so the LTE traffic going there is not too much as
• MOBILE2 and MOBILE3 need to be statically routed via LTE, always, to check their reachability

• WAN1 needs to be set to your main gateway, in my case a FritzBox
• WAN2 and WAN3 need to be set to pages you usually want to reach, but it is not so bad to be unreachable in case of a downtime of the WAN gateway as
• WAN2 and WAN3 need to be statically routed via WAN, always, to check their reachability

The script will log changes and send mails to the email address configured in pfSense.

#!/usr/local/bin/bash

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin

ROUTE=/sbin/route
LOG=/root/switched.txt

# Destinations, die man fast nie braucht (Traffic-Beschraenkung via MOBILE) und von MOBILE gehen sollten, muessen via MOBILE geroutet werden!
# Router selber
MOBILE1=192.168.166.1
# www.t-online.de
MOBILE2=217.6.164.162
# www.lrz.de
MOBILE3=129.187.255.234

# Normale IPs, muessen via WAN geroutet werden!
# Router selber
WAN1=192.168.178.1
# www.berlin.de
WAN2=212.45.111.17
# www.hamburg.de
WAN3=212.1.41.12

TRIES=3
SLEEP=0

WAN_SUCCESS=0
MOBILE_SUCCESS=0

# Router does not count
let MIN_TIMES=$TRIES*2

if [ $(netstat -nr4 | grep default | grep $WAN1 | wc -l) -eq 1 ]
then
  ON_WAN=1
  ON_MOBILE=0
else
  ON_WAN=0
  ON_MOBILE=1
fi

for try in $(seq 1 $TRIES)
do
  for ip in $WAN1 $WAN2 $WAN3;
  do
    ping -t 2 -c 1 $ip 2>/dev/null >/dev/null && let WAN_SUCCESS=$WAN_SUCCESS+1
    sleep $SLEEP
  done
done
echo "3 WAN IPs reachable $WAN_SUCCESS times in $TRIES runs, minimum is $MIN_TIMES."

if [ $WAN_SUCCESS -ge $MIN_TIMES ] && [ $ON_WAN -eq 1 ]
then
  echo "Enough WAN results and on WAN, nothing to do, exiting!"
  exit
elif [ $WAN_SUCCESS -ge $MIN_TIMES ] && [ $ON_MOBILE -eq 1 ]
then
  echo "Enough WAN results but on MOBILE, change back to WAN"
  echo -n "Switching to WAN at " >> $LOG
  date >> $LOG
  echo "" | /usr/local/bin/mail.php -s"Switching to WAN"

  $ROUTE del default && $ROUTE add default $WAN1
  exit
fi

# Not enough WAN results, continue with MOBILE checks/actions

for try in $(seq 1 $TRIES)
do
  for ip in $MOBILE1 $MOBILE2 $MOBILE3;
  do
    ping -t 2 -c 1 $ip 2>/dev/null >/dev/null && let MOBILE_SUCCESS=$MOBILE_SUCCESS+1
    sleep $SLEEP
  done
done
echo "3 MOBILE IPs reachable $MOBILE_SUCCESS times in $TRIES runs, minimum is $MIN_TIMES."

if [ $MOBILE_SUCCESS -ge $MIN_TIMES ] && [ $ON_MOBILE -eq 1 ]
then
  echo "Enough MOBILE results and on MOBILE, nothing to do, exiting!"
  exit
elif [ $MOBILE_SUCCESS -ge $MIN_TIMES ] && [ $ON_WAN -eq 1 ]
then
  echo "Enough MOBILE results but on WAN, change to MOBILE"
  echo -n "Switching to MOBILE at " >> $LOG
  date >> $LOG
  echo "" | /usr/local/bin/mail.php -s"Switching to MOBILE"

  $ROUTE del default && $ROUTE add default $MOBILE1
  exit
fi

For systems that do not provide Sixxs’ aiccu package to setup a GIF tunnel automatically, you can easily start the tunnel (not setup the routing 🙂 ) by executing the following script once per minute via cron:

#!/usr/local/bin/python2.7

import time,hashlib,subprocess,socket,os

localv6="2001:3880:fe20:121::2"
password="abcdef1234567890abcdef123456"
remotev4="98.76.54.123"
remotev6="2001:3880:fe20:121::1"

hbBase="HEARTBEAT TUNNEL " + localv6 + " sender " + str(int(time.time()))
hbToSend=hbBase + " " + hashlib.md5(hbBase + " " + password).hexdigest()
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.sendto(hbToSend, (remotev4, 3740))
sock.close()
with open(os.devnull, "w") as fnull:
  subprocess.call(["/sbin/ping6", "-s", "8", "-c", "1", remotev6], stdout=fnull, stderr=fnull)

This solution was first posted here:

UBNT – Sixxs Tunnel

This code will read in a JSON file and print it out again in readable form and keys in sorted order – making two files diffable!

Usage: jsonsort.py

#!/usr/bin/env python

import json
import sys

file = sys.argv[1]

with open(file, 'r') as f:
  read_data = f.read()
f.closed

print json.dumps(json.loads(read_data),sort_keys=True, indent=4, separators=(',', ': '))