Tuesday, September 13, 2016

Connecting to wifi without NetworkManager on Ubuntu 16.04 (using wpa_supplicant, dnsmasq, etc)

Suppose NetworkManager in Ubuntu is getting in your way and not letting you configure your wifi how you'd like. You'd like to temporarily turn off NetworkManager and connect to wifi in a lower-level way, without making any permanent changes that will interfere with NetworkManager. Below is the series of commands that worked for me. The approach below plays well with the underlying dnsmasq and wpa_supplicant that NetworkManager uses.

The first step is turning off NetworkManager. This seems to persist across sleeping/waking my laptop.

sudo service NetworkManager stop

Next, get the name of your wireless interface. On my laptop it is wlp3s0, but it might be some other 'w' name like wlan0. All the code snippets below use wlp3s0. Change appropriately if your interface name is different. To find the interface name, you could do "ip link" and look for an entry that begins with "w", or more programatically:

$ iw dev | awk '/Interface/{print $2;}'
wlp3s0

Next, take down the interface, make changes, and bring it back up.

sudo ip link set dev wlp3s0 down
sudo ip link set dev wlp3s0 blah blah blah  # make any changes you want here
sudo ip link set dev wlp3s0 up

Now for some magic. wpa_supplicant is a process that knows how to connect/associate with an access point using wpa security (unlike iwconfig, which apparently only knows about the weaker WEP). It looks like, when you turn off NetworkManager, it tells wpa_supplicant to forget about wireless interfaces. This means that command-line tools like wpa_cli, that talk with wpa_supplicant, won't work. So the way to tell wpa_supplicant about the wireless interface again is:

$ sudo gdbus call --system --dest=fi.w1.wpa_supplicant1 --object-path /fi/w1/wpa_supplicant1 --method fi.w1.wpa_supplicant1.CreateInterface "{'Ifname':<'wlp3s0'>,'Driver':<'nl80211,wext'>}"
(objectpath '/fi/w1/wpa_supplicant1/Interfaces/25',)
Note down the path it returns (/fi/w1/wpa_supplicant1/Interfaces/25 in this example). You'll use it when restoring NetworkManager, below.

Now wpa_cli should start working. It looks like NetworkManager configures wpa_supplicant to listen on a control socket different from the default place wpa_cli looks, so you'll need an extra arg to wpa_cli, "-p /run/wpa_supplicant", as below.

wpa_supplicant and thus wpa_cli have the notion of a 'network', which is an access point you'd like wpa_supplicant to try to connect to. Taking down NetworkManager should have removed any networks, but to be sure, try:

sudo wpa_cli -p /run/wpa_supplicant -i wlp3s0 list_networks

At this point, you can scan for wifi networks you'd like to connect to (I fuzzed the output below to not reveal actual BSSIDs or SSIDs):

$ sudo wpa_cli -p /run/wpa_supplicant -i wlp3s0 scan
OK
$ sudo wpa_cli -p /run/wpa_supplicant -i wlp3s0 scan_results
bssid / frequency / signal level / flags / ssid
20:e5:2a:2b:34:d1 2442 -58 [WPA2-PSK-CCMP][WPS][ESS] Josh's Network
22:86:8c:e1:33:70 2462 -78 [ESS] xfinitywifi

Create a 'network' to associate with. This prints an integer. It should be I think 0, since NetworkManager removed networks when we stopped it. That '0' appears in the wpa_cli commands, below. Change if add_network returns something other than 0.

$ sudo wpa_cli -p /run/wpa_supplicant -i wlp3s0 add_network
0

Pick an ssid and set the wpa password

$ sudo wpa_cli -p /run/wpa_supplicant -i wlp3s0 set_network 0 ssid "\"Josh's Network\""
OK
$ sudo wpa_cli -p /run/wpa_supplicant -i wlp3s0 set_network 0 psk '"a passphrase"'
OK
# Alternatively, to connect without any passphrase, you can say
# sudo wpa_cli -p /run/wpa_supplicant -i wlp3s0 set_network 0 key_mgmt NONE

Enabling the network should cause wpa_supplicant to connect

$ sudo wpa_cli -p /run/wpa_supplicant -i wlp3s0 enable_network 0
OK

To check status of connection (output below a bit fuzzed to not reveal actual address info):

$ sudo wpa_cli -p /run/wpa_supplicant -i wlp3s0 status
bssid=20:e5:2a:2b:34:d1
freq=2442
ssid=Josh's Network
id=0
mode=station
pairwise_cipher=CCMP
group_cipher=CCMP
key_mgmt=WPA2-PSK
wpa_state=COMPLETED
address=00:19:d1:d4:d9:22
uuid=61294555-153f-568c-9ed7-36af41fff2e0

Once you're connected, which I think is when the "wpa_state=COMPLETED" shows up in status, get an IP

sudo dhclient -v wlp3s0

Next set up DNS. NetworkManager is set up to use dnsmasq. To communicate with dnsmasq, do something like this (to set it to use opendns servers):

$ sudo dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager.dnsmasq /uk/org/thekelleys/dnsmasq uk.org.thekelleys.SetDomainServers array:string:"208.67.222.222","208.67.220.220"
method return time=1473770256.196485 sender=:1.165 -> destination=:1.197 serial=11 reply_serial=2
That's it!
$ ping google.com
PING google.com (209.85.232.139) 56(84) bytes of data.
64 bytes from qt-in-f139.1e100.net (209.85.232.139): icmp_seq=1 ttl=41 time=32.0 ms
64 bytes from qt-in-f139.1e100.net (209.85.232.139): icmp_seq=2 ttl=41 time=32.3 ms

To then to tear things down when done and restart NetworkManager:

$ sudo wpa_cli -p /run/wpa_supplicant -i wlp3s0 disable_network 0
OK
$ sudo wpa_cli -p /run/wpa_supplicant -i wlp3s0 remove_network 0
OK
sudo gdbus call --system --dest=fi.w1.wpa_supplicant1 --object-path /fi/w1/wpa_supplicant1 --method fi.w1.wpa_supplicant1.RemoveInterface "'/fi/w1/wpa_supplicant1/Interfaces/13'"
()
$ sudo service network-manager start

Below is a script putting this all together, for your reference. I didn't focus much on robustness, so it's a little fragile:

#!/bin/bash

set -o pipefail # Make it so return code from pipe is last one to fail

# https://w1.fi/wpa_supplicant/devel/dbus.html - super handy API reference for talking with wpa_supplicant via gdbus

setup() {
    wif=$(iw dev | awk '/Interface/{print $2;}')
    expectNoNetworks=0
    if (sudo service network-manager status | egrep '^\s+Active: active ' > /dev/null); then
 echo "Stopping NetworkManager"
 sudo service network-manager stop || exit
 sleep 1 # Give /run/wpa_supplicant time to disappear
 if [ -e /run/wpa_supplicant ]; then
     echo "error: /run/wpa_supplicant exists"
     exit 1
 fi
 
 sudo ip link set dev $wif down || exit
        #
 #  Make any changes to the device in here.....
        #
        #
 sudo ip link set dev $wif up || exit
 expectNoNetworks=1
    fi

    echo "Checking wpa_supplicant interface for $wif"
    if ! pre="$(sudo gdbus call --system --dest=fi.w1.wpa_supplicant1 --object-path /fi/w1/wpa_supplicant1 --method fi.w1.wpa_supplicant1.GetInterface "'$wif'" 2> /dev/null )"; then
 echo "  Creating interface"
 pre="$(sudo gdbus call --system --dest=fi.w1.wpa_supplicant1 --object-path /fi/w1/wpa_supplicant1 --method fi.w1.wpa_supplicant1.CreateInterface "{'Ifname':<'$wif'>,'Driver':<'nl80211,wext'>}")" || exit
    fi
    # Converts something like (objectpath '/fi/w1/wpa_supplicant1/Interfaces/25',)   ==>    /fi/w1/wpa_supplicant1/Interfaces/25
    ifPath=$(echo $pre | sed -e "s/^.*'\(.*\)'.*$/\1/")
    
    if [ ! -e /run/wpa_supplicant ]; then
 echo "error: /run/wpa_supplicant does not exist"
 exit 1
    fi

    if ! netInt=$(sudo wpa_cli -p /run/wpa_supplicant -i $wif list_networks | awk '/^[0-9]+\s/ { print $1;}'); then
 echo "List networks failed"
 exit 1
    fi
    if [ "$netInt" != "0" ] || [ $expectNoNetworks -eq 1 ]; then
 if [ "$netInt" ]; then
     echo "Removing any networks"
     sudo gdbus call --system --dest=fi.w1.wpa_supplicant1 --object-path $ifPath --method fi.w1.wpa_supplicant1.Interface.RemoveAllNetworks || exit
 fi
     # We don't expect NetworkManager to have left around any networks, but get rid of them in any case
 netInt="$(sudo wpa_cli -p /run/wpa_supplicant -i $wif add_network)" || exit
    fi
    echo "netInt is $netInt, ifPath is $ifPath"
}
command="start"
if [ "$1" ]; then
    command="$1"
fi
case $command in
    setup)
 setup
 exit 0
 ;;
    list)
 setup
 echo "Doing a AP scan"
 sudo wpa_cli -p /run/wpa_supplicant -i $wif status
 sudo wpa_cli -p /run/wpa_supplicant -i $wif status | grep 'wpa_state'
 sudo wpa_cli -p /run/wpa_supplicant -i $wif scan > /dev/null || exit
 while sudo wpa_cli -p /run/wpa_supplicant -i $wif status | grep 'wpa_state=SCANNING' > /dev/null; do
     sleep 0.1
 done
 # Reformat scan results to be a bit prettier.
 sudo wpa_cli -p /run/wpa_supplicant -i $wif scan_results | grep -v 'bssid / frequency / signal level / flags / ssid' | sort -nr -k 3 | awk 'BEGIN { FS="\t"; printf "Signal\n"; printf "%6s %-30s %-50s %s %s\n", "Level", "SSID", "Flags", "Frequency", "BSSID"; } { printf "%6s %-30s %-50s %-9s %s\n", $3, $5, $4, $2, $1; }'

 read -p "Enter a SSID to connect to: " aSsid
 if [ -z "$aSsid" ]; then
     echo "Can not have empty ssid, i think"
     exit 1
 fi
 #sudo wpa_cli -p /run/wpa_supplicant -i $wif status
 sudo wpa_cli -p /run/wpa_supplicant -i $wif status | grep 'wpa_state'
 if ! sudo wpa_cli -p /run/wpa_supplicant -i $wif status | egrep 'wpa_state=(INACTIVE|DISCONNECTED)' > /dev/null; then
     echo "Disconnecting current connection"
     sudo gdbus call --system --dest=fi.w1.wpa_supplicant1 --object-path $ifPath --method fi.w1.wpa_supplicant1.Interface.Disconnect > /dev/null || exit
 fi
 echo "Disabling network"
 sudo wpa_cli -p /run/wpa_supplicant -i $wif disable_network 0  > /dev/null || exit
 sudo wpa_cli -p /run/wpa_supplicant -i $wif status | grep 'wpa_state'
 echo "Setting ssid"
 sudo wpa_cli -p /run/wpa_supplicant -i $wif set_network 0 ssid "\"$aSsid\""  > /dev/null || exit
 read -sp "Enter WPA password, or leave blank to try connecting with no password (chars will not echo): " aPassword
 if [ -z "$aPassword" ]; then
     echo "using no passowrd"
     sudo wpa_cli -p /run/wpa_supplicant -i $wif set_network 0 key_mgmt NONE > /dev/null || exit
 else
     sudo wpa_cli -p /run/wpa_supplicant -i $wif set_network 0 psk "\"$aPassword\""  > /dev/null || exit
 fi
 echo "Enabling network"
 sudo wpa_cli -p /run/wpa_supplicant -i $wif status | grep 'wpa_state'
 sudo wpa_cli -p /run/wpa_supplicant -i $wif enable_network 0  > /dev/null || exit
 if sudo wpa_cli -p /run/wpa_supplicant -i $wif status | grep 'wpa_state=DISCONNECTED' > /dev/null ; then
     echo "Selecting network $netInt"
     sudo gdbus call --system --dest=fi.w1.wpa_supplicant1 --object-path $ifPath --method fi.w1.wpa_supplicant1.Interface.SelectNetwork "$ifPath/Networks/$netInt" > /dev/null || exit
 fi

 echo "Waiting for wpa_supplicant to get into COMPLETED state"
 while ! sudo wpa_cli -p /run/wpa_supplicant -i $wif status | grep 'wpa_state=COMPLETED' > /dev/null ; do
     sudo wpa_cli -p /run/wpa_supplicant -i $wif status | grep 'wpa_state'
     sleep 0.5
 done
 
 echo "Now getting IP"
 sudo dhclient -v $wif || exit

 echo "Setting openDNS"
 sudo dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager.dnsmasq /uk/org/thekelleys/dnsmasq uk.org.thekelleys.SetDomainServers array:string:"208.67.222.222","208.67.220.220" > /dev/null || exit
 echo "Done!"
 exit 0
 ;;
    done)
 if (sudo service network-manager status | egrep '^\s+Active: active ' > /dev/null); then
     echo "Looks like NetworkManager is active, so I think we're done"
     exit 0
 fi
 setup
 echo "Disabling Network"
 sudo wpa_cli -p /run/wpa_supplicant -i $wif disable_network 0  > /dev/null || exit
 echo "Removing all networks"
 sudo wpa_cli -p /run/wpa_supplicant -i $wif remove_network $netInt || exit
 #sudo gdbus call --system --dest=fi.w1.wpa_supplicant1 --object-path $ifPath --method fi.w1.wpa_supplicant1.Interface.RemoveAllNetworks || exit
 sudo wpa_cli -p /run/wpa_supplicant -i $wif status | grep 'wpa_state'
 echo "Removing interface"
 sudo gdbus call --system --dest=fi.w1.wpa_supplicant1 --object-path /fi/w1/wpa_supplicant1 --method fi.w1.wpa_supplicant1.RemoveInterface "'$ifPath'" || exit
 sleep 1
 echo "Restarting NetworkManager"
 sudo service network-manager start || exit
 echo "Done"
 exit 0
 ;;
    *)
 echo "Command arg is (setup | list | done)"
 exit 1
    ;;
    
esac