After upgrading to Ubuntu Jammy and Asterisk 18.10, I saw the following messages in my logs:

WARNING[360166]: loader.c:2487 in load_modules: Module 'chan_sip' has been loaded but was deprecated in Asterisk version 17 and will be removed in Asterisk version 21.
WARNING[360174]: chan_sip.c:35468 in deprecation_notice: chan_sip has no official maintainer and is deprecated.  Migration to
WARNING[360174]: chan_sip.c:35469 in deprecation_notice: chan_pjsip is recommended.  See guides at the Asterisk Wiki:
WARNING[360174]: chan_sip.c:35470 in deprecation_notice: https://wiki.asterisk.org/wiki/display/AST/Migrating+from+chan_sip+to+res_pjsip
WARNING[360174]: chan_sip.c:35471 in deprecation_notice: https://wiki.asterisk.org/wiki/display/AST/Configuring+res_pjsip

and so I decided it was time to stop postponing the overdue migration of my working setup from chan_sip to res_pjsip. It turns out that it was not as painful as I expected, though the conversion script bundled with Asterisk didn't work for me out of the box.

Debugging Before you start, one very important thing to note is that the SIP debug information you used to see when running this in the asterisk console (asterisk -r):

sip set debug on

now lives behind this command:

pjsip set logger on

SIP phones The first thing I migrated was the config for my two SIP phones (Snom 300 and Snom D715). The original config for them in sip.conf was:

[2000]
; Snom 300
type=friend
qualify=yes
secret=password123
encryption=no
context=full
host=dynamic
nat=no
directmedia=no
mailbox=10@internal
vmexten=707
dtmfmode=rfc2833
call-limit=2
disallow=all
allow=g722
allow=ulaw
[2001]
; Snom D715
type=friend
qualify=yes
secret=password456
encryption=no
context=full
host=dynamic
nat=no
directmedia=yes
mailbox=10@internal
vmexten=707
dtmfmode=rfc2833
call-limit=2
disallow=all
allow=g722
allow=ulaw

and that became the following in pjsip.conf:

[transport-udp]
type = transport
protocol = udp
bind = 0.0.0.0
external_media_address = myasterisk.dyn.example.com
external_signaling_address = myasterisk.dyn.example.com
local_net = 192.168.0.0/255.255.0.0
[2000]
type = aor
max_contacts = 1
[2000]
type = auth
username = 2000
password = password123
[2000]
type = endpoint
context = full
dtmf_mode = rfc4733
disallow = all
allow = g722
allow = ulaw
direct_media = no
mailboxes = 10@internal
auth = 2000
outbound_auth = 2000
aors = 2000
[2001]
type = aor
max_contacts = 1
[2001]
type = auth
username = 2001
password = password456
[2001]
type = endpoint
context = full
dtmf_mode = rfc4733
disallow = all
allow = g722
allow = ulaw
direct_media = yes
mailboxes = 10@internal
auth = 2001
outbound_auth = 2001
aors = 2001

The different direct_media line between the two phones has to do with how they each connect to my Asterisk server and whether or not they have access to the Internet.

Internal calls For some reason, my internal calls (from one SIP phone to the other) didn't work when using "aliases". I fixed it by changing this blurb in extensions.conf from:

[speeddial]
exten => 1000,1,Dial(SIP/2000,20)
exten => 1001,1,Dial(SIP/2001,20)

to:

[speeddial]
exten => 1000,1,Dial($ PJSIP_DIAL_CONTACTS(2000) ,20)
exten => 1001,1,Dial($ PJSIP_DIAL_CONTACTS(2001) ,20)

I have not yet dug into what this changes or why it's necessary and so feel free to leave a comment if you know more here.

PSTN trunk Once I had the internal phones working, I moved to making and receiving phone calls over the PSTN, for which I use VoIP.ms with encryption. I had to change the following in my sip.conf:

[general]
register => tls://555123_myasterisk:password789@vancouver2.voip.ms
externhost=myasterisk.dyn.example.com
localnet=192.168.0.0/255.255.0.0
tcpenable=yes
tlsenable=yes
tlscertfile=/etc/asterisk/asterisk.cert
tlsprivatekey=/etc/asterisk/asterisk.key
tlscapath=/etc/ssl/certs/
[voipms]
type=peer
host=vancouver2.voip.ms
secret=password789
defaultuser=555123_myasterisk
context=from-voipms
disallow=all
allow=ulaw
allow=g729
insecure=port,invite
canreinvite=no
trustrpid=yes
sendrpid=yes
transport=tls
encryption=yes

to the following in pjsip.conf:

[transport-tls]
type = transport
protocol = tls
bind = 0.0.0.0
external_media_address = myasterisk.dyn.example.com
external_signaling_address = myasterisk.dyn.example.com
local_net = 192.168.0.0/255.255.0.0
cert_file = /etc/asterisk/asterisk.cert
priv_key_file = /etc/asterisk/asterisk.key
ca_list_path = /etc/ssl/certs/
method = tlsv1_2
[voipms]
type = registration
transport = transport-tls
outbound_auth = voipms
client_uri = sip:555123_myasterisk@vancouver2.voip.ms
server_uri = sip:vancouver2.voip.ms
[voipms]
type = auth
password = password789
username = 555123_myasterisk
[voipms]
type = aor
contact = sip:555123_myasterisk@vancouver2.voip.ms
[voipms]
type = identify
endpoint = voipms
match = vancouver2.voip.ms
[voipms]
type = endpoint
context = from-voipms
disallow = all
allow = ulaw
allow = g729
from_user = 555123_myasterisk
trust_id_inbound = yes
media_encryption = sdes
auth = voipms
outbound_auth = voipms
aors = voipms
rtp_symmetric = yes
rewrite_contact = yes
send_rpid = yes
timers = no

The TLS method line is needed since the default in Debian OpenSSL is too strict. The timers line is to prevent outbound calls from getting dropped after 15 minutes. Finally, I changed the Dial() lines in these extensions.conf blurbs from:

[from-voipms]
exten => 5551231000,1,Goto(2000,1)
exten => 2000,1,Dial(SIP/2000&SIP/2001,20)
exten => 2000,n,Goto(in2000-$ DIALSTATUS ,1)
exten => 2000,n,Hangup
exten => in2000-BUSY,1,VoiceMail(10@internal,su)
exten => in2000-BUSY,n,Hangup
exten => in2000-CONGESTION,1,VoiceMail(10@internal,su)
exten => in2000-CONGESTION,n,Hangup
exten => in2000-CHANUNAVAIL,1,VoiceMail(10@internal,su)
exten => in2000-CHANUNAVAIL,n,Hangup
exten => in2000-NOANSWER,1,VoiceMail(10@internal,su)
exten => in2000-NOANSWER,n,Hangup
exten => _in2000-.,1,Hangup(16)
[pstn-voipms]
exten => _1NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _1NXXNXXXXXX,n,Dial(SIP/voipms/$ EXTEN )
exten => _1NXXNXXXXXX,n,Hangup()
exten => _NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _NXXNXXXXXX,n,Dial(SIP/voipms/1$ EXTEN )
exten => _NXXNXXXXXX,n,Hangup()
exten => _011X.,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _011X.,n,Authenticate(1234)
exten => _011X.,n,Dial(SIP/voipms/$ EXTEN )
exten => _011X.,n,Hangup()
exten => _00X.,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _00X.,n,Authenticate(1234)
exten => _00X.,n,Dial(SIP/voipms/$ EXTEN )
exten => _00X.,n,Hangup()

to:

[from-voipms]
exten => 5551231000,1,Goto(2000,1)
exten => 2000,1,Dial(PJSIP/2000&PJSIP/2001,20)
exten => 2000,n,Goto(in2000-$ DIALSTATUS ,1)
exten => 2000,n,Hangup
exten => in2000-BUSY,1,VoiceMail(10@internal,su)
exten => in2000-BUSY,n,Hangup
exten => in2000-CONGESTION,1,VoiceMail(10@internal,su)
exten => in2000-CONGESTION,n,Hangup
exten => in2000-CHANUNAVAIL,1,VoiceMail(10@internal,su)
exten => in2000-CHANUNAVAIL,n,Hangup
exten => in2000-NOANSWER,1,VoiceMail(10@internal,su)
exten => in2000-NOANSWER,n,Hangup
exten => _in2000-.,1,Hangup(16)
[pstn-voipms]
exten => _1NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _1NXXNXXXXXX,n,Dial(PJSIP/$ EXTEN @voipms)
exten => _1NXXNXXXXXX,n,Hangup()
exten => _NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _NXXNXXXXXX,n,Dial(PJSIP/1$ EXTEN @voipms)
exten => _NXXNXXXXXX,n,Hangup()
exten => _011X.,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _011X.,n,Authenticate(1234)
exten => _011X.,n,Dial(PJSIP/$ EXTEN @voipms)
exten => _011X.,n,Hangup()
exten => _00X.,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _00X.,n,Authenticate(1234)
exten => _00X.,n,Dial(PJSIP/$ EXTEN @voipms)
exten => _00X.,n,Hangup()

Note that it's not just replacing SIP/ with PJSIP/, but it was also necessary to use a format supported by pjsip for the channel since SIP/trunkname/extension isn't supported by pjsip.

JMP offers VoIP calling via XMPP, but it's also possibly to use the VoIP using SIP. The underlying VoIP calling functionality in JMP is provided by Bandwidth, but their old Asterisk instructions didn't quite work for me. Here's how I set it up in my Asterisk server.

Get your SIP credentials After signing up for JMP and setting it up in your favourite XMPP client, send the following message to the cheogram.com gateway contact:

reset sip account

In response, you will receive a message containing:

• a numerical username
• a password (e.g. three lowercase words separated by spaces)

Add SIP account to your Asterisk config First of all, I added the following near the top of my /etc/asterisk/sip.conf:

[general]
register => username:three secret words@jmp.cbcbc7.auth.bandwidth.com:5008

The other non-default options I have set in [general] are:

context=public
allowoverlap=no
udpbindaddr=0.0.0.0
tcpenable=yes
tcpbindaddr=0.0.0.0
tlsenable=yes
transport=udp
srvlookup=no
vmexten=voicemail
relaxdtmf=yes
useragent=Asterisk PBX
tlscertfile=/etc/asterisk/asterisk.cert
tlsprivatekey=/etc/asterisk/asterisk.key
tlscapath=/etc/ssl/certs/
externhost=machinename.dyndns.org
localnet=192.168.0.0/255.255.0.0

Note that you can have more than one register line in your config if you use more than one SIP provider, but you must register with the server whether you want to receive incoming calls or not. Then I added a new blurb to the bottom of the same file:

[jmp]
type=peer
host=mp.cbcbc7.auth.bandwidth.com
port=5008
secret=three secret words
defaultuser=username
context=from-jmp
disallow=all
allow=ulaw
allow=g729
insecure=port,invite
canreinvite=no
dtmfmode=rfc2833

and for reference, here's the blurb for my Snom 300 SIP phone:

[1001]
; Snom 300
type=friend
qualify=yes
secret=password
encryption=no
context=full
host=dynamic
nat=no
directmedia=no
mailbox=1000@internal
vmexten=707
dtmfmode=rfc2833
call-limit=2
disallow=all
allow=g722
allow=ulaw

I checked that the registration was successful by running asterisk -r and then typing:

sip set debug on

before reloading the configuration using:

reload

Create Asterisk extensions to send and receive calls Once I got registration to work, I hooked this up with my other extensions so that I could send and receive calls using my JMP number. In /etc/asterisk/extensions.conf, I added the following:

[from-jmp]
include => home
exten => s,1,Goto(1000,1)

where home is the context which includes my local SIP devices and 1000 is the extension I want to ring. Then I added the following to enable calls to any destination within the North American Numbering Plan:

[pstn-jmp]
exten => _1NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231434>)
exten => _1NXXNXXXXXX,n,Dial(SIP/jmp/$ EXTEN )
exten => _1NXXNXXXXXX,n,Hangup()
exten => _NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231234>)
exten => _NXXNXXXXXX,n,Dial(SIP/jmp/1$ EXTEN )
exten => _NXXNXXXXXX,n,Hangup()

Here 5551231234 is my JMP phone number, not my bwsip numerical username. For reference, here's the rest of my dialplan in /etc/asterisk/extensions.conf:

[general]
static=yes
writeprotect=no
clearglobalvars=no
[public]
exten => _X.,1,Hangup(3)
[sipdefault]
exten => _X.,1,Hangup(3)
[default]
exten => _X.,1,Hangup(3)
[internal]
include => home
[full]
include => internal
include => pstn-jmp
exten => 707,1,VoiceMailMain(1000@internal)
[home]
; Internal extensions
exten => 1000,1,Dial(SIP/1001,20)
exten => 1000,n,Goto(in1000-$ DIALSTATUS ,1)
exten => 1000,n,Hangup
exten => in1000-BUSY,1,Hangup(17)
exten => in1000-CONGESTION,1,Hangup(3)
exten => in1000-CHANUNAVAIL,1,VoiceMail(1000@internal,su)
exten => in1000-CHANUNAVAIL,n,Hangup(3)
exten => in1000-NOANSWER,1,VoiceMail(1000@internal,su)
exten => in1000-NOANSWER,n,Hangup(16)
exten => _in1000-.,1,Hangup(16)

Firewall Finally, I opened a few ports in my firewall by putting the following in /etc/network/iptables.up.rules:

# SIP and RTP on UDP (jmp.cbcbc7.auth.bandwidth.com)
-A INPUT -s 67.231.2.13/32 -p udp --dport 5008 -j ACCEPT
-A INPUT -s 216.82.238.135/32 -p udp --dport 5008 -j ACCEPT
-A INPUT -s 67.231.2.13/32 -p udp --sport 5004:5005 --dport 10001:20000 -j ACCEPT
-A INPUT -s 216.82.238.135/32 -p udp --sport 5004:5005 --dport 10001:20000 -j ACCEPT

Outbound calls not working While the above setup works for me for inbound calls, it doesn't currently work for outbound calls. The hostname currently resolves to one of two IP addresses:

$ dig +short jmp.cbcbc7.auth.bandwidth.com
67.231.2.13
216.82.238.135

If I pin it to the first one by putting the following in my /etc/hosts file:

67.231.2.13 jmp.cbcbc7.auth.bandwidth.com

then I get a 486 error back from the server when I dial 1-555-456-4567:

<--- SIP read from UDP:67.231.2.13:5008 --->
SIP/2.0 486 Busy Here
Via: SIP/2.0/UDP 127.0.0.1:5060;branch=z9hG4bK03210a30
From: "Francois Marier" <sip:5551231234@127.0.0.1>
To: <sip:15554564567@jmp.cbcbc7.auth.bandwidth.com:5008>
Call-ID: 575f21f36f57951638c1a8062f3a5201@127.0.0.1:5060
CSeq: 103 INVITE
Content-Length: 0

On the other hand, if I pin it to 216.82.238.135, then I get a 600 error:

<--- SIP read from UDP:216.82.238.135:5008 --->
SIP/2.0 600 Busy Everywhere
Via: SIP/2.0/UDP 127.0.0.1:5060;branch=z9hG4bK7b7f7ed9
From: "Francois Marier" <sip:5551231234@127.0.0.1>
To: <sip:15554564567@jmp.cbcbc7.auth.bandwidth.com:5008>
Call-ID: 5bebb8d05902c1732c6b9f4776844c66@127.0.0.1:5060
CSeq: 103 INVITE
Content-Length: 0

If you have any idea what might be wrong here, or if you got outbound calls to work on Bandwidth.com, please leave a comment!

One of the most awesome helpers I carry around in my ~/bin since the early '00s is the sanity.pl script written by Andreas Gohr. It just recently came back to use when I started to archive some awesome Corona enforced live session music with youtube-dl. Update: Francois Marier pointed out that Debian contains the detox package, which has a similar functionality.

In addition to selecting the right monitor after docking my ThinkPad, I wanted to set the correct sound output since I have headphones connected to my Ultra Dock. This can be done fairly easily using Pulseaudio.

Switching to a different pulseaudio output To find the device name and the output name I need to provide to pacmd, I ran pacmd list-sinks:

2 sink(s) available.
...
  * index: 1
    name: <alsa_output.pci-0000_00_1b.0.analog-stereo>
    driver: <module-alsa-card.c>
...
    ports:
        analog-output: Analog Output (priority 9900, latency offset 0 usec, available: unknown)
            properties:
        analog-output-speaker: Speakers (priority 10000, latency offset 0 usec, available: unknown)
            properties:
                device.icon_name = "audio-speakers"

From there, I extracted the soundcard name (alsa_output.pci-0000_00_1b.0.analog-stereo) and the names of the two output ports (analog-output and analog-output-speaker). To switch between the headphones and the speakers, I can therefore run the following commands:

pacmd set-sink-port alsa_output.pci-0000_00_1b.0.analog-stereo analog-output
pacmd set-sink-port alsa_output.pci-0000_00_1b.0.analog-stereo analog-output-speaker

Listening for headphone events Then I looked for the ACPI event triggered when my headphones are detected by the laptop after docking. After looking at the output of acpi_listen, I found jack/headphone HEADPHONE plug. Combining this with the above pulseaudio names, I put the following in /etc/acpi/events/thinkpad-dock-headphones:

event=jack/headphone HEADPHONE plug
action=su francois -c "pacmd set-sink-port alsa_output.pci-0000_00_1b.0.analog-stereo analog-output"

to automatically switch to the headphones when I dock my laptop.

Finding out whether or not the laptop is docked While it is possible to hook into the docking and undocking ACPI events and run scripts, there doesn't seem to be an easy way from a shell script to tell whether or not the laptop is docked. In the end, I settled on detecting the presence of USB devices. I ran lsusb twice (once docked and once undocked) and then compared the output:

lsusb  > docked 
lsusb  > undocked 
colordiff -u docked undocked 

This gave me a number of differences since I have a bunch of peripherals attached to the dock:

--- docked  2017-07-07 19:10:51.875405241 -0700
+++ undocked    2017-07-07 19:11:00.511336071 -0700
@@ -1,15 +1,6 @@
 Bus 001 Device 002: ID 8087:8000 Intel Corp. 
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
-Bus 003 Device 081: ID 0424:5534 Standard Microsystems Corp. Hub
-Bus 003 Device 080: ID 17ef:1010 Lenovo 
 Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
-Bus 002 Device 041: ID xxxx:xxxx ...
-Bus 002 Device 040: ID xxxx:xxxx ...
-Bus 002 Device 039: ID xxxx:xxxx ...
-Bus 002 Device 038: ID 17ef:100f Lenovo 
-Bus 002 Device 037: ID xxxx:xxxx ...
-Bus 002 Device 042: ID 0424:2134 Standard Microsystems Corp. Hub
-Bus 002 Device 036: ID 17ef:1010 Lenovo 
 Bus 002 Device 002: ID xxxx:xxxx ...
 Bus 002 Device 004: ID xxxx:xxxx ...
 Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

I picked 17ef:1010 as it appeared to be some internal bus on the Ultra Dock (none of my USB devices were connected to Bus 003) and then ended up with the following port toggling script:

#!/bin/bash
if /usr/bin/lsusb   grep 17ef:1010 > /dev/null ; then
    # docked
    pacmd set-sink-port alsa_output.pci-0000_00_1b.0.analog-stereo analog-output
else
    # undocked
    pacmd set-sink-port alsa_output.pci-0000_00_1b.0.analog-stereo analog-output-speaker
fi

While upgrading Libravatar to a more recent version of Django, I ran into a mysterious 400 error. In debug mode, my site was working fine, but with DEBUG = False, I would only a page containing this error:

Bad Request (400)

with no extra details in the web server logs.

Turning on extra error logging To see the full error message, I configured logging to a file by adding this to settings.py:

LOGGING =  
    'version': 1,
    'disable_existing_loggers': False,
    'handlers':  
        'file':  
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/tmp/debug.log',
         ,
     ,
    'loggers':  
        'django':  
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
         ,
     ,
 

Then I got the following error message:

Invalid HTTP_HOST header: 'www.example.com'. You may need to add u'www.example.com' to ALLOWED_HOSTS.

Temporary hack Sure enough, putting this in settings.py would make it work outside of debug mode:

ALLOWED_HOSTS = ['*']

which means that there's a mismatch between the HTTP_HOST from Apache and the one that Django expects.

Root cause The underlying problem was that the Libravatar config file was missing the square brackets around the ALLOWED_HOSTS setting. I had this:

ALLOWED_HOSTS = 'www.example.com'

instead of:

ALLOWED_HOSTS = ['www.example.com']

A laptop that was installed using the default Ubuntu 16.10 (xenial) full-disk encryption option stopped booting after receiving a kernel update somewhere on the way to Ubuntu 17.04 (zesty). After showing the boot screen for about 30 seconds, a busybox shell pops up:

BusyBox v.1.21.1 (Ubuntu 1:1.21.1-1ubuntu1) built-in shell (ash)
Enter 'help' for list of built-in commands.
(initramfs)

Typing exit will display more information about the failure before bringing us back to the same busybox shell:

Gave up waiting for root device. Common problems:
  - Boot args (cat /proc/cmdline)
    - Check rootdelay= (did the system wait long enough?)
    - Check root= (did the system wait for the right device?)
  - Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/mapper/ubuntu--vg-root does not exist. Dropping to a shell! 
BusyBox v.1.21.1 (Ubuntu 1:1.21.1-1ubuntu1) built-in shell (ash)   
Enter 'help' for list of built-in commands.  
(initramfs)

which now complains that the /dev/mapper/ubuntu--vg-root root partition (which uses LUKS and LVM) cannot be found. There is some comprehensive advice out there but it didn't quite work for me. This is how I ended up resolving the problem.

Boot using a USB installation disk First, create bootable USB disk using the latest Ubuntu installer:

1. Download an desktop image.
2. Copy the ISO directly on the USB stick (overwriting it in the process):

    dd if=ubuntu.iso of=/dev/sdc1
   

and boot the system using that USB stick (hold the option key during boot on Apple hardware).

Mount the encrypted partition Assuming a drive which is partitioned this way:

• /dev/sda1: EFI partition
• /dev/sda2: unencrypted boot partition
• /dev/sda3: encrypted LVM partition

Open a terminal and mount the required partitions:

cryptsetup luksOpen /dev/sda3 sda3_crypt
vgchange -ay
mount /dev/mapper/ubuntu--vg-root /mnt
mount /dev/sda2 /mnt/boot
mount -t proc proc /mnt/proc
mount -o bind /dev /mnt/dev

Note:

• When running cryptsetup luksOpen, you must use the same name as the one that is in /etc/crypttab on the root parition (sda3_crypt in this example).
• All of these partitions must be present (including /proc and /dev) for the initramfs scripts to do all of their work. If you see errors or warnings, you must resolve them.

Regenerate the initramfs on the boot partition Then "enter" the root partition using:

chroot /mnt

and make sure that the lvm2 package is installed:

apt install lvm2

before regenerating the initramfs for all of the installed kernels:

update-initramfs -c -k all

I use Let's Encrypt TLS certificates on my Debian servers along with the Certbot tool. Since I use the "temporary webserver" method of proving domain ownership via the ACME protocol, I cannot use the cert renewal cronjob built into Certbot. Instead, this is the script I put in /etc/cron.daily/certbot-renew:

#!/bin/bash
/usr/bin/certbot renew --quiet --pre-hook "/bin/systemctl stop apache2.service" --post-hook "/bin/systemctl start apache2.service"
pushd /etc/ > /dev/null
/usr/bin/git add letsencrypt
DIFFSTAT="$(/usr/bin/git diff --cached --stat)"
if [ -n "$DIFFSTAT" ] ; then
    /usr/bin/git commit --quiet -m "Renewed letsencrypt certs"
    echo "$DIFFSTAT"
fi
popd > /dev/null

It temporarily disables my Apache webserver while it renews the certificates and then only outputs something to STDOUT (since my cronjob will email me any output) if certs have been renewed. Since I'm using etckeeper to keep track of config changes on my servers, my renewal script also commits to the repository if any certs have changed.

External Monitoring In order to catch mistakes or oversights, I use ssl-cert-check to monitor my domains once a day:

ssl-cert-check -s fmarier.org -p 443 -q -a -e francois@fmarier.org

I also signed up with Cert Spotter which watches the Certificate Transparency log and notifies me of any newly-issued certificates for my domains. In other words, I get notified:

• if my cronjob fails and a cert is about to expire, or
• as soon as a new cert is issued.

The whole thing seems to work well, but if there's anything I could be doing better, feel free to leave a comment!

Here are the notes I took while manually expanding an non-LVM encrypted RAID1 array on an Ubuntu machine. My original setup consisted of a 1 TB drive along with a 2 TB drive, which meant that the RAID1 array was 1 TB in size and the second drive had 1 TB of unused capacity. This is how I replaced the old 1 TB drive with a new 3 TB drive and expanded the RAID1 array to 2 TB (leaving 1 TB unused on the new 3 TB drive).

Partition the new drive In order to partition the new 3 TB drive, I started by creating a temporary partition on the old 2 TB drive (/dev/sdc) to use up all of the capacity on that drive:

$ parted /dev/sdc
unit s
print
mkpart
print

Then I initialized the partition table and creating the EFI partition partition on the new drive (/dev/sdd):

$ parted /dev/sdd
unit s
mktable gpt
mkpart

Since I want to have the RAID1 array be as large as the smaller of the two drives, I made sure that the second partition (/home) on the new 3 TB drive had:

• the same start position as the second partition on the old drive
• the end position of the third partition (the temporary one I just created) on the old drive

I created the partition and flagged it as a RAID one:

mkpart
toggle 2 raid

and then deleted the temporary partition on the old 2 TB drive:

$ parted /dev/sdc
print
rm 3
print

Create a temporary RAID1 array on the new drive With the new drive properly partitioned, I created a new RAID array for it:

mdadm /dev/md10 --create --level=1 --raid-devices=2 /dev/sdd1 missing

and added it to /etc/mdadm/mdadm.conf:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf

which required manual editing of that file to remove duplicate entries.

Create the encrypted partition With the new RAID device in place, I created the encrypted LUKS partition:

cryptsetup -h sha256 -c aes-xts-plain64 -s 512 luksFormat /dev/md10
cryptsetup luksOpen /dev/md10 chome2

I took the UUID for the temporary RAID partition:

blkid /dev/md10

and put it in /etc/crypttab as chome2. Then, I formatted the new LUKS partition and mounted it:

mkfs.ext4 -m 0 /dev/mapper/chome2
mkdir /home2
mount /dev/mapper/chome2 /home2

Copy the data from the old drive With the home paritions of both drives mounted, I copied the files over to the new drive:

eatmydata nice ionice -c3 rsync -axHAX --progress /home/* /home2/

making use of wrappers that preserve system reponsiveness during I/O-intensive operations.

Switch over to the new drive After the copy, I switched over to the new drive in a step-by-step way:

1. Changed the UUID of chome in /etc/crypttab.
2. Changed the UUID and name of /dev/md1 in /etc/mdadm/mdadm.conf.
3. Rebooted with both drives.
4. Checked that the new drive was the one used in the encrypted /home mount using: df -h.

Add the old drive to the new RAID array With all of this working, it was time to clear the mdadm superblock from the old drive:

mdadm --zero-superblock /dev/sdc1

and then change the second partition of the old drive to make it the same size as the one on the new drive:

$ parted /dev/sdc
rm 2
mkpart
toggle 2 raid
print

before adding it to the new array:

mdadm /dev/md1 -a /dev/sdc1

Rename the new array To change the name of the new RAID array back to what it was on the old drive, I first had to stop both the old and the new RAID arrays:

umount /home
cryptsetup luksClose chome
mdadm --stop /dev/md10
mdadm --stop /dev/md1

before running this command:

mdadm --assemble /dev/md1 --name=mymachinename:1 --update=name /dev/sdd2

and updating the name in /etc/mdadm/mdadm.conf. The last step was to regenerate the initramfs:

update-initramfs -u

before rebooting into something that looks exactly like the original RAID1 array but with twice the size.

Here is how I managed to extend my OpenVPN setup on my Linode VPS to include IPv6 traffic. This ensures that clients can route all of their traffic through the VPN and avoid leaking IPv6 traffic, for example. It also enables clients on IPv4-only networks to receive a routable IPv6 address and connect to IPv6-only servers (i.e. running your own IPv6 broker).

Request an additional IPv6 block The first thing you need to do is get a new IPv6 address block (or "pool" as Linode calls it) from which you can allocate a single address to each VPN client that connects to the server. If you are using a Linode VPS, there are instructions on how to request a new IPv6 pool. Note that you need to get an address block between /64 and /112. A /116 like Linode offers won't work in OpenVPN. Thankfully, Linode is happy to allocate you an extra /64 for free.

Setup the new IPv6 address If your server only has an single IPv4 address and a single IPv6 address, then a simple DHCP-backed network configuration will work fine. To add the second IPv6 block on the other hand, I had to change my network configuration (/etc/network/interfaces) to this:

auto lo
iface lo inet loopback
allow-hotplug eth0
iface eth0 inet dhcp
    pre-up iptables-restore /etc/network/iptables.up.rules
iface eth0 inet6 static
    address 2600:3c01::xxxx:xxxx:xxxx:939f/64
    gateway fe80::1
    pre-up ip6tables-restore /etc/network/ip6tables.up.rules
iface tun0 inet6 static
    address 2600:3c01:xxxx:xxxx::/64
    pre-up ip6tables-restore /etc/network/ip6tables.up.rules

where 2600:3c01::xxxx:xxxx:xxxx:939f/64 (bound to eth0) is your main IPv6 address and 2600:3c01:xxxx:xxxx::/64 (bound to tun0) is the new block you requested. Once you've setup the new IPv6 block, test it from another IPv6-enabled host using:

ping6 2600:3c01:xxxx:xxxx::1

OpenVPN configuration The only thing I had to change in my OpenVPN configuration (/etc/openvpn/server.conf) was to change:

proto udp

to:

proto udp6

in order to make the VPN server available over both IPv4 and IPv6, and to add the following lines:

server-ipv6 2600:3c01:xxxx:xxxx::/64
push "route-ipv6 2000::/3"

to bind to the right V6 address and to tell clients to tunnel all V6 Internet traffic through the VPN. In addition to updating the OpenVPN config, you will need to add the following line to /etc/sysctl.d/openvpn.conf:

net.ipv6.conf.all.forwarding=1

and the following to your firewall (e.g. /etc/network/ip6tables.up.rules):

# openvpn
-A INPUT -p udp --dport 1194 -j ACCEPT
-A FORWARD -m state --state NEW -i tun0 -o eth0 -s 2600:3c01:xxxx:xxxx::/64 -j ACCEPT
-A FORWARD -m state --state NEW -i eth0 -o tun0 -d 2600:3c01:xxxx:xxxx::/64 -j ACCEPT
-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

in order to ensure that IPv6 packets are forwarded from the eth0 network interface to tun0 on the VPN server. With all of this done, apply the settings by running:

sysctl -p /etc/sysctl.d/openvpn.conf
ip6tables-apply
systemctl restart openvpn.service

Testing the connection Now connect to the VPN using your desktop client and check that the default IPv6 route is set correctly using ip -6 route. Then you can ping the server's new IP address:

ping6 2600:3c01:xxxx:xxxx::1

and from the server, you can ping the client's IP (which you can see in the network settings):

ping6 2600:3c01:xxxx:xxxx::1002

Once both ends of the tunnel can talk to each other, you can try pinging an IPv6-only server from your client:

ping6 ipv6.google.com

and then pinging your client from an IPv6-enabled host somewhere:

ping6 2600:3c01:xxxx:xxxx::1002

If that works, other online tests should also work.

I recently setup a music server on my home server using the Music Player Daemon, a cross-platform free software project which has been around for a long time.

Basic setup Start by installing the server and the client package:

apt install mpd mpc

then open /etc/mpd.conf and set these:

music_directory    "/path/to/music/"
bind_to_address    "192.168.1.2"
bind_to_address    "/run/mpd/socket"
zeroconf_enabled   "yes"
password           "Password1"

before replacing the alsa output:

audio_output  
   type    "alsa"
   name    "My ALSA Device"
 

with a pulseaudio one:

audio_output  
   type    "pulse"
   name    "Pulseaudio Output"
 

In order for the automatic detection (zeroconf) of your music server to work, you need to prevent systemd from creating the network socket:

systemctl stop mpd.service
systemctl stop mpd.socket
systemctl disable mpd.socket

otherwise you'll see this in /var/log/mpd/mpd.log:

zeroconf: No global port, disabling zeroconf

Once all of that is in place, start the mpd daemon:

systemctl start mpd.service

and create an index of your music files:

MPD_HOST=Password1@/run/mpd/socket mpc update

while watching the logs to notice any files that the mpd user doesn't have access to:

tail -f /var/log/mpd/mpd.log

Enhancements I also added the following in /etc/logcheck/ignore.server.d/local-mpd to silence unnecessary log messages in logcheck emails:

^\w 3  [ :0-9] 11  [._[:alnum:]-]+ systemd\[1\]: Started Music Player Daemon.$
^\w 3  [ :0-9] 11  [._[:alnum:]-]+ systemd\[1\]: Stopped Music Player Daemon.$
^\w 3  [ :0-9] 11  [._[:alnum:]-]+ systemd\[1\]: Stopping Music Player Daemon...$

and created a cronjob in /etc/cron.d/mpd-francois to update the database daily and stop the music automatically in the evening:

# Refresh DB once a day
5 1 * * *  mpd  MPD_HOST=Password1@/run/mpd/socket /usr/bin/mpc --quiet update
# Think of the neighbours
0 22 * * 0-4  mpd  MPD_HOST=Password1@/run/mpd/socket /usr/bin/mpc --quiet stop
0 23 * * 5-6  mpd  MPD_HOST=Password1@/run/mpd/socket /usr/bin/mpc --quiet stop

Clients To let anybody on the local network connect, I opened port 6600 on the firewall (/etc/network/iptables.up.rules since I'm using Debian's iptables-apply):

-A INPUT -s 192.168.1.0/24 -p tcp --dport 6600 -j ACCEPT

Then I looked at the long list of clients on the mpd wiki.

Desktop The official website suggests two clients which are available in Debian and Ubuntu:

• Ario (apt install ario)
• GNOME Music Player Client (apt install gmpc gmpc-plugins)

Both of them work well, but haven't had a release since 2011, even though there is some activity in 2013 and 2015 in their respective source control repositories. Ario has a simpler user interface but gmpc has cover art download working out of the box, which is why I might stick with it. In both cases, it is possible to configure a polipo proxy so that any external resources are fetched via Tor.

Android On Android, I got these two to work:

• M.A.L.P. (requires Android 5 or later)
• MPDroid

I picked M.A.L.P. since it includes a nice widget for the homescreen.

iOS On iOS, these are the most promising clients I found:

• MPDRemote (free software, not on the AppStore)
• MPDluxe (proprietary, sold on the AppStore)

since MPoD and MPaD don't appear to be available on the AppStore anymore.

I used to rely on ifupdown to bring up my iptables firewall automatically using a config like this in /etc/network/interfaces:

allow-hotplug eth0
iface eth0 inet dhcp
    pre-up /sbin/iptables-restore /etc/network/iptables.up.rules
    pre-up /sbin/ip6tables-restore /etc/network/ip6tables.up.rules
allow-hotplug wlan0
iface wlan0 inet dhcp
    pre-up /sbin/iptables-restore /etc/network/iptables.up.rules
    pre-up /sbin/ip6tables-restore /etc/network/ip6tables.up.rules

but that doesn't seem to work very well in the brave new NetworkManager world. What does work reliably is a "pre-up" NetworkManager script, something that gets run before a network interface is brought up. However, despite what the documentation says, a dispatcher script in /etc/NetworkManager/dispatched.d/ won't work on my Debian and Ubuntu machines. Instead, I had to create a new iptables script in /etc/NetworkManager/dispatcher.d/pre-up.d/:

#!/bin/sh
LOGFILE=/var/log/iptables.log
if [ "$1" = lo ]; then
    echo "$0: ignoring $1 for \ $2'" >> $LOGFILE
    exit 0
fi
case "$2" in
    pre-up)
        echo "$0: restoring iptables rules for $1" >> $LOGFILE
        /sbin/iptables-restore /etc/network/iptables.up.rules >> $LOGFILE 2>&1
        /sbin/ip6tables-restore /etc/network/ip6tables.up.rules >> $LOGFILE 2>&1
        ;;
    *)
        echo "$0: nothing to do with $1 for \ $2'" >> $LOGFILE
        ;;
esac
exit 0

and then make that script executable:

chmod a+x /etc/NetworkManager/dispatcher.d/pre-up.d/iptables

With this in place, I can put my iptables rules in the usual place (/etc/network/iptables.up.rules and /etc/network/ip6tables.up.rules) and use the handy iptables-apply and ip6tables-apply commands to test any changes to my firewall rules.

The Referer header has been a part of the web for a long time. Websites rely on it for a few different purposes (e.g. analytics, ads, CSRF protection) but it can be quite problematic from a privacy perspective. Thankfully, there are now tools in Firefox to help users and developers mitigate some of these problems.

Description In a nutshell, the browser adds a Referer header to all outgoing HTTP requests, revealing to the server on the other end the URL of the page you were on when you placed the request. For example, it tells the server where you were when you followed a link to that site, or what page you were on when you requested an image or a script. There are, however, a few limitations to this simplified explanation. First of all, by default, browsers won't send a referrer if you place a request from an HTTPS page to an HTTP page. This would reveal potentially confidential information (such as the URL path and query string which could contain session tokens or other secret identifiers) from a secure page over an insecure HTTP channel. Firefox will however include a Referer header in HTTPS to HTTPS transitions unless network.http.sendSecureXSiteReferrer (removed in Firefox 52) is set to false in about:config. Secondly, using the new Referrer Policy specification web developers can override the default behaviour for their pages, including on a per-element basis. This can be used both to increase or reduce the amount of information present in the referrer.

Legitimate Uses Because the Referer header has been around for so long, a number of techniques rely on it. Armed with the Referer information, analytics tools can figure out:

• where website traffic comes from, and
• how users are navigating the site.

Another place where the Referer is useful is as a mitigation against cross-site request forgeries. In that case, a website receiving a form submission can reject that form submission if the request originated from a different website. It's worth pointing out that this CSRF mitigation might be better implemented via a separate header that could be restricted to particularly dangerous requests (i.e. POST and DELETE requests) and only include the information required for that security check (i.e. the origin).

Problems with the Referrer Unfortunately, this header also creates significant privacy and security concerns. The most obvious one is that it leaks part of your browsing history to sites you visit as well as all of the resources they pull in (e.g. ads and third-party scripts). It can be quite complicated to fix these leaks in a cross-browser way. These leaks can also lead to exposing private personally-identifiable information when they are part of the query string. One of the most high-profile example is the accidental leakage of user searches by healthcare.gov.

Solutions for Firefox Users While web developers can use the new mechanisms exposed through the Referrer Policy, Firefox users can also take steps to limit the amount of information they send to websites, advertisers and trackers. In addition to enabling Firefox's built-in tracking protection by setting privacy.trackingprotection.enabled to true in about:config, which will prevent all network connections to known trackers, users can control when the Referer header is sent by setting network.http.sendRefererHeader to:

• 0 to never send the header
• 1 to send the header only when clicking on links and similar elements
• 2 (default) to send the header on all requests (e.g. images, links, etc.)

It's also possible to put a limit on the maximum amount of information that the header will contain by setting the network.http.referer.trimmingPolicy to:

• 0 (default) to send the full URL
• 1 to send the URL without its query string
• 2 to only send the scheme, host and port

or using the network.http.referer.XOriginTrimmingPolicy option (added in Firefox 52) to only restrict the contents of referrers attached to cross-origin requests. Site owners can opt to share less information with other sites, but they can't share any more than what the user trimming policies allow. Another approach is to disable the Referer when doing cross-origin requests (from one site to another). The network.http.referer.XOriginPolicy preference can be set to:

• 0 (default) to send the referrer in all cases
• 1 to send a referrer only when the base domains are the same
• 2 to send a referrer only when the full hostnames match

Breakage If you try to remove all referrers (i.e. network.http.sendRefererHeader = 0, you will most likely run into problems on a number of sites, for example:

• anything that uses the default Django authentication
• Launchpad logins
• AMD driver downloads
• some CDN-hosted images

The first two have been worked-around successfully by setting network.http.referer.spoofSource to true, an advanced setting which always sends the destination URL as the referrer, thereby not leaking anything about the original page. Unfortunately, the last two are examples of the kind of breakage that can only be fixed through a whitelist (an approach supported by the smart referer add-on) or by temporarily using a different browser profile.

My Recommended Settings As with my cookie recommendations, I recommend strengthening your referrer settings but not disabling (or spoofing) it entirely. While spoofing does solve many the breakage problems mentioned above, it also effectively disables the anti-CSRF protections that some sites may rely on and that have tangible user benefits. A better approach is to limit the amount of information that leaks through cross-origin requests. If you are willing to live with some amount of breakage, you can simply restrict referrers to the same site by setting:

network.http.referer.XOriginPolicy = 2

or to sites which belong to the same organization (i.e. same ETLD/public suffix) using:

network.http.referer.XOriginPolicy = 1

This prevent leaks to third-parties while giving websites all of the information that they can already see in their own server logs. On the other hand, if you prefer a weaker but more compatible solution, you can trim cross-origin referrers down to just the scheme, hostname and port:

network.http.referer.XOriginTrimmingPolicy = 2

I have not yet found user-visible breakage using this last configuration. Let me know if you find any!

After upgrading an Ubuntu 14.04 ("trusty") machine to the latest 16.04 Hardware Enablement packages, I ran into login problems. I could log into my user account and see the GNOME desktop for a split second before getting thrown back into the LightDM login manager. The solution I found was to install this missing package:

apt install libwayland-egl1-mesa-lts-xenial

Looking for clues in the logs The first place I looked was the log file for the login manager (/var/log/lightdm/lightdm.log) where I found the following:

DEBUG: Session pid=12743: Running command /usr/sbin/lightdm-session gnome-session --session=gnome
DEBUG: Creating shared data directory /var/lib/lightdm-data/username
DEBUG: Session pid=12743: Logging to .xsession-errors

This told me that the login manager runs the gnome-session command and gets it to create a session of type gnome. That command line is defined in /usr/share/xsessions/gnome.desktop (look for Exec=):

[Desktop Entry]
Name=GNOME
Comment=This session logs you into GNOME
Exec=gnome-session --session=gnome
TryExec=gnome-shell
X-LightDM-DesktopName=GNOME

I couldn't see anything unexpected there, but it did point to another log file (~/.xsession-errors) which contained the following:

Script for ibus started at run_im.
Script for auto started at run_im.
Script for default started at run_im.
init: Le processus gnome-session (GNOME) main (11946) s'est achev  avec l' tat 1
init: D connect  du bus D-Bus notifi 
init: Le processus logrotate main (11831) a  t  tu  par le signal TERM
init: Le processus update-notifier-crash (/var/crash/_usr_bin_unattended-upgrade.0.crash) main (11908) a  t  tu  par le signal TERM

Seaching for French error messages isn't as useful as searching for English ones, so I took a look at /var/log/syslog and found this:

gnome-session[4134]: WARNING: App 'gnome-shell.desktop' exited with code 127
gnome-session[4134]: WARNING: App 'gnome-shell.desktop' exited with code 127
gnome-session[4134]: WARNING: App 'gnome-shell.desktop' respawning too quickly
gnome-session[4134]: CRITICAL: We failed, but the fail whale is dead. Sorry....

It looks like gnome-session is executing gnome-shell and that this last command is terminating prematurely. This would explain why gnome-session exits immediately after login.

Increasing the amount of logging In order to get more verbose debugging information out of gnome-session, I created a new type of session (GNOME debug) by copying the regular GNOME session:

cp /usr/share/xsessions/gnome.desktop /usr/share/xsessions/gnome-debug.desktop

and then adding --debug to the command line inside gnome-debug.desktop:

[Desktop Entry]
Name=GNOME debug
Comment=This session logs you into GNOME debug
Exec=gnome-session --debug --session=gnome
TryExec=gnome-shell
X-LightDM-DesktopName=GNOME debug

After restarting LightDM (service lightdm restart), I clicked the GNOME logo next to the password field and chose GNOME debug before trying to login again. This time, I had a lot more information in ~/.xsession-errors:

gnome-session[12878]: DEBUG(+): GsmAutostartApp: starting gnome-shell.desktop: command=/usr/bin/gnome-shell startup-id=10d41f1f5c81914ec61471971137183000000128780000
gnome-session[12878]: DEBUG(+): GsmAutostartApp: started pid:13121
...
/usr/bin/gnome-shell: error while loading shared libraries: libwayland-egl.so.1: cannot open shared object file: No such file or directory
gnome-session[12878]: DEBUG(+): GsmAutostartApp: (pid:13121) done (status:127)
gnome-session[12878]: WARNING: App 'gnome-shell.desktop' exited with code 127

which suggests that gnome-shell won't start because of a missing library.

Finding the missing library To find the missing library, I used the apt-file command:

apt-file update
apt-file search libwayland-egl.so.1

and found that this file is provided by the following packages:

• libhybris
• libwayland-egl1-mesa
• libwayland-egl1-mesa-dbg
• libwayland-egl1-mesa-lts-utopic
• libwayland-egl1-mesa-lts-vivid
• libwayland-egl1-mesa-lts-wily
• libwayland-egl1-mesa-lts-xenial

Since I installed the LTS Enablement stack, the package I needed to install to fix this was libwayland-egl1-mesa-lts-xenial. I filed a bug for this on Launchpad.

Traduction de l'article original anglais https://feeding.cloud.geek.nz/posts/replacing-a-failed-raid-drive/. Voici la proc dure que j'ai suivi pour remplacer un disque RAID d fectueux sur une machine Debian.

Remplacer le disque Apr s avoir remarqu que /dev/sdb a t expuls de mon RAID, j'ai utilis smartmontools pour identifier le num ro de s rie du disque retirer :

smartctl -a /dev/sdb

Cette information en main, j'ai ferm l'ordinateur, retir le disque d fectueux et mis un nouveau disque vide la place.

Initialiser le nouveau disque Apr s avoir d marr avec le nouveau disque vide, j'ai copi la table de partitions avec parted. Premi rement, j'ai examin la table de partitions sur le disque dur non-d fectueux :

$ parted /dev/sda
unit s
print

et cr une nouvelle table de partitions sur le disque de remplacement :

$ parted /dev/sdb
unit s
mktable gpt

Ensuite j'ai utilis la commande mkpart pour mes 4 partitions et je leur ai toutes donn la m me taille que les partitions quivalentes sur /dev/sda. Finalement, j'ai utilis les commandes toggle 1 bios_grub (partition d'amorce) et toggle X raid (o X est le num ro de la partition) pour toutes les partitions RAID, avant de v rifier avec la commande print que les deux tables de partitions sont maintenant identiques.

Resynchroniser/recr er les RAID Pour synchroniser les donn es du bon disque (/dev/sda) vers celui de remplacement (/dev/sdb), j'ai ex cut les commandes suivantes sur mes partitions RAID1 :

mdadm /dev/md0 -a /dev/sdb2
mdadm /dev/md2 -a /dev/sdb4

et j'ai gard un oeil sur le statut de la synchronisation avec :

watch -n 2 cat /proc/mdstat

Pour acc l rer le processus, j'ai utilis le truc suivant :

blockdev --setra 65536 "/dev/md0"
blockdev --setra 65536 "/dev/md2"
echo 300000 > /proc/sys/dev/raid/speed_limit_min
echo 1000000 > /proc/sys/dev/raid/speed_limit_max

Ensuite, j'ai recr ma partition swap RAID0 comme suit :

mdadm /dev/md1 --create --level=0 --raid-devices=2 /dev/sda3 /dev/sdb3
mkswap /dev/md1

Par que la partition swap est toute neuve (il n'est pas possible de restorer une partition RAID0, il faut la re-cr er compl tement), j'ai d faire deux choses:

• remplacer le UUID pour swap dans /etc/fstab, avec le UUID donn par la commande mkswap (ou bien en utilisant la command blkid et en prenant le UUID pour /dev/md1)
• remplacer le UUID de /dev/md1 dans /etc/mdadm/mdadm.conf avec celui retourn pour /dev/md1 par la commande mdadm --detail --scan

S'assurer que l'on peut d marrer avec le disque de remplacement Pour tre certain de bien pouvoir d marrer la machine avec n'importe quel des deux disques, j'ai r install le boot loader grub sur le nouveau disque :

grub-install /dev/sdb

avant de red marrer avec les deux disques connect s. Ceci confirme que ma configuration fonctionne bien. Ensuite, j'ai d marr sans le disque /dev/sda pour m'assurer que tout fonctionnerait bien si ce disque d cidait de mourir et de me laisser seulement avec le nouveau (/dev/sdb). Ce test brise videmment la synchronisation entre les deux disques, donc j'ai d red marrer avec les deux disques connect s et puis r -ajouter /dev/sda tous les RAID1 :

mdadm /dev/md0 -a /dev/sda2
mdadm /dev/md2 -a /dev/sda4

Une fois le tout fini, j'ai red marrer nouveau avec les deux disques pour confirmer que tout fonctionne bien :

cat /proc/mdstat

et j'ai ensuite ex cuter un test SMART complet sur le nouveau disque :

smartctl -t long /dev/sdb

Traduction de l'article original anglais https://feeding.cloud.geek.nz/posts/replacing-a-failed-raid-drive/. Voici la proc dure que j'ai suivi pour remplacer un disque RAID d fectueux sur une machine Debian.

Remplacer le disque Apr s avoir remarqu que /dev/sdb a t expuls de mon RAID, j'ai utilis smartmontools pour identifier le num ro de s rie du disque retirer :

smartctl -a /dev/sdb

Cette information en main, j'ai ferm l'ordinateur, retir le disque d fectueux et mis un nouveau disque vide la place.

Initialiser le nouveau disque Apr s avoir d marr avec le nouveau disque vide, j'ai copi la table de partitions avec parted. Premi rement, j'ai examin la table de partitions sur le disque dur non-d fectueux :

$ parted /dev/sda
unit s
print

et cr une nouvelle table de partitions sur le disque de remplacement :

$ parted /dev/sdb
unit s
mktable gpt

Ensuite j'ai utilis la commande mkpart pour mes 4 partitions et je leur ai toutes donn la m me taille que les partitions quivalentes sur /dev/sda. Finalement, j'ai utilis les commandes toggle 1 bios_grub (partition d'amorce) et toggle X raid (o X est le num ro de la partition) pour toutes les partitions RAID, avant de v rifier avec la commande print que les deux tables de partitions sont maintenant identiques.

Resynchroniser/recr er les RAID Pour synchroniser les donn es du bon disque (/dev/sda) vers celui de remplacement (/dev/sdb), j'ai ex cut les commandes suivantes sur mes partitions RAID1 :

mdadm /dev/md0 -a /dev/sdb2
mdadm /dev/md2 -a /dev/sdb4

et j'ai gard un oeil sur le statut de la synchronisation avec :

watch -n 2 cat /proc/mdstat

Pour acc l rer le processus, j'ai utilis le truc suivant :

blockdev --setra 65536 "/dev/md0"
blockdev --setra 65536 "/dev/md2"
echo 300000 > /proc/sys/dev/raid/speed_limit_min
echo 1000000 > /proc/sys/dev/raid/speed_limit_max

Ensuite, j'ai recr ma partition swap RAID0 comme suit :

mdadm /dev/md1 --create --level=0 --raid-devices=2 /dev/sda3 /dev/sdb3
mkswap /dev/md1

Par que la partition swap est toute neuve (il n'est pas possible de restorer une partition RAID0, il faut la re-cr er compl tement), j'ai d faire deux choses:

• remplacer le UUID pour swap dans /etc/fstab, avec le UUID donn par la commande mkswap (ou bien en utilisant la command blkid et en prenant le UUID pour /dev/md1)
• remplacer le UUID de /dev/md1 dans /etc/mdadm/mdadm.conf avec celui retourn pour /dev/md1 par la commande mdadm --detail --scan

S'assurer que l'on peut d marrer avec le disque de remplacement Pour tre certain de bien pouvoir d marrer la machine avec n'importe quel des deux disques, j'ai r install le boot loader grub sur le nouveau disque :

grub-install /dev/sdb

avant de red marrer avec les deux disques connect s. Ceci confirme que ma configuration fonctionne bien. Ensuite, j'ai d marr sans le disque /dev/sda pour m'assurer que tout fonctionnerait bien si ce disque d cidait de mourir et de me laisser seulement avec le nouveau (/dev/sdb). Ce test brise videmment la synchronisation entre les deux disques, donc j'ai d red marrer avec les deux disques connect s et puis r -ajouter /dev/sda tous les RAID1 :

mdadm /dev/md0 -a /dev/sda2
mdadm /dev/md2 -a /dev/sda4

Une fois le tout fini, j'ai red marrer nouveau avec les deux disques pour confirmer que tout fonctionne bien :

cat /proc/mdstat

et j'ai ensuite ex cuter un test SMART complet sur le nouveau disque :

smartctl -t long /dev/sdb

Here's the complete procedure I followed to replace a failed drive from a RAID array on a Debian machine.

Replace the failed drive After seeing that /dev/sdb had been kicked out of my RAID array, I used smartmontools to identify the serial number of the drive to pull out:

smartctl -a /dev/sdb

Armed with this information, I shutdown the computer, pulled the bad drive out and put the new blank one in.

Initialize the new drive After booting with the new blank drive in, I copied the partition table using parted. First, I took a look at what the partition table looks like on the good drive:

$ parted /dev/sda
unit s
print

and created a new empty one on the replacement drive:

$ parted /dev/sdb
unit s
mktable gpt

then I ran mkpart for all 4 partitions and made them all the same size as the matching ones on /dev/sda. Finally, I ran toggle 1 bios_grub (boot partition) and toggle X raid (where X is the partition number) for all RAID partitions, before verifying using print that the two partition tables were now the same.

Resync/recreate the RAID arrays To sync the data from the good drive (/dev/sda) to the replacement one (/dev/sdb), I ran the following on my RAID1 partitions:

mdadm /dev/md0 -a /dev/sdb2
mdadm /dev/md2 -a /dev/sdb4

and kept an eye on the status of this sync using:

watch -n 2 cat /proc/mdstat

In order to speed up the sync, I used the following trick:

blockdev --setra 65536 "/dev/md0"
blockdev --setra 65536 "/dev/md2"
echo 300000 > /proc/sys/dev/raid/speed_limit_min
echo 1000000 > /proc/sys/dev/raid/speed_limit_max

Then, I recreated my RAID0 swap partition like this:

mdadm /dev/md1 --create --level=0 --raid-devices=2 /dev/sda3 /dev/sdb3
mkswap /dev/md1

Because the swap partition is brand new (you can't restore a RAID0, you need to re-create it), I had to update two things:

• replace the UUID for the swap mount in /etc/fstab, with the one returned by mkswap (or running blkid and looking for /dev/md1)
• replace the UUID for /dev/md1 in /etc/mdadm/mdadm.conf with the one returned for /dev/md1 by mdadm --detail --scan

Ensuring that I can boot with the replacement drive In order to be able to boot from both drives, I reinstalled the grub boot loader onto the replacement drive:

grub-install /dev/sdb

before rebooting with both drives to first make sure that my new config works. Then I booted without /dev/sda to make sure that everything would be fine should that drive fail and leave me with just the new one (/dev/sdb). This test obviously gets the two drives out of sync, so I rebooted with both drives plugged in and then had to re-add /dev/sda to the RAID1 arrays:

mdadm /dev/md0 -a /dev/sda2
mdadm /dev/md2 -a /dev/sda4

Once that finished, I rebooted again with both drives plugged in to confirm that everything is fine:

cat /proc/mdstat

Then I ran a full SMART test over the new replacement drive:

smartctl -t long /dev/sdb

As part of regular operating system hygiene, I run a cron job which updates package metadata and looks for obsolete packages and configuration files. While there is already some easily available information on how to purge unneeded or obsolete packages and how to clean up config files properly in maintainer scripts, the guidance on how to delete obsolete config files is not easy to find and somewhat incomplete. These are the obsolete conffiles I started with:

$ dpkg-query -W -f='$ Conffiles \n'   grep 'obsolete$'
 /etc/apparmor.d/abstractions/evince ae2a1e8cf5a7577239e89435a6ceb469 obsolete
 /etc/apparmor.d/tunables/ntpd 5519e4c01535818cb26f2ef9e527f191 obsolete
 /etc/apparmor.d/usr.bin.evince 08a12a7e468e1a70a86555e0070a7167 obsolete
 /etc/apparmor.d/usr.sbin.ntpd a00aa055d1a5feff414bacc89b8c9f6e obsolete
 /etc/bash_completion.d/initramfs-tools 7eeb7184772f3658e7cf446945c096b1 obsolete
 /etc/bash_completion.d/insserv 32975fe14795d6fce1408d5fd22747fd obsolete
 /etc/dbus-1/system.d/com.redhat.NewPrinterNotification.conf 8df3896101328880517f530c11fff877 obsolete
 /etc/dbus-1/system.d/com.redhat.PrinterDriversInstaller.conf d81013f5bfeece9858706aed938e16bb obsolete

To get rid of the /etc/bash_completion.d/ files, I first determined what packages they were registered to:

$ dpkg -S /etc/bash_completion.d/initramfs-tools
initramfs-tools: /etc/bash_completion.d/initramfs-tools
$ dpkg -S /etc/bash_completion.d/insserv
initramfs-tools: /etc/bash_completion.d/insserv

and then followed Paul Wise's instructions:

$ rm /etc/bash_completion.d/initramfs-tools /etc/bash_completion.d/insserv
$ apt install --reinstall initramfs-tools insserv

For some reason that didn't work for the /etc/dbus-1/system.d/ files and I had to purge and reinstall the relevant package:

$ dpkg -S /etc/dbus-1/system.d/com.redhat.NewPrinterNotification.conf
system-config-printer-common: /etc/dbus-1/system.d/com.redhat.NewPrinterNotification.conf
$ dpkg -S /etc/dbus-1/system.d/com.redhat.PrinterDriversInstaller.conf
system-config-printer-common: /etc/dbus-1/system.d/com.redhat.PrinterDriversInstaller.conf
$ apt purge system-config-printer-common
$ apt install system-config-printer

The files in /etc/apparmor.d/ were even more complicated to deal with because purging the packages that they come from didn't help:

$ dpkg -S /etc/apparmor.d/abstractions/evince
evince: /etc/apparmor.d/abstractions/evince
$ apt purge evince
$ dpkg-query -W -f='$ Conffiles \n'   grep 'obsolete$'
 /etc/apparmor.d/abstractions/evince ae2a1e8cf5a7577239e89435a6ceb469 obsolete
 /etc/apparmor.d/usr.bin.evince 08a12a7e468e1a70a86555e0070a7167 obsolete

I was however able to get rid of them by also purging the apparmor profile packages that are installed on my machine:

$ apt purge apparmor-profiles apparmor-profiles-extra evince ntp
$ apt install apparmor-profiles apparmor-profiles-extra evince ntp

Not sure why I had to do this but I suspect that these files used to be shipped by one of the apparmor packages and then eventually migrated to the evince and ntp packages directly and dpkg got confused. If you're in a similar circumstance, you want want to search for the file you're trying to get rid of on Google and then you might end up on http://apt-browse.org/ which could lead you to the old package that used to own this file.

In order to monitor some of the machines I maintain, I rely on a simple email setup using logcheck. Unfortunately that system completely breaks down if mail delivery stops. This is the simple setup I've come up with to ensure that mail doesn't pile up on the remote machine.

Server setup The first thing I did on the server-side is to follow Sean Whitton's advice and configure postfix so that it keeps undelivered emails for 10 days (instead of 5 days, the default):

postconf -e maximal_queue_lifetime=10d

Then I created a new user:

adduser mailq-check

with a password straight out of pwgen -s 32. I gave ssh permission to that user:

adduser mailq-check sshuser

and then authorized my new ssh key (see next section):

sudo -u mailq-check -i
mkdir ~/.ssh/
cat - > ~/.ssh/authorized_keys

Laptop setup On my laptop, the machine from where I monitor the server's mail queue, I first created a new password-less ssh key:

ssh-keygen -t ed25519 -f .ssh/egilsstadir-mailq-check
cat ~/.ssh/egilsstadir-mailq-check.pub

which I then installed on the server. Then I added this cronjob in /etc/cron.d/egilsstadir-mailq-check:

0 2 * * * francois /usr/bin/ssh -i /home/francois/.ssh/egilsstadir-mailq-check mailq-check@egilsstadir mailq   grep -v "Mail queue is empty"

and that's it. I get a (locally delivered) email whenever the mail queue on the server is non-empty. There is a race condition built into this setup since it's possible that the server will want to send an email at 2am. However, all that does is send a spurious warning email in that case and so it's a pretty small price to pay for a dirt simple setup that's unlikely to break.

While there is real progress being made towards eliminating insecure HTTP traffic, DNS is a fundamental Internet service that still usually relies on unauthenticated cleartext. There are however a few efforts to try and fix this problem. Here is the setup I use on my Debian laptop to make use of both DNSSEC and DNSCrypt.

DNSCrypt DNSCrypt was created to enable end-users to encrypt the traffic between themselves and their chosen DNS resolver. To switch away from your ISP's default DNS resolver to a DNSCrypt resolver, simply install the dnscrypt-proxy package and then set it as the default resolver either in /etc/resolv.conf:

nameserver 127.0.2.1

if you are using a static network configuration or in /etc/dhcp/dhclient.conf:

supersede domain-name-servers 127.0.2.1;

if you rely on dynamic network configuration via DHCP. There are two things you might want to keep in mind when choosing your DNSCrypt resolver:

• whether or not they keep any logs of the DNS traffic
• whether or not they support DNSSEC

I have personally selected a resolver located in Iceland by setting the following in /etc/default/dnscrypt-proxy:

DNSCRYPT_PROXY_RESOLVER_NAME=ns0.dnscrypt.is

DNSSEC While DNSCrypt protects the confidentiality of our DNS queries, it doesn't give us any assurance that the results of such queries are the right ones. In order to authenticate results in that way and prevent DNS poisoning, a hierarchical cryptographic system was created: DNSSEC. In order to enable it, I have setup a local unbound DNSSEC resolver on my machine and pointed /etc/resolv.conf (or /etc/dhcp/dhclient.conf) to my unbound installation at 127.0.0.1. Then I put the following in /etc/unbound/unbound.conf.d/dnscrypt.conf:

server:
    # Remove localhost from the donotquery list
    do-not-query-localhost: no
forward-zone:
    name: "."
    forward-addr: 127.0.2.1@53

to stop unbound from resolving DNS directly and to instead go through the encrypted DNSCrypt proxy.

Reliability In my experience, unbound and dnscrypt-proxy are fairly reliable but they eventually get confused (presumably) by network changes and start returning errors. The ugly but dependable work-around I have found is to create a cronjob at /etc/cron.d/restart-dns.conf that restarts both services once a day:

0 3 * * *    root    /usr/sbin/service dnscrypt-proxy restart
1 3 * * *    root    /usr/sbin/service unbound restart

Captive portals The one remaining problem I need to solve has to do with captive portals. This can be quite annoying when travelling because it requires me to use the portal's DNS resolver in order to connect to the splash screen that unlocks the wifi connection. The dnssec-trigger package looked promising but when I tried it on my jessie laptop, it wasn't particularly reliable. My temporary work-around is to comment out this line in /etc/dhcp/dhclient.conf whenever I need to connect to such annoying wifi networks:

#supersede domain-name-servers 127.0.0.1;

If you've found a better solution to this problem, please leave a comment!

Firefox has had support for Google's Safe Browsing since 2005 when it started as a stand-alone Firefox extension. At first it was only available in the USA, but it was opened up to the rest of the world in 2006 and moved to the Google Toolbar. It then got integrated directly into Firefox 2.0 before the public launch of the service in 2007. Many people seem confused by this phishing and malware protection system and while there is a pretty good explanation of how it works on our support site, it doesn't go into technical details. This will hopefully be of interest to those who have more questions about it.

Browsing Protection The main part of the Safe Browsing system is the one that watches for bad URLs as you're browsing. Browsing protection currently protects users from:

• malware sites,
• deceptive sites (including phishing and social engineering sites), and
• sites hosting potentially unwanted software.

If a Firefox user attempts to visit one of these sites, a warning page will show up instead, which you can see for yourself here:

• fake malware page
• fake unwanted software page
• fake phishing page

The first two warnings can be toggled using the browser.safebrowsing.malware.enabled preference (in about:config) whereas the last one is controlled by browser.safebrowsing.enabled.

List updates It would be too slow (and privacy-invasive) to contact a trusted server every time the browser wants to establish a connection with a web server. Instead, Firefox downloads a list of bad URLs every 30 minutes from the server (browser.safebrowsing.provider.google.updateURL) and does a lookup against its local database before displaying a page to the user. Downloading the entire list of sites flagged by Safe Browsing would be impractical due to its size so the following transformations are applied:

1. each URL on the list is canonicalized,
2. then hashed,
3. of which only the first 32 bits of the hash are kept.

The lists that are requested from the Safe Browsing server and used to flag pages as malware/unwanted or phishing can be found in urlclassifier.malwareTable and urlclassifier.phishTable respectively. If you want to see some debugging information in your terminal while Firefox is downloading updated lists, turn on browser.safebrowsing.debug. Once downloaded, the lists can be found in the cache directory:

• ~/.cache/mozilla/firefox/XXXX/safebrowsing/ on Linux
• ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/ on Mac
• C:\Users\XXXX\AppData\Local\mozilla\firefox\profiles\XXXX\safebrowsing\ on Windows

Resolving partial hash conflicts Because the Safe Browsing database only contains partial hashes, it is possible for a safe page to share the same 32-bit hash prefix as a bad page. Therefore when a URL matches the local list, the browser needs to know whether or not the rest of the hash matches the entry on the Safe Browsing list. In order resolve such conflicts, Firefox requests from the Safe Browsing server (browser.safebrowsing.provider.mozilla.gethashURL) all of the hashes that start with the affected 32-bit prefix and adds these full-length hashes to its local database. Turn on browser.safebrowsing.debug to see some debugging information on the terminal while these "completion" requests are made. If the current URL doesn't match any of these full hashes, the load proceeds as normal. If it does match one of them, a warning interstitial page is shown and the load is canceled.

Download Protection The second part of the Safe Browsing system protects users against malicious downloads. It was launched in 2011, implemented in Firefox 31 on Windows and enabled in Firefox 39 on Mac and Linux. It roughly works like this:

1. Download the file.
2. Check the main URL, referrer and redirect chain against a local blocklist (urlclassifier.downloadBlockTable) and block the download in case of a match.
3. On Windows, if the binary is signed, check the signature against a local whitelist (urlclassifier.downloadAllowTable) of known good publishers and release the download if a match is found.
4. If the file is not a binary file then release the download.
5. Otherwise, send the binary file's metadata to the remote application reputation server (browser.safebrowsing.downloads.remote.url) and block the download if the server indicates that the file isn't safe.

Blocked downloads can be unblocked by right-clicking on them in the download manager and selecting "Unblock". While the download protection feature is automatically disabled when malware protection (browser.safebrowsing.malware.enabled) is turned off, it can also be disabled independently via the browser.safebrowsing.downloads.enabled preference. Note that Step 5 is the only point at which any information about the download is shared with Google. That remote lookup can be suppressed via the browser.safebrowsing.downloads.remote.enabled preference for those users concerned about sending that metadata to a third party.

Types of malware The original application reputation service would protect users against "dangerous" downloads, but it has recently been expanded to also warn users about unwanted software as well as software that's not commonly downloaded. These various warnings can be turned on and off in Firefox through the following preferences:

• browser.safebrowsing.downloads.remote.block_dangerous
• browser.safebrowsing.downloads.remote.block_dangerous_host
• browser.safebrowsing.downloads.remote.block_potentially_unwanted
• browser.safebrowsing.downloads.remote.block_uncommon

and tested using Google's test page. If you want to see how often each "verdict" is returned by the server, you can have a look at the telemetry results for Firefox Beta.

Privacy One of the most persistent misunderstandings about Safe Browsing is the idea that the browser needs to send all visited URLs to Google in order to verify whether or not they are safe. While this was an option in version 1 of the Safe Browsing protocol (as disclosed in their privacy policy at the time), support for this "enhanced mode" was removed in Firefox 3 and the version 1 server was decommissioned in late 2011 in favor of version 2 of the Safe Browsing API which doesn't offer this type of real-time lookup. Google explicitly states that the information collected as part of operating the Safe Browsing service "is only used to flag malicious activity and is never used anywhere else at Google" and that "Safe Browsing requests won't be associated with your Google Account". In addition, Firefox adds a few privacy protections:

• Query string parameters are stripped from URLs we check as part of the download protection feature.
• Cookies set by the Safe Browsing servers to protect the service from abuse are stored in a separate cookie jar so that they are not mixed with regular browsing/session cookies.
• When requesting complete hashes for a 32-bit prefix, Firefox throws in a number of extra "noise" entries to obfuscate the original URL further.

On balance, we believe that most users will want to keep Safe Browsing enabled, but we also make it easy for users with particular needs to turn it off.

Learn More If you want to learn more about how Safe Browsing works in Firefox, you can find all of the technical details on the Safe Browsing and Application Reputation pages of the Mozilla wiki or you can ask questions on our mailing list. Google provides some interesting statistics about what their systems detect in their transparency report and offers a tool to find out why a particular page has been blocked. Some information on how phishing sites are detected is also available on the Google Security blog, but for more detailed information about all parts of the Safe Browsing system, see the following papers:

• All Your IFrames Are Belong to Us
• Content-Agnostic Malware Protection
• Ghost in the Browser